Distributed training and inference: Google’s agent projects

Kubernetes has become the platform of choice for distributed training and inference, but what happens to computing power when running AI agents that spend most of their time waiting?

If an agent’s container or virtual machine remains provisioned, customers pay for both wait time and labor. To help customers control these costs and make agent workloads more predictable, Google LLC has donated three projects to the Cloud Native Computing Foundation, said Allan Naim (pictured left), head of product management, AI infrastructure at Google Cloud. For Kubernetes itself, the new challenge is to free up computing power between bursts of activity while allowing each agent to pick up where it left off.

“Act 2 was a few years ago; Kubernetes became the de facto inference and training system,” Naim said. “What’s really interesting right now is that we’re starting this third act… where the typical patterns are very, very different from how Kubernetes was designed.”

Naim and Gari Singh (right), product manager at Google Cloud, spoke with theCUBE’s Rebecca Knight at KubeCon + CloudNativeCon NA during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed how Google Cloud uses open source tools to reduce computing costs when AI agents remain idle. (*Disclosure below.)

Open building blocks for distributed training and inference meet agent AI

In agent workloads, cost is a serious concern because agents sit idle 90 to 95 percent of the time, waiting for models, test runs or human approvals, Naim explained. Google has developed Agent Substrate, an open source runtime environment that hosts AI agents in isolated environments on Kubernetes. When an agent is stopped, the runtime saves a snapshot of its memory and local files and releases its computing power to other agents. When work resumes, the agent is restored from this snapshot into an available sandbox.

“From an economic perspective, it helps,” Naim said. “We have seen a 10x cost reduction when running Agent Substrate on Kubernetes compared to running these agents directly on dedicated virtual machines and containers.”

Security is another concern. In response, Google open-sourced gVisor in 2018, which sits on top of the host kernel to block or monitor every system call made by an agent, according to Singh.

“Agents and agent harnesses are designed to solve a problem – and try to solve a problem – until they can no longer solve the problem,” Singh said. “They’ll find all sorts of clever workarounds. They know everything there is. They want to make sure there haven’t been any minor changes.” [your] low security level, no outbound network call made, no attempt to open a file it shouldn’t.”

Google is also working with Anyscale Inc. on KubeRay, which runs Anyscale’s Ray platform natively on Kubernetes, allowing platform teams to offer distributed training and inference to many research groups, Singh noted. OpenRL brings this approach to reinforcement learning, which provides guidance to retrained models, Naim added.

“Reinforcement learning used to require hiring a dozen PhD researchers,” Naim said. “Our goal was, ‘How can we bring reinforcement learning into the world of platform engineering so that you can run reinforcement learning in your Kubernetes environment and use it to actually drive continuous improvement in your agent workload?’”

Here’s the full video interview, part of SiliconANGLE and theCUBE’s coverage of KubeCon + CloudNativeCon NA:

(*Disclosure: Google Cloud sponsored this segment of theCUBE. Neither Google Cloud nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE