Enterprise AI encounters a storage bottleneck

Over the past two years, corporate AI investment has been dominated by one theme: computing. Companies raced to secure GPU allocations, negotiated infrastructure deals, and deployed machine learning platforms at a rapid pace. What many only discovered after the fact was that the bottleneck they thought they were solving wasn’t the only one in the room. The memory now throttles the very accelerators that companies fought to acquire.

In the latest episode of the AppDevANGLE podcast, I spoke with Ronen Schwartz, vice president of products for Google Cloud Storage at Google LLC, about why storage has quietly become the defining constraint in enterprise AI infrastructure and what a purpose-built architecture for AI workloads actually looks like. The conversation is about a problem that is real, measurable, and still largely underestimated in most discussions about AI strategies.

“Storage has just been promoted,” Schwartz said. “Instead of being an enabler on the backend, it moves to the forefront as a crucial part of the accelerator execution.”

The hidden tax on GPU investments

The economics of this problem are straightforward. Accelerators are, as Schwartz put it, a company’s single most expensive item. Every second they spend idly waiting for data is money lost – not through a theoretical efficiency calculation, but through direct, traceable waste. But when teams finally get access to their TPUs or GPUs, the data delivery path often doesn’t receive the same attention as the compute layer.

“I talk to a lot of customers, and I hear that the team has struggled for a long time to get their TPUs, their GPUs, to get their hands on accelerators. And once they get them, they’re actually not always able to get the output that they needed. Very often it’s actually because there wasn’t enough thought put into the path of data delivery.”

This is not a fringe scenario. Schwartz described a pattern in which companies only realize the storage bottleneck when it hits them – when GPU utilization metrics fall short of targets and training schedules exceed forecasts. At this point, the costs of under-investing in the storage architecture are already built into the project.

Training, Inference, and Agents: Three Different Memory Problems

One of the clearest takeaways from the conversation is that AI is not a single workload and that treating it as a single workload leads to ever-widening gaps in the storage architecture. Schwartz described how training, inference and agent workloads have fundamentally different “heartbeats” with different data movement patterns, each requiring targeted design.

Training requires extensive sequential reads, but the equally critical pressure point is checkpoint management. Approximately every 15 minutes, a training cluster must stop and write a multi-terabyte checkpoint to protect the work being done from possible errors. “Slow memory in this location freezes the entire GPU cluster,” Schwartz said. He found that Google’s Rapid object storage technology can write checkpoints more than three times faster and restore more than five times faster than standard object storage, and this difference is significant at scale. He cited Torin, an autonomous vehicle company, as an example: millions of sensor data points flow into memory to train vehicle models, with checkpoint performance not being an edge case but a core operational requirement.

Inference introduces a completely different limitation, that of real-time latency-sensitive interaction, where response time directly impacts user experience and token economics. Agent workloads are based on a third set of requirements: shared workspaces, state persistence between agents, low latency polling, and the need to support both short-lived and long-running agent processes that can be hibernated and reactivated as needed.

“AI is not a single workload. In fact, the movement of data looks completely different. It’s like a different heartbeat depending on where you are in the lifecycle.”

Colossus-derived architecture as enterprise infrastructure

Google’s differentiation argument centers on the credibility of its infrastructure, built over years of running hyperscale AI workloads internally before offering the same capabilities externally. Schwartz gave specific performance numbers: 15 terabytes per second for Rapid Object Storage and 10 terabytes per second for Google Cloud-managed clusters, which he described as verified industry-leading numbers.

The technical foundation includes zonal colocation, or physically placing data near GPU and TPU clusters to minimize the time it takes a byte to get to the accelerator. Schwartz described this as a first-to-market feature that required rethinking the assumptions embedded in traditional storage design, where multi-zone or multi-region distribution was the default goal. For AI workloads, proximity to the accelerator is a priority. Google’s Rapid Bucket technology reduces processing steps in the storage path, allowing a single bucket to deliver throughput of more than 15 terabytes per second and support 20 million queries per second with latency of less than a millisecond.

“That’s because it’s built on Google’s infrastructure, which has been used internally to support accelerators and their performance longer than any other. But it’s also because Google works with nine of the world’s top 10 AI frontier model companies.” A memory architecture proven to improve GPU utilization can enable faster iteration through less idle processing power and a lower effective cost per training run.

Make unstructured data agent-ready

The agent dimension of the conversation revealed an underappreciated wrinkle in enterprise AI readiness. Agents require more than just web browsing and database queries; You need access to unstructured data at scale, including images, videos, documents and PDFs, which make up the majority of most organizations’ real-world data assets. Without a way to make this content semantically accessible, agents work with a limited and often unrepresentative view of corporate knowledge.

Google’s answer is what Schwartz called a smart storage initiative anchored by a recently introduced Object Enrichment feature. When a file or image lands in a storage area, Google’s AI models automatically extract content and attach structured metadata to the object – in seconds and at scale. The result is that what was previously unstructured becomes queryable by agents without manual intervention or downstream processing pipelines.

“An agent can now interact with so-called unstructured data because there is actually a very good understanding of the content.” The storage layer is also integrated into Google’s broader agent data cloud and borderless lakehouse architecture, allowing agents to access data stored outside of Google’s environment.

For a scalable agent infrastructure, Schwartz described dedicated Filestore agent volumes that integrate directly with Google Kubernetes Engine and the agent sandbox environment and are specifically designed to address the cost, responsiveness, and data sharing requirements of running thousands or millions of agents simultaneously.

Google Cloud Storage for AI: The Conclusion

The broader market change here is architectural rather than incremental. Enterprise AI infrastructure was built in layers (first compute, then frameworks, then data) and storage was often treated as a procurement checkbox rather than a performance variable. This sequence turns out to be costly. As GPU utilization targets become measurable KPIs and AI operating budgets come under greater scrutiny, the storage tier is being reevaluated in ways it hasn’t been seen in years.

Google’s position shows that hyperscale operational experience at the storage layer translates directly into business benefits – not as a marketing claim, but as a measurable difference in checkpoint speed, throughput, latency and agent responsiveness. The companies that have conducted frontier model training and inference at scale have been forced to solve these problems; Corporate teams are now encountering them for the first time.

For platform engineers, ML infrastructure teams, and enterprise architects, the implication is specific: storage architecture decisions made during the initial build of the AI ​​platform constrain or enable performance for each subsequent workload. Reconsidering these decisions after the bottleneck has occurred is significantly more expensive than considering them at the design stage.

The market is heading to a point where conversations about AI infrastructure routinely include storage as a primary layer rather than an afterthought. Organizations that recognize this change early will see measurably more value from the accelerator investments they have already made.

To hear the full conversation with Ronen Schwartz, check out the latest episode of the AppDevANGLE podcast. To learn more about how Google Cloud Storage can support the evolving needs of enterprise AI, explore Google’s storage solutions or connect to Google Cloud team to discuss the right approach for your organization’s workloads.

Here is theCUBE Research’s full interview with Google’s Ronen Schwartz:

Image: SiliconANGLE