Inference drives two-thirds of AI workloads and is transforming chip deployment

AI spending is shifting from massive training clusters to smaller, quickly built sites, and modular data centers are becoming a natural home for inference workloads. Power, cooling and storage supply, as well as pure chip performance, now determine where tokens are generated.

Inference-oriented chip manufacturers want to take advantage of this change. Positron AI Inc., which raised $875 million in September at a $5 billion valuation, builds inference devices based on mass storage rather than scarce high-bandwidth storage. According to Cameron McCaskill (pictured), head of sales and business development at Positron, inference has overtaken training as the driving force behind infrastructure purchasing decisions.

“We’re at a point where about two-thirds of the workloads running through these AI infrastructures are inference, while a third is training. And in 2023, it was about 85% training and 15% inference,” McCaskill said. “Now you can see why inferences drive economic decisions about how to build out their infrastructure.”

McCaskill spoke with Rebecca Knight and John Furrier of theCUBE Research at SC26 during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed inference overtaking training, Positron’s commodity memory chip design, and why low-power modular data centers are suitable for inference. (*Disclosure below.)

Why modular data centers are suitable for low-power inference

Positron’s first-generation Atlas server ran on a programmable chip, and the company plans to roll out its own silicon for its Titan server early next year. McCaskill noted that Oracle Corp. is rolling out approximately 340 Atlas servers in its cloud this quarter. A Neocloud provider has also signed up for Atlas and committed to Titan. Positron’s key design decision was to avoid high-bandwidth memory.

“Everyone else in the industry uses what’s called HBM (high-bandwidth memory). The supply is very limited and very expensive,” McCaskill said. “We chose mass storage.”

This choice keeps power consumption low enough even for air-cooled rooms. A single Positron server uses about 4,500 watts, and a full rack of Titan servers holds a maximum of 40 kilowatts, McCaskill explained. This footprint is suitable for locations where operators can find a few megawatts of cheap electricity without major regulatory hurdles. For inference providers monetizing new capacity, speed of deployment is a big factor, according to McCaskill.

“One of the groups of companies that we’re seeing growth right now that are coming to us for products are these modular data centers, where they have half-megawatt or 1-megawatt or 2.5-megawatt modular data centers and they can deploy them very quickly,” McCaskill said. “And the goal of many of these inference players is that I want to go from land and power to monetizable tokens within 90 to 120 days. And these modular data centers, coupled with our technology, can be a great solution.”

Here’s the full video interview, part of SiliconANGLE and theCUBE’s coverage of SC26:

(*Disclosure: NYSE Wired sponsored this segment of theCUBE. Neither NYSE Wired nor other sponsors have editorial control over the content of theCUBE or SiliconANGLE.)

Photo: SiliconANGLE

Support our mission to keep content open and free by interacting with theCUBE community. Join theCUBE Alumni Trust Networkwhere technology leaders connect, share information and create opportunities.

  • Over 15 million viewers of theCUBE videosto spark conversations about AI, cloud, cybersecurity and more
  • Over 11.4k theCUBE alumni — Connect with more than 11,400 technology and business leaders shaping the future through a unique, trusted network
SiliconANGLE Media is a recognized leader in digital media innovation, combining breakthrough technology, strategic insights and real-time audience engagement. As the parent company of SiliconANGLE, theCUBE Network, theCUBE Research, CUBE365, theCUBE AI and theCUBE SuperStudios – with flagship locations in Silicon Valley and the New York Stock Exchange – SiliconANGLE Media operates at the intersection of media, technology and AI.

Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a dynamic ecosystem of industry-leading digital media brands, reaching over 15 million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is a game-changer in audience engagement, leveraging the theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of the industry conversation.

Avatar photo
Written by

Mira Edora

Mira Edora is a writer and contributor at CKSOR, creating clear and engaging articles on current topics, technology, science, lifestyle, and stories of interest to readers. She enjoys researching new developments and presenting useful information in a simple, accessible way. Through her writing, Mira aims to keep readers informed with timely, informative, and easy-to-understand content.

Leave a Comment