The cost of hybrid infrastructure is pushing AI inference to the sidelines

Enterprise artificial intelligence is entering a cost-conscious phase, and hybrid infrastructure is where this reckoning is playing out. As inference spending increases, business leaders are considering where models run, what models they use, and how much of their data they are willing to give to someone else.

These cost considerations shape purchasing decisions across industries, including among financial institutions trying to turn siled data into real-time action. According to Dell, more than 6,500 companies have adopted Dell AI Factory with Nvidia updates. According to Kyle Dufresne (pictured), senior vice president of sales at Dell, a drastic shift in the token economy is changing what customers want.

“It is incredible to see that in the last three months, 80% of tokens were generated through frontier models. Three months later, 80% of tokens worldwide are generated through open models,” said Dufresne. “Why? For cost reasons. And everyone is talking about tokenomics.”

Dufresne spoke with Dave Vellante of theCUBE Research at the Dell Technologies New York Symposium during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed tokenomics, hybrid infrastructure and how Dell helps customers plan AI investments. (*Disclosure below.)

Inference costs are pushing hybrid infrastructure to the sidelines

According to Dufresne, inference is now growing faster than training, although both areas are expanding rapidly. This shift is changing where AI work happens, as Dell sees inference running wherever data is created – in the cloud, in the data center and on PCs.

“We’re seeing business owners say, ‘Oh my God, the cost of inference is getting higher and higher. How can we reduce that cost without stifling innovation?'” Dufresne said. “We’re finding that it’s happening within that continuum, but increasingly right at the very edge.”

Economics also raises a strategic question about control. Companies must decide whether to outsource their data and differentiation, and they are now calculating AI based on where workloads are running, what models they are using and where their data resides, Dufresne explained. Dell has designed its latest AI Factory with Nvidia updates to help customers achieve results for the infrastructure they control.

“We see every single one [chief financial officer] In the world that has implemented these structures, he says, ‘Gosh, my token costs are exploding,'” he said. “‘How can we bring the costs down?'”

The offer is the other side of the ledger. Memory prices are driving technology costs up rather than down, and about 20% more storage will come to market next year while demand is about 200% higher, Dufresne estimated. Dell responds with collaborative planning, directing customers to more readily available components and providing advance insight into upcoming cost increases. Many customers find that their cost of capital is lower than component inflation, so they buy sooner.

“It’s not just about being able to provide the technology,” Dufresne said. “But also to be able to tell customers what will happen in the future so they can make better decisions today about how to use their capital.”

Here’s the full video interview, part of SiliconANGLE and theCUBE’s coverage of the Dell Technologies New York Symposium:

(*Disclosure: TheCUBE is a paid media partner for the Dell Technologies New York Symposium. Neither Dell, the sponsor of theCUBE’s event coverage, nor other sponsors have editorial control over theCUBE or SiliconANGLE content.)

Photo: SiliconANGLE