Google introduces TorchTPU to break Nvidia’s stranglehold on AI computing

Google LLC is trying to wrest Nvidia Corp.’s hold on the artificial intelligence industry by launching a special version of the open-source PyTorch framework called TorchTPU that is designed to eliminate a key advantage of its rival.

One of the reasons Nvidia and its GPUs remain so dominant in the AI ​​industry is because the vast majority of AI applications are written specifically to run on its GPUs, and it’s that reality that Google is trying to change with today’s release.

The PyTorch framework is a popular open-source toolkit that developers use to build, train, and run AI models and neural networks. By some reports, more than 85% of all AI applications are built using PyTorch, which was developed by Meta Platforms Inc. in 2016, long before anyone realized that large language models would emerge to enable the AI ​​boom.

Even back then, Nvidia’s GPUs were the preferred hardware used by developers for AI research, so Meta designed PyTorch specifically to work with that company’s chips. The focus on Nvidia’s hardware remained even after Meta released PyTorch as an open source solution in 2022 and handed over responsibility to the nonprofit Linux Foundation.

With TorchTPU, Google has created an alternative version of PyTorch that makes it easier for developers to build models and applications that run on its Tensor Processing Units, or TPUs, dedicated chips for AI workloads. TorchTPU also provides developers with an easy way to run existing apps and models on TPUs by making a few simple configuration changes without having to rewrite the underlying code.

Google isn’t the only one trying to undermine the huge advantage Nvidia had thanks to PyTorch. In fact, Amazon Web Services Inc. announced its own PyTorch variant called TorchNeuron just earlier this week. Similar to TorchTPU, TorchNeuron makes it easier to run AI applications on different hardware, in this case Amazon’s Trainum chips. Amazon went even further and announced that it has contributed code to the original PyTorch project that will decouple a number of key features from Nvidia’s GPUs, with the goal of making it more compatible with any type of hardware.

Google didn’t do that, but it may have gained an advantage over AWS in other ways. For one, the company said it has worked closely with Meta, Hugging Face and a few startups called Inferact Inc. and RadixArk Inc., the developers of vLLM and SGLang, to ensure that various TorchTPU features and code libraries are compatible with their developer tools. This negated another advantage of PyTorch.

Google also did something else that AWS didn’t do. A new tool called TPU Agent Suite has been introduced to help developers move their AI applications to the Google Cloud Platform and automatically fix any errors that arise during the migration process. It also updated an existing tool called LiteRT, which creates AI applications that run directly on smartphones, laptops and embedded devices rather than in the cloud. Therefore, it should now be compatible with PyTorch-based apps, Google said.

“It’s clear that both cloud infrastructure providers are more eager than ever to win their fair share of AI workloads, and they understand that they need to be responsive to developers to get it,” said Holger Mueller of Constellation Research. “Rebuilding PyTorch is one of the best ways to achieve this, as it makes it easier for developers to move their application code to Google and Amazon’s respective computing platforms,” the analyst said. “It’s a win for developers and businesses alike because it means they have more choices about where they run their AI workloads, but it remains to be seen how much they can chip away at Nvidia’s entrenched position.”

Of course, it’ll take more than that to convince millions of developers to ditch Nvidia’s GPUs and switch to Google’s TPUs instead. However, it does suggest that Google is confident it can narrow Nvidia’s lead in the longer term.

Image: Google