AI infrastructure hardware is now a systems game: 10 takeaways

Faster chips alone no longer determine AI expansion. Hundreds of thousands of processors now work together as one machine, so the AI ​​infrastructure hardware is designed and evaluated as a complete system. This system extends from the optical connections between racks to the memory, storage, and power that employ the GPUs.

The next phase is determined by the bottlenecks around the processor. Copper gives way to the light and the memory wall moves instead of falling. Energy systems built in cycles of up to a decade must keep pace with chips that turn over every year. There is no doubt about the demand: storage and optical laser capacities have been fully booked for years.

“AI has had a lot of focus on computing power and storage, but our view is that AI is really a data system,” said Irving Tan (pictured), CEO of Western Digital Corp. “Ultimately, the entire ecosystem must be able to work together and act in lockstep.”

Tan and other industry experts spoke with John Furrier of theCUBE Research during the CEO Summit in Santa Clara, California. The inaugural summit marked the 35th anniversary of the Chinese American Semiconductor Professional Association. Interviews covered optical interconnects, inference economics, memory pooling, maskless lithography, physics-based computing, energy resilience, and AI-driven chip design.

Here are 11 standout takeaways from the event:

1. Hard drives gain throughput as storage becomes the AI ​​data layer.

“Data creates intelligence and storage ensures it persists. Therefore, data needs to be divided into hot, warm and cold tiers across the cloud, edge and devices,” Tan explained. Citing IDC, he noted that the amount of data generated is expected to triple within three years, rising to over 700 zettabytes. Western Digital expects demand for data storage to grow at a compound annual rate of at least 25% over five years.

Discover theCUBE’s in-depth discussion.

2. Connectivity becomes a bottleneck for AI data centers.

Semtech Corp. supplies the chips in the optical transceivers and copper cables that connect thousands of AI processors in a single computer. According to Hong Hou, president and CEO of Semtech, any connection failure can force a job to restart, so connectivity now determines GPU availability and therefore revenue. Hyperscalers want more bandwidth with lower power consumption, leaving more of a data center’s fixed energy budget for computing power.

Check out theCUBE’s full interview.

3. Inference economics pushes chipmakers toward workload-aware racks.

Agents now consume tokens 24/7, making performance per dollar per watt the crucial metric of the inference era, rather than pure teraflops, explained Sunghyun Park, co-founder and CEO of Rebellions Inc. Moving beyond today’s rack-scale systems are racks tailored to whether a workload is latency-critical, throughput-hungry or power-constrained. Models change every few months, so Rebellions designs flexible hardware and sees the network as the next bottleneck as context windows grow.

Watch theCUBE’s full interview.

4. Maskless e-beam lithography aims to transition from Moore’s Law to advanced packaging.

“Moore’s Law is reaching its limits and advanced packaging that combines logic, memory and input/output chips is becoming the industry’s driving force,” noted David Lam, founder of Lam Research Corp. and CEO of Multibeam Corp. Multibeam has reduced an 80 centimeter high electron beam column to the size of a marking pen, allowing column arrays to write on a wafer in parallel. The resulting maskless lithography avoids the reticle limit that forces the assembly of large quantum and AI chips.

Check out theCUBE’s full coverage.

5. CXL memory pooling takes on new life when the memory wall persists.

The memory wall may not be solved for another five years, said Greg Ernst, chief business officer of ScaleFlux Inc., who recently joined Intel Corp. came. Even an 8 billion parameter model that serves many users requires far more memory than high-bandwidth memory can accommodate. Compute Express Link or CXL allows multiple servers to share a storage pool. Hyperscalers reusing older DDR4 modules in these pools are fueling interest in the technology.

Don’t miss the full interview on theCUBE.

6. In-memory computing targets memory shortage.

According to Faris Sbahi, co-founder and CEO of Normal Computing Corp., it takes three to four years for new memory factories to come online, so most memory from top vendors will be sold out by 2028. The startup is moving away from high-bandwidth memory toward standard LPDDR5. It also performs some inference calculations directly in memory to reduce cost and energy consumption.

Hear the full story on theCUBE.

7. Physics-based computing aims to deliver human intelligence in a single rack.

“Unconventional AI Inc. is building a fully integrated machine that uses the time-varying physics of the real world to calculate, just as a rolling ball inherently knows its next position,” shared Naveen Rao, co-founder and CEO of Unconventional AI. The company’s first system targets human intelligence with about 10,000 watts in a single rack, compared to 20 watts in the brain. Rao wants to further reduce that number until it surpasses biology, thereby reducing the cost of intelligence enough to open up new form factors.

Follow the full conversation on theCUBE.

8. Optics replace copper as AI clusters grow beyond electrical connections.

Power limits are forcing operators to connect multiple smaller data centers with fiber, and converting electrical connections within the data center to optical connections is the next growth driver, emphasized Michael Hurlston, president and CEO of Lumentum Holdings Inc. The company has grown fourfold in just over a year, but is about 70% below demand and is essentially booked through 2029. Indium phosphide laser chips are a key obstacle, so Lumentum is investing in factories and packaging as if the expansion would last another four to five years.

Watch theCUBE’s full session.

9. Energy systems compete to keep up with AI’s one-year hardware cycle.

The power infrastructure has historically evolved in cycles of six to ten years, while the AI ​​hardware is replaced approximately every year. Melton Chang, president of the Power Systems Division at Schneider Electric SE, described this mismatch as the industry’s biggest challenge. AI loads can fluctuate from 30% to 150% in milliseconds. These fluctuations threaten voltage and frequency stability and cause resonances that shorten the life of on-site generators. Schneider uses digital twins, simulations and real-time data to anticipate these fluctuations.

Don’t miss the entire section on theCUBE.

10. Self-verifying AI agents accelerate chip design without sacrificing trust.

According to Ankur Gupta, head of EDA IC and systems at Siemens Digital Industries Software, custom silicon can no longer take 18 to 24 months when models change every three to six months. However, designers still need deterministic, trustworthy results from electronic design automation tools. Siemens runs its design engines on GPUs from Nvidia Corp. for a 10x to 12x speed increase. AI agents are also used to automate the setup and debug phases of the design.

Check out theCUBE’s full interview.

11. d-Matrix scales memory-centric XPUs from chip to rack for inference.

According to Sid Sheth, founder and CEO of d-Matrix, d-Matrix Corp. quickly transitioned from accelerator cards to complete rack systems based on its Corsair XPU. By stacking memory directly on the processor – rather than placing it on an interposer alongside processing power – the architecture unlocks both capacity and bandwidth, two of the key bottlenecks in AI inference. The company accelerated this transformation by integrating with Nvidia’s AI factory ecosystem to avoid rebuilding power, cooling, networking and software.

You can find the full interview on theCUBE.

Photo: SiliconANGLE