ACL Digital
5 Minutes read
Why Connectivity Is the Real Bottleneck in Next-Gen AI Factories?
For a long time, the immediate challenge in hosting and updating artificial intelligence ecosystem technologies was securing raw compute power. Once that hurdle was cleared, memory limitations troubled the industry and raised new questions for AI deployment. Today, as we move ahead to build true AI Factories, connectivity has emerged as the major bottleneck. As Matt Murphy, CEO of Marvell Technology, highlighted during his keynote at Computex 2026, building massive AI models isn’t just a compute problem anymore, it’s a networking problem.
When we talk about an “AI Factory,” we are transitioning from traditional data centers (which host independent applications) to a giant, interconnected supercomputer where thousands of GPUs must act as a single brain. This shift makes high-speed, near-zero-latency networks an absolute necessity for anyone looking to play in the AI Factory space. This is being reflected across the industry, where leaders such as NVIDIA and frontier AI labs like OpenAI have emphasized that scaling AI systems is no longer just about compute density, but about how efficiently distributed systems can communicate at scale.
This shift is also being driven by the explosive growth of modern AI workloads, including large language models (LLMs), generative AI, and real-time inference systems. These applications require massive parallelism and continuous synchronization. As organizations race to reduce training times and optimize cost per token, AI networking infrastructure is no longer a secondary consideration but is a primary lever for competitive advantage in building scalable, production-ready AI systems.
Why is Connectivity Important for AI Infrastructure?
In a traditional data center, applications run independently. However, AI infrastructure is heavily dominated by East-West traffic (data moving horizontally between servers and GPUs inside the data center).
During Large Language Model (LLM) training, billions of parameters must be constantly shared across GPUs, accounting for over 90% of the total network traffic. Because AI models are too large to fit onto a single GPU, they are split across thousands of nodes using distributed training.
Here is exactly where the network becomes critical:
- The Process: Each GPU processes a chunk of data, calculates its gradients (updates), and must share those updates with every other GPU before moving to the next step.
- All-Reduce: This is a collective communication pattern where all GPUs sum up their gradients so everyone has the exact same, updated model parameters.
- The Catch (Tail Latency): If a single network link slows down, every single GPU sits idle waiting for it to finish. This is known as the “tail latency” problem—your entire multi-million-dollar AI cluster only moves as fast as its slowest network link.
Put simply, GPU costs are high, and network delays prove extremely expensive. A few microseconds of delay can leave thousands of GPUs idle, which is why AI networking is now considered just as vital as the GPU infrastructure itself.
Maximizing Efficiency and Time-to-Market
In an AI Factory, the unit of production is no longer a physical item, but a “token” (the building block of AI reasoning, text, images, or code). Running AI on traditional, unoptimized cloud infrastructure becomes financially unsustainable due to high token costs. An AI Factory uses tightly co-designed hardware and backend networks to maximize hardware utilization, drastically reducing per-token cost and energy to make mass AI deployment profitable.
Furthermore, it solves the time-to-market dilemma. If a company takes 6 months to train in a proprietary, domain-specific model, that model might be obsolete by the time it launches. Because AI Factories eliminate the network bottlenecks that leave GPUs sitting idle, they can compress a 3-month training workload into just a few days.
This is where emerging concepts like AI-driven infrastructure optimization come into play. By leveraging intelligent scheduling, workload-aware routing, and predictive congestion management, next-generation AI factories are beginning to use AI itself to optimize network performance, further blurring the line between compute and connectivity.
Introducing Network Fabric for AI Factories
To handle massive, simultaneous data exchanges without creating choke points, AI data centers utilize specialized topologies. Traditional networks are tree-shaped and narrow at the top, which inherently causes congestion. Instead, AI networks employ a Non-blocking Fat-Tree (or Clos) topology.
As you move “up” the network layers from leaf switches to spine switches, the available bandwidth increases (the “branches” get fatter). This ensures that any GPU can communicate with any other GPU at full speed simultaneously, without creating bottlenecks.
Key Benefits of Clos Topology
- Multiple paths available
- Dynamic load balancing
- High resilience
- Near non-blocking bandwidth
This architecture seamlessly scales across thousands, tens of thousands, or even hundreds of thousands of GPUs.
GPU Interconnects: Inside vs. Across Racks
Data transmission within an AI Factory changes based on physical distance:
- Inside the Rack (Ultra-Fast): GPUs within the same server or rack communicate over proprietary, ultra-high-bandwidth interconnects like NVIDIA NVLink or AMD Infinity Fabric. These operate at blazing speeds measured in terabytes per second.
- Across Racks (Scale-Out): When a signal must leave the physical rack to talk to GPUs elsewhere, it transitions to the standard backend network fabric (InfiniBand or Ethernet) via Network Interface Cards (NICs) like PCIe Gen5 or specialized SuperNICs.
Concept of The Frontend vs. the Backend Network
An AI Factory splits its operations into two completely isolated physical networks to prevent traffic interference: the Frontend Network and the Backend Network.
Historically, the backend network required NVIDIA’s proprietary InfiniBand. Today, Ethernet-based “AI Fabrics” are being engineered to take standard, open Ethernet hardware and rewrite the software protocols to provide lossless, deterministic, and predictable throughput at a massive scale.
Key Networking Technologies
The debate over which technology should rule the backend AI network comes down to three major players:
| Feature / Tech | InfiniBand | RoCEv2 (RDMA over Converged Ethernet) | Ultra Ethernet Consortium (UEC) |
| What It Is | A purpose-built, lossless networking architecture designed for High-Performance Computing (HPC). | A protocol that allows RDMA to run over standard, everyday Ethernet networks. | A new, open-industrial-standard redesign of Ethernet specifically for AI workloads. |
| Pros |
|
|
|
| Cons |
|
|
|
| Current Status | The gold standard for massive LLM training clusters today. | Widely deployed by hyperscalers seeking cost-efficiency. | Backed by AMD, Intel, Meta, and Broadcom, the future challenger to InfiniBand. |
Business Cases
The operational model relies heavily on high infrastructure utilization, divided across providers and a diverse customer base:
For Hyperscalers, the business model is simple: Build the AI Factory -> Offer AI as a Service -> Customers consume GPU resources. If the network lags, their billable assets sit idle.
Conclusion
Ultimately, the success of large-scale AI training is no longer determined by GPU compute performance alone. In modern AI factories, the network fabric has evolved into a first-class compute resource. A cluster packed with the world’s fastest GPUs can easily underperform compared to a cluster with slightly slower chips connected by a well-designed, non-blocking, low-latency network fabric.
Without reliable, ultra-low-latency connectivity, even the largest investments in advanced silicon lose much of their value. This fundamental truth explains why networking companies like NVIDIA, Cisco, Arista Networks, Juniper Networks, and Broadcom have shifted from supporting roles to becoming critical, central players in the AI infrastructure landscape. High-performance connectivity isn’t just an administrative necessity; it is the vital circulatory system that allows an AI Factory to think, scale, and generate value.
Looking ahead, the evolution of AI factories will increasingly depend on how intelligent networks can scale alongside compute. With the rise of Ethernet-based AI fabrics, software-defined networking, and autonomous traffic optimization, connectivity is set to become not just faster, but smarter. In this next phase of AI infrastructure, the winners will be those who treat networking not as a constraint, but as a strategic enabler of innovation, efficiency, and real-time intelligence.
This shift is further reinforced by the growing focus of companies such as NVIDIA, Cisco, and Broadcom, all of which are investing heavily in AI-optimized networking to support the next generation of large-scale AI deployments. In our next article, we will take a closer look at the actual players building these AI Factories and dive into the emerging networking innovations on the horizon that will enable the world to move even faster in AI innovation.
Frequently Asked Questions (FAQs)
1. What is an AI Factory in simple terms?
An AI Factory is a specialized data center where compute, storage, and networking are tightly integrated to efficiently train and run large-scale AI models.
2. Why is connectivity a bottleneck in AI systems?
Because AI workloads require constant data exchange between GPUs, even small network delays can cause large-scale inefficiencies and idle compute resources.
3. What is East-West traffic in AI infrastructure?
It refers to data movement within the data center, especially between GPUs and servers, which dominate AI model training workloads.
4. How does network latency affect AI training?
High latency slows down synchronization between GPUs, increasing training time and cost while reducing overall system efficiency.
5. What networking technologies are used in AI factories?
Technologies such as InfiniBand, RoCEv2, and emerging Ultra Ethernet standards enable high-speed, low-latency communication.
Related Insights
