Scaling AI Infrastructure: A Passive Physical Layer Perspective
As Artificial Intelligence (AI) and Machine Learning (ML) workloads rapidly grow, design focus usually centers on integrating active compute elements and networking hardware; like the graphics processing unit (GPU) and other accelerators, customized Application Specific Integrated Chips (ASICs), and high-speed switches. However, the actual choke point of modern AI performance frequently lies in the underlying physical interconnects, especially the passive infrastructure.
This is an acute problem because of the incongruity that fast-changing compute engines deployed for AI must interact with slower-changing and longer lasting physical infrastructure. Without ultra-low latency, high-density, low-loss passive cabling systems, even the most advanced (and expensive) compute clusters will suffer from synchronization delays and deleterious compute idle times.
AI network design is usually defined by the physical and logical wiring topology used to connect them; for instance, Spine & Leaf architecture. However, a higher level, simpler understanding can be gained by understanding the boundaries and semantics of how AI resources expand on the macro and micro scale. This is termed scaling dimension or domain. Within AI networks, the transmission of data is categorized into scaling domains traditionally called Scale-Up, Scale-Out, and Scale-Across. New scaling domains continue to be defined, like Scale-In and Scale-To/Above.
Here is a broad rather than in-depth discussion of these scaling domains from a passive infrastructure perspective, examining physical media challenges and structural shifts between AI Training and AI Inference architecture.
Scale-Up (Accelerator-to-Accelerator, single rack)
Scale-Up is associated with interconnecting AI related processors, like GPUs, TPUs, NPUs or CPUs, within a single server chassis, tray, rack or pod. The design objective is to create a single, unified domain using proprietary or open network fabrics.
Here the passive layer focuses on delivering short-reach, very high-density, and sub-nanosecond signal integrity. The physical connections must handle high channel bandwidths exceeding 224G per lane, aggregating to 800G or 1.6T bandwidth.
The principal physical challenges when interconnecting this type of network are thermal management, managing cable bend radii inside tight spaces, and extreme port density constraints on switch faceplates. Depending on the bandwidth and distance of travel within the rack, the passive media used may be copper or optical fiber.
Scale-Out (rack-to-rack or node-to-node)
Scale-Out connects individual server nodes across multiple racks into a dense parallel computing cluster where traffic flow is mostly East-West (server to server) and highly synchronised. This is the network topology used for centralised AI workload execution associated with AI training models. It is traditionally routed through InfiniBand or RoCEv2 (RDMA over Converged Ethernet) fabrics for the very low-latency and zero-packet-loss transport required for optimal synchronization and use of tens of thousands of compute cores.
In this case, the passive layer most often deployed is a structured, modular system of cabling and connectivity with a focus on providing sufficient rack-to-rack pathways for future scalability, high-density patch panels and preterminated fiber trunk cables.
The main physical infrastructure challenges here are associated with cable tray/pathway congestion. High-density AI racks demand many thousands of individual optical fibers per rack, which can physically overload standard overhead basket trays and lead to pathway exhaustion unless adequate advanced planning of cable pathways has been done.
Scale-Across (Campus & Data Center interconnect – DCI)
Scale-Across extends AI compute connectivity across multiple physical buildings, geographically separated data center halls or facilities, enabling the accelerators to operate as a single virtualized cluster (a single ‘brain’). The physical separation of AI compute across multiple buildings demands ultra-high fiber count cable, often Intermittently Bonded Ribbon (IBR) cable, and low-loss connectivity with Very Small Form Factor (VSFF) connectors housed in high-density patch panels.
The passive layer, in this case, focuses on relatively long optical fiber cable runs, underground duct pathways, and dense outside plant (OSP) and whitespace connectivity. Making available redundant cable paths to each DC hall is also a fundamental design requirement, significantly increasing volume requirements for high-fiber count DCI cables. In this domain we also see increasing interest in newer fiber technologies like Multi-Core Fiber (MCF) and Hollow Core Fiber (HCF) which offer space saving and low-latency advantages, respectively.
The main challenge is consequently duct space availability and also calculating latency constraints driven by light propagation limits travelling over distance.
Scale-To or Scale-Above (user/device-to-AI compute)
This is a relatively new term associated with the inevitable move of AI workloads from AI training to AI inference. While Scale-Up, Scale-Out and Scale-Across cover connectivity inside and between data centers, Scale-To/Above extends inference connectivity outward to the distributed edge, and across AI radio access networks (AI-RAN). Ultimately AI end-users or devices are connected to inference compute points over existing cloud infrastructure, cellular networks and satellite or space links.
The primary challenges with Scale-To/Above from a cabling infrastructure perspective are two-fold:
- Bridging the physical-to-wireless boundary; transitioning largely static ultra-high density optical fiber networks to dynamically aligned free-space optical laser and radio communications. Fiber cabling based networks are fixed, guided and deterministic. Often it is a combination of copper, fiber or a hybrid infrastructure that is used for connectivity.
- Integrating high-performance, ultra-low latency AI networks with existing cloud infrastructure as the inference compute capability needs to be located as close as possible to the user or device requesting the AI prompt.
Often remote edge compute is housed in small, hardened, and hermetically sealed enclosures or containerised facilities. Connectivity must be engineered to resist moisture, dust, and vibration over a 10-15 year operational lifespan with minimal technician support. Of course, as AI compute moves beyond terrestrial locations to space, a whole new set of environmental challenges are presented. (This will not be covered here).
Scale-In (Intra-chip connectivity)
The rapid shift towards AI is driving technology advancements at the integrated chip and silicon substrate level. AI workloads demand huge amounts of data to be moved between compute cores and memory with ultra-low latency; this creates a major bandwidth bottleneck. One way to overcome this is via the use of silicon photonics to transfer data between chips. Here photons move over optical paths instead of electrical signals moving over copper paths to enable the ultra-fast transfer of data with significantly less heat and power.
Usually, optical infrastructure instead of copper is deployed to overcome the distance/bandwidth barrier and heat/power limitations of copper connections across distances of meters through to many kilometers. However, because of the large data flows within AI processors, the same concept is now being applied to the tiny distances, often millimeters, that data travels within an integrated circuit.
This had led to a frenzy of research and development in adjacent technologies; near packaged optics (NPO) and co-packaged optics (CPO), novel material science to modulate light faster whilst running cooler, and automated sub-micron alignment to enable the mass-production of direct fiber-to-chip packaging.
By replacing electrical traces with light, chip designers can significantly reduce power consumption, reduce thermal strain and effectively eliminate the input-output bottleneck between processors and memory.
Summary
As the use of AI applications exponentially grows, network design has focused on AI training related architectures. Moving forward we should prioritize metro and edge inference related capacity, robust route diversity, and infrastructure that can grow without costly redesign or service disruption.
AI compute advances will continue to push physical limits, and passive infrastructure will also need to transition from a simple connectivity layer to an critical enabler of AI performance. Optimizing the physical layer across Scale-In, Scale-Up, Scale-Out, Scale-Across, and Scale-To/Above topologies, through the strategic application of copper, fiber and advanced fiber technology like Multi-Core Fiber and Hollow-Core fibers, ensures that next-generation AI workloads run at optimal capacity.
