Networking & Interconnect
Layer 04 of seven · Networking & Interconnect
- Why it matters
- Idle accelerators are the most expensive thing in a data centre. Interconnect determines how much of the compute a buyer has paid for actually gets used, which makes it a first order economic question rather than a technical footnote.
- The bottleneck
- Optical component supply and the physics of moving high data rates over distance. Copper stops working sooner than people expect, and optics is where much of the incremental spend lands.
- Who captures value?
- Suppliers whose products are designed into a platform, and those holding hard engineering in optics and high speed signalling. Standardisation tends to erode pricing over time.
- What could change?
- Ethernet based fabrics maturing into a full alternative to proprietary interconnect would widen the supplier list and change pricing across the layer. Co-packaged optics, which places the optical engine next to the switch silicon, would change the supply chain again.
Training a large model means tens of thousands of processors behaving as one machine. Networking can become the bottleneck even when there is plenty of compute available.
A cluster is only as fast as its slowest exchange. During training, accelerators must repeatedly share intermediate results, and every one of them waits for the last to arrive. Add enough machines and the network, not the silicon, sets the pace. The engineering runs across four distances: GPU to GPU inside a server, server to server inside a rack, rack to rack across a hall or campus, and data centre to data centre between regions. Each distance has its own technology and its own suppliers.
Tap or hover a box to see what it does
GPU to GPU inside a server
The shortest and fastest hop. NVIDIA's NVLink and NVSwitch connect accelerators inside a server so that several GPUs behave as one larger device, with far more bandwidth than a standard network link could carry.
NVIDIA
Designs the GPUs and accelerated computing platforms used for much of modern AI training and inference, and sells the networking that ties them together.
The hardware is only half of it. CUDA, the software libraries built on top of it, and a very large developer base make NVIDIA systems the default choice for teams who want to ship rather than port.
Custom accelerators built by its own largest customers, a stronger AMD, revenue concentrated in a handful of buyers, export restrictions, and model architectures that shift the balance of demand.
Server to server and rack to rack
Across a rack and a hall, clusters run on InfiniBand or high speed Ethernet. NVIDIA supplies InfiniBand and its Spectrum-X Ethernet platform following the Mellanox acquisition, while Arista and Broadcom drive the open Ethernet alternative and Credo supplies the high speed connectivity between them.
NVIDIA
Designs the GPUs and accelerated computing platforms used for much of modern AI training and inference, and sells the networking that ties them together.
The hardware is only half of it. CUDA, the software libraries built on top of it, and a very large developer base make NVIDIA systems the default choice for teams who want to ship rather than port.
Custom accelerators built by its own largest customers, a stronger AMD, revenue concentrated in a handful of buyers, export restrictions, and model architectures that shift the balance of demand.
Arista Networks
Builds high speed Ethernet switching and the network operating system used in large data centres and AI clusters.
Ethernet is the incumbent everywhere else in computing, and the industry is working hard to make it a first class option for AI clusters. Arista is the clearest listed way to hold that view.
A few very large customers drive much of the revenue, and merchant switching silicon lets others compete on price.
Broadcom
Co-designs custom AI accelerators for hyperscalers and supplies much of the switching and connectivity silicon inside data centres.
It sits on both sides of the custom silicon story. When a cloud provider builds its own chip to reduce dependence on merchant GPUs, Broadcom is often the partner making that possible.
Custom programmes are concentrated in a few customers who can change direction, and the software business carries its own integration risk.
Credo Technology
Makes high speed connectivity products including active electrical cables and retimers used inside racks.
As data rates climb, plain copper stops behaving. Active cabling is a small line item that quietly determines whether a rack works at the speed it was sold at.
Small, concentrated and exposed to design cycle timing at a few customers.
Data centre to data centre
Beyond the hall, traffic moves over optics. Transceivers, lasers and photonic components carry campus and long haul links, and co-packaged optics is the direction of travel as data rates climb.
Coherent
Makes optical transceivers, lasers and photonic components used to move data between racks and buildings.
Optics is where a very large share of the incremental networking spend in AI clusters lands, because copper runs out of reach quickly at these speeds.
Historically a competitive, price eroding market with rapid generational transitions.
Lumentum
Supplies photonics including lasers and transceivers for data centre interconnect.
A second established supplier in a market where hyperscalers actively want more than one source.
Legacy telecom exposure and the same pricing dynamics that affect the whole optics sector.
Marvell Technology
Designs data infrastructure silicon including optical interconnect, custom compute and storage controllers.
Moving a signal off a chip, across a rack and between buildings is its own engineering discipline. Marvell sells into the parts of that path most people never think about.
Custom programme timing is lumpy and dependent on a handful of hyperscaler decisions.
Lightwave Logic
A pre-revenue research company developing polymer based electro-optic materials for high speed optical modulation.
If the materials work at production scale they could improve the speed and power profile of optical links. That remains a research outcome rather than a shipping product.
No meaningful revenue, ongoing funding needs, and a long path from demonstration to qualified supply.
Region to region and out to users
Between regions, and then to the people and devices actually making requests. Latency at this distance shapes where inference is served from rather than where models are trained.
Cloudflare
Operates a global network providing content delivery, DDoS protection, zero trust services and edge compute.
It sits in front of a large share of web traffic, which gives it both a security vantage point and a natural place to put inference close to users.
Monetising a wide free tier is a long game, and the developer platform competes with the hyperscalers.
How to think about the economics
- GrowthVery high
How quickly demand in this part of the ecosystem is expanding.
- Capital intensityModerate
How much money has to be spent up front before revenue arrives.
- Competitive moatModerate
How difficult it is for a credible new entrant to take the business.
- Customer concentrationVery high
How much revenue depends on a small number of buyers.
- Disruption riskModerate
How exposed the layer is to a technical or commercial shift.
This is a framework for thinking about the economics of a layer, not a recommendation. An important AI company, a strategically advantaged company, an investable security and an attractively valued security are four different things.
Where value may accrue
Design wins, switching costs inside a platform, and hard physics. Once a fabric is chosen for a cluster generation it rarely changes mid-flight, which gives incumbents a useful few years.
Key terms in this layer
- Fibre optics
- Glass threads that carry data as pulses of light. Inside AI data centres and between them, fibre is what keeps thousands of GPUs in step.
- ExampleCoherent and Lumentum make the transceivers that turn electricity into light.
- Related termsNetwork switchData centre
- Network switch
- The hardware that shuttles data between machines. If the switch is slow, thousands of expensive GPUs sit idle waiting for data.
- ExampleArista builds the switches inside many AI data centres.
- Related termsFibre opticsGPUData centre
- InfiniBand
- A very low latency networking standard used to link accelerators inside AI clusters, supplied largely by NVIDIA since it acquired Mellanox. High speed Ethernet is the competing approach.
- ExampleLarge training clusters connect thousands of GPUs over InfiniBand.
- Related termsNetwork switchFibre opticsGPU
Related questions
Doesn't NVIDIA manufacture its own chips?
No, and this surprises a lot of people. NVIDIA designs the chips and outsources manufacturing, mostly to TSMC in Taiwan. It is a fabless company: engineers and intellectual property rather than factories. AMD and Apple work the same way. Physical manufacturing is extraordinarily capital intensive, and TSMC has spent decades becoming very difficult to replace at the leading edge.
What is the difference between a CPU and a GPU?
CPUs are optimised for flexible, general purpose and low latency computing. They handle branching, unpredictable work and the everyday business of running an operating system. GPUs contain many more, simpler processing units and are especially effective at performing large numbers of similar calculations in parallel. That parallel arithmetic is most of what training and inference consist of, which is why a design originally refined for rendering graphics turned out to suit AI so well.
Sources and further reading (3)+
- 1InfiniBand and Spectrum-X networking documentation · NVIDIA
- 2Investor relations and product documentation · Arista Networks
- 3Ultra Ethernet Consortium specification work · Ultra Ethernet Consortium
The AI ecosystem changes rapidly. Company positions, technologies and market data reflect information available at the date above.