AI Fabrics Need Different Optics — GPU Clusters Are Not Normal Data Centers
Your data center runs web applications. Your procurement team buys optics for web application traffic. Then someone installs a GPU cluster, and the optics team discovers that AI fabric traffic has nothing in common with what they have been buying.
A traditional data center has a front-end network: users hit load balancers, load balancers hit application servers, servers query databases. Traffic patterns are north-south. Flows are short. Packet loss at 0.1% is invisible to end users.
An AI training cluster has a back-end network: GPUs talk to GPUs. Traffic is east-west, all-to-all. A single training job can push 400G sustained per GPU for hours. Packet loss at 0.001% causes a collective operation to stall, and every GPU in the job waits.
Flow duration Front-end: milliseconds | Back-end: hours to days
Utilization Front-end: 30–50% avg | Back-end: 80–95% sustained
Loss tolerance Front-end: 0.1% | Back-end: < 0.001%
Protocol Front-end: TCP | Back-end: RoCEv2 / InfiniBand (RDMA)
Latency target Front-end: < 5 ms | Back-end: < 5 µs
Failure impact Front-end: retransmit | Back-end: entire job checkpoint/restart
RDMA does not handle packet loss the way TCP does. A dropped packet in a RoCEv2 fabric triggers a go-back-N retransmission that stalls the entire collective. A 1,000-GPU training job that loses one packet on one link pauses all 1,000 GPUs until recovery completes.
Front-end optics optimize for cost per port. You buy the cheapest validated 400G-DR4 or FR4, and the network absorbs occasional link flaps without visible impact.
Back-end optics optimize for reliability and latency. A link flap during all-reduce causes a training checkpoint rollback. Depending on model size, that rollback costs 10 to 60 minutes of GPU time. At $2–3 per GPU-hour for H100s, a single link flap on a 1,000-GPU job costs $30–$50 in wasted compute. Five flaps per day across the fabric adds up fast.
Form factor OSFP (800G) or QSFP-DD (400G)
FEC latency Matters. KP4 adds ~100 ns. oFEC adds 5–15 µs.
Power per port 12–25W × thousands of ports = MW-scale problem
Connector type MPO-16 (SR8) or LC duplex (DR/FR)
Burn-in requirement 72–168 hours recommended before production
Spare ratio 10–15% (link flap cost justifies higher spare stock)
A GPU rack with 8× H100 or B200 systems draws 40–120 kW. The back-end network adds 200–800W per rack in optics alone. Multiply by a thousand racks.
Standard data center power distribution was designed for 8–15 kW per rack. AI clusters push 40–120 kW per rack. The optics contribution looks small in percentage terms but adds up to hundreds of kilowatts across the fabric.
Network switches ~200 × 51.2T (500W each) = 100 kW
Optics (400G pluggable) ~12,000 modules × 15W = 180 kW
Optics (800G pluggable) ~6,000 modules × 22W = 132 kW
Optics as % of total 2–3% of total power, 100% of link reliability
Cooling overhead 1.3× PUE factor on all of the above
Linear Pluggable Optics (LPO) cut module power to 3–5W by removing the DSP retimer. For AI fabrics where reach is short and the signal path is clean, LPO eliminates 10W per module. Across 12,000 modules, that saves 120 kW and its associated cooling load.
Front-end optics procurement runs quarterly. You order based on forecasted port growth, and a two-week delay has minimal business impact.
AI cluster optics procurement runs in lockstep with GPU delivery. The GPUs arrive, the fabric must be ready. A two-week optics delay means 1,000 GPUs sit idle. At $2/GPU-hour, two weeks of idle time on 1,000 GPUs costs $672,000.
Optics lead time (400G) 12–20 weeks
Optics lead time (800G) 20–30+ weeks (early production)
Order trigger Same day as GPU PO, not after rack install
Validation timeline 4–6 weeks before GPU arrival
Buffer stock 15–20% over planned port count
You order optics the same day you order GPUs. You validate them before the GPUs arrive. You stock 15–20% buffer because a single bad batch at 800G can delay an entire training cluster by the time it takes to get replacements. The team that buys optics and the team that buys GPUs need to share a spreadsheet. In practice, they rarely do.
This is also where optics stops being an abstract architecture topic and becomes a sourcing problem.
If the fabric is InfiniBand, or if the cluster is built around NVIDIA / Mellanox-compatible environments, the transceiver choice cannot be an afterthought. Compatibility, validation, lead time, spare stock, power draw and link stability all matter before the GPUs arrive, not after the first failed training run.
At FLEXOPTIX, we have been seeing this shift very clearly. AI-capable optics are not about putting the word "AI" on a product page. They are about supporting the actual fabrics people are building now: high-density, low-latency, high-utilization GPU clusters where one bad optical decision can hold back a very expensive machine.
For teams planning these deployments, the relevant starting points are here:
InfiniBand transceivers:
https://www.flexoptix.net/en/transceiver?fo_tra_protocols_category=InfiniBand
NVIDIA / Mellanox-compatible optics:
https://www.flexoptix.net/en/supported-vendors/index/name/Nvidia+(ex.+Mellanox)-compatible
The point is simple: if the GPU purchase is real, the optics plan has to be real on the same day.