NVMe-oF: TCP vs RDMA vs Fibre Channel — Which Transport for Your Storage in 2026?
NVMe over Fabrics decoded: when TCP beats RDMA, why FC-NVMe still wins in finance, latency comparison data, and what hardware you actually need to deploy each.
NVMe-oF: The Real Decision Behind Modern Storage Networks
NVMe over Fabrics (NVMe-oF) is the protocol that lets you put NVMe SSDs behind a network instead of inside the server — getting near-local latency with the flexibility of a SAN. By 2026 it has matured to the point where the question is no longer "should we use NVMe-oF" but "which transport: TCP, RDMA, or Fibre Channel?"
This guide compares the three real-world transports — what they do, what hardware they need, what they cost, and which workloads each one actually wins.
Why NVMe-oF Exists
Traditional iSCSI was designed when SATA HDDs were 7,200 RPM and IOPS were measured in hundreds. NVMe SSDs deliver 1.5 million IOPS and sub-100µs latency. Putting them behind iSCSI is like running a Bugatti on a dirt road — the bottleneck is the protocol, not the drive.
NVMe-oF extends the native NVMe command set over a network, keeping the parallelism (64K queues × 64K commands deep) and the low protocol overhead. The result: storage networks with 20–50µs added latency vs ~500µs for iSCSI.
The Three Transports — Side by Side
| Feature | NVMe/TCP | NVMe/RDMA (RoCE v2) | NVMe/FC (FC-NVMe) |
|---|---|---|---|
| Network | Standard Ethernet | Lossless Ethernet (DCB/PFC) | Fibre Channel fabric |
| Added latency | ~50–100µs | ~10–30µs | ~20–40µs |
| NIC required | Any 25/100GbE | RoCE-capable (Mellanox CX-6, Broadcom BCM57508) | FC HBA (Emulex LPe35002, QLogic 2872) |
| Switch required | Any 25/100GbE | Lossless DCB-capable | FC switch (Cisco MDS, Brocade) |
| Cost (per port) | $200–500 | $800–1,500 | $1,500–3,500 |
| Complexity | Low | High (DCB, PFC, ECN tuning) | Medium (zoning) |
| Best for | General-purpose, mixed environments | Hyper-low-latency AI/ML, hyperscale | Banking, healthcare, mission-critical |
When NVMe/TCP Wins
NVMe/TCP runs on any 25/100GbE network without lossless requirements. The trade-off is ~50–100µs extra latency vs RDMA. For 95% of workloads, that's invisible.
Choose NVMe/TCP when:
- You already have a converged 25/100GbE network
- Workload tolerates 100µs storage latency (databases up to ~50k IOPS, VDI, general virtualization)
- You don't want to operate a separate fabric
- You need to span across L3 networks (TCP routes; RDMA struggles)
Hardware you need:
- Any modern 25GbE NIC (Intel E810, Broadcom 57414, Mellanox ConnectX-5)
- Any 25/100GbE switch (Cisco Nexus 9300, Aruba 8320, Dell S5232F)
- VMware vSphere 7.0u2+, Linux kernel 5.0+, Windows Server 2022
NVMe/TCP is what 80% of new deployments use. It's "good enough" for everything except the most latency-sensitive workloads.
When RDMA (RoCE v2) Wins
NVMe over RDMA cuts another 60–80µs off TCP. For a hyperscale Elasticsearch cluster doing 10M IOPS against a shared storage backend, that latency difference compounds into measurable revenue.
Choose NVMe/RDMA when:
- AI/ML training feeding GPU clusters (H100, B200, MI300)
- High-frequency trading workloads
- Real-time analytics with sub-100µs SLAs
- VMware vSphere with vSAN ESA at scale
- You can afford to operate a lossless network (DCB, PFC, no oversubscription)
Hardware you need:
- RoCE-capable NICs: NVIDIA ConnectX-6/7 ($600–1,200), Broadcom BCM57508 ($800)
- DCB-capable switches: Mellanox SN3700, Cisco Nexus 9336C, Arista 7050X3
- Lossless network tuning (PFC priority queues, ECN marking, oversubscription <2:1)
The complexity is real. We've seen RoCE deployments fail because of one untuned switch buffer. Budget for a network engineer who has done this before.
When Fibre Channel (FC-NVMe) Still Wins
FC was supposed to die in 2010. In 2026 it's still shipping at record numbers — because for some workloads, the FC zoning model, deterministic latency, and 20-year operational maturity beat anything Ethernet offers.
Choose FC-NVMe when:
- You already run an FC SAN (don't rip it out — extend it)
- Financial services, healthcare, government — anywhere an outage triggers a compliance event
- You need true storage isolation from the data network
- Existing storage array supports FC-NVMe (NetApp ASA, Pure FlashArray, IBM FlashSystem)
Hardware you need:
- 32G/64G FC HBA: Emulex LPe35002, QLogic 2872 ($800–1,400)
- FC switch: Cisco MDS 9148T, Brocade G620 ($15k–40k)
- 32GFC SFP+ optics ($150–250 each)
FC's death has been exaggerated. Among enterprise storage refreshes in 2024–2025, 41% chose FC-NVMe over RoCE for primary tier-1 workloads.
Real Latency Numbers (Not Marketing)
Measured on a Dell PowerEdge R750 with PM1735a NVMe SSDs across each transport, 4KB random reads, queue depth 32:
| Transport | Median latency | P99 latency | Max IOPS |
|---|---|---|---|
| Local NVMe | 18µs | 35µs | 1.5M |
| NVMe/FC (32G) | 42µs | 78µs | 1.2M |
| NVMe/RDMA (100G) | 38µs | 65µs | 1.4M |
| NVMe/TCP (100G) | 95µs | 180µs | 950k |
| iSCSI (10G) | 480µs | 1.2ms | 180k |
Note: NVMe/RDMA can match or beat FC at the median but has wider P99 because of the underlying Ethernet. FC's deterministic latency is its real selling point.
Compatibility Matrix — Storage Arrays
| Array | NVMe/TCP | NVMe/RDMA | NVMe/FC |
|---|---|---|---|
| Pure FlashArray //X/XL | ✓ | ✓ | ✓ |
| NetApp AFF A-Series | ✓ | ✓ | ✓ |
| Dell PowerStore | ✓ | ✓ | ✓ |
| HPE Alletra MP | ✓ | – | ✓ |
| IBM FlashSystem 7300/9500 | – | – | ✓ |
| Hitachi VSP One Block | ✓ | – | ✓ |
| Pavilion HyperParallel | ✓ | ✓ | – |
What This Means for Your Next Refresh
If you're refreshing in 2026:
- Default to NVMe/TCP unless you have a specific reason not to. It's the cheapest, the simplest, and good enough for almost everything.
- Upgrade to NVMe/RDMA if you're feeding AI/ML GPU clusters or running database workloads with > 100k IOPS sustained.
- Stay on FC-NVMe if you already run an FC SAN and depend on its operational model. There's nothing wrong with FC in 2026.
- Don't deploy iSCSI for new storage. It's a 2005 protocol on a 2025 drive. Replace with NVMe/TCP at minimum.
FAQ
Q: Can I run NVMe/TCP and NVMe/RDMA on the same network? Yes. Modern arrays (Pure, NetApp, PowerStore) expose both. Use TCP for general workloads and RDMA for latency-sensitive ones over the same physical network.
Q: Does NVMe-oF require new server hardware? For NVMe/TCP, no — any 25/100GbE NIC and a modern OS works. For RDMA, you need RoCE-capable NICs. For FC-NVMe, you need 32G or 64G FC HBAs.
Q: How does NVMe-oF compare to local NVMe (NVMe-attached drives)? Local NVMe is still faster (~20µs vs 40-95µs). NVMe-oF trades a small latency penalty for the flexibility of pooled, shared, snapshotted storage.
Q: Is NVMe/TCP secure? Yes — Linux kernel 5.0+ supports NVMe/TCP over TLS (in-flight encryption). Most enterprise arrays support TLS 1.3.
Q: What's the maximum distance for NVMe-oF? NVMe/TCP: routable over WAN (latency-dependent). NVMe/RDMA: same data center, typically <2km. FC-NVMe: 10km with single-mode fiber, 100+ km with DWDM.
Next Steps
If you're speccing storage hardware for NVMe-oF in 2026, you'll want:
- NICs: Mellanox ConnectX-6 (RDMA) or Broadcom 57414 (TCP)
- HBAs: Emulex LPe35002-M2 or QLogic 2872 (FC-NVMe)
- Switches: Cisco Nexus 9300 (Ethernet) or Cisco MDS 9148T (FC)
- Cables/Optics: 25/100G AOC for short reach, SR4 optics for medium reach
Pro Disk Network stocks all three transport ecosystems with same-day US dispatch. If you need help speccing — send your storage array model and we'll cross-reference compatible host hardware.