NVMe-oF Over Ethernet — How to Build a High-Performance Storage Fabric (2026)

Published April 17, 2026 · 15 min read · By Pro Disk Network Team

Complete technical guide to NVMe over Fabrics (NVMe-oF) over Ethernet. Covers NVMe/TCP vs NVMe/RoCE, switch requirements, lossless Ethernet configuration, transceiver selection, and real-world deployment architectures.

Topics: NVMe-oF, NVMe over Fabrics, RoCE, Storage Networking, 25G, 100G

Why NVMe-oF Changes Storage Networking

NVMe (Non-Volatile Memory Express) was designed from scratch for flash storage. Where SCSI was engineered around spinning disk latency (milliseconds), NVMe targets flash latency (microseconds) with a queue depth of 65,535 simultaneous I/O commands per queue versus SCSI's maximum of 254.

The challenge: NVMe is a PCIe protocol. It was designed for local storage, not networked storage. NVMe over Fabrics (NVMe-oF) extends the NVMe command set across a network fabric — delivering local-flash-like latency and throughput to networked storage arrays.

When deployed over Ethernet, NVMe-oF makes your switching infrastructure a direct participant in storage performance. A misconfigured or undersized network fabric becomes the bottleneck between your compute and the all-flash array underneath it.

NVMe-oF Transport Options Over Ethernet

NVMe/TCP

• Transport: TCP/IP over standard Ethernet
• Latency overhead: 10 to 20 microseconds (TCP stack overhead)
• Network requirements: Any 10G/25G/100G Ethernet switch with low latency
• Host requirements: Standard NIC, software driver — no specialized hardware required
• Best for: General-purpose shared storage, secondary storage tiers, environments prioritizing simplicity over raw latency

NVMe/TCP runs on commodity Ethernet with no specialized hardware. A standard 25G SFP28 NIC and a 25G leaf switch are sufficient. The TCP overhead is acceptable for most workloads but rules out NVMe/TCP for latency-critical OLTP databases or real-time analytics.

NVMe/RoCE (RDMA over Converged Ethernet)

• Transport: RDMA (Remote Direct Memory Access) over lossless Ethernet
• Latency overhead: 2 to 5 microseconds (kernel bypass, RDMA)
• Network requirements: Lossless Ethernet — PFC and ECN mandatory
• Host requirements: RoCE-capable RDMA NIC (RNIC) — Mellanox ConnectX, Broadcom BCM57xxx
• Best for: High-performance databases, AI/ML storage, latency-critical primary storage

RoCE delivers RDMA over standard Ethernet hardware, bypassing the kernel entirely. The NIC directly reads from and writes to host memory without CPU involvement. Latencies approach InfiniBand performance at a fraction of the cost.

Critical requirement: RDMA cannot tolerate packet loss. A single dropped packet causes RDMA connection termination and retransmission. This requires lossless Ethernet throughout the storage fabric path.

Network Requirements for Lossless Ethernet (RoCE)

Lossless Ethernet for NVMe/RoCE requires three components configured on every switch port in the storage fabric path:

PFC (Priority Flow Control, 802.1Qbb): A per-priority pause mechanism. When a switch buffer fills for a specific traffic class, it sends a pause frame to the upstream device for that priority only — preventing packet drop without pausing all traffic classes. Configure on all storage fabric ports with storage traffic in a dedicated priority class (typically Priority 3 or 4).

ECN (Explicit Congestion Notification): Early congestion signaling via WRED (Weighted Random Early Detection) with ECN bit marking. This signals congestion to endpoints before buffers fill, allowing TCP or RDMA CC to back off before PFC pause frames are needed. Prevents PFC storms from propagating across multiple hops.

DCQCN (Data Center Quantized Congestion Notification): End-to-end congestion management algorithm implemented in the RNIC driver. Works with ECN signals from the network to rate-limit RDMA senders before buffer exhaustion. Configure in the MLNX-OFED or driver settings on RoCE-capable NICs.

Switch Configuration Example (Cisco NX-OS)

Configure PFC on storage priority class:

``` system qos service-policy type qos input ROCE_INPUT

policy-map type qos ROCE_INPUT class ROCE_CLASS set qos-group 3

class-map type qos ROCE_CLASS match cos 3

qos pause pfc-cos 3

interface Ethernet1/1 priority-flow-control mode on service-policy type qos input ROCE_INPUT ```

Then enable WRED with ECN on the storage queue so congestion is signaled before PFC kicks in.

Network Architecture for NVMe-oF

Single-Tier Switched Fabric

The most common small-to-medium scale architecture:

• Servers connect to 25G leaf switches via SFP28 DAC or fiber
• Storage arrays connect to the same or dedicated 25G/100G leaf switches
• One hop from server to storage
• Latency: 2 to 5 microseconds end-to-end including switch hop

Two-Tier Leaf-Spine Storage Fabric

For scale-out environments:

• Server leaf switches handle compute-to-network edge
• Storage leaf switches (or a dedicated storage tier) handle array connectivity
• Spine switches interconnect all leaves
• Storage traffic crosses two hops: server leaf to spine to storage leaf
• Latency: 4 to 10 microseconds end-to-end

Dedicated vs Converged Storage Network

Dedicated storage fabric: Separate physical switches for storage traffic. No bandwidth competition with application or management traffic. Simpler to configure lossless Ethernet without risking impact on other traffic classes. Recommended for RoCE deployments.

Converged fabric: Storage traffic shares switches with application traffic. Requires careful QoS — storage traffic in a PFC-enabled priority class, other traffic in drop-eligible classes. More complex but reduces switch count. Acceptable for NVMe/TCP.

Switch Selection for NVMe-oF

Transceiver and Cable Selection

Expected Performance Targets

With a correctly configured NVMe-oF fabric:

Real-World Deployment Examples

Financial services primary storage: A trading firm deploys 12 all-flash NVMe arrays connected to dedicated Arista 7050CX3 storage switches via 100G QSFP28 DAC cables. RoCE v2 with PFC and DCQCN configured throughout. End-to-end storage latency: 18 microseconds. Database query response time reduced by 40 percent compared to the previous iSCSI over 10G environment.

Media and entertainment rendering: A visual effects studio connects 48 DCC workstations to a shared NVMe-oF storage cluster via NVMe/TCP over 25G SFP28 infrastructure. Aggregate read throughput: 2.4Tbps to the storage cluster during peak rendering. No specialized RDMA hardware required — standard SFP28 NICs and a 25G managed leaf switch.

Healthcare PACS storage: A hospital radiology department deploys a two-tier NVMe-oF fabric — 25G server leaf for workstations, 100G storage leaf for the all-flash PACS array. NVMe/TCP used for simplicity. Radiologists receive DICOM images in under 200 milliseconds for any study in the 4-petabyte active archive.

Key Takeaway: NVMe/TCP is the right starting point for most organizations — standard hardware, no special configuration, meaningful latency improvement over iSCSI. Upgrade to NVMe/RoCE when your workloads require sub-50 microsecond storage latency and your team has the operational capability to maintain lossless Ethernet configurations.