Setup NVMe-oF (TCP) on Dedicated Servers to Crush AI Data Bottlenecks

2.1 Load Kernel Modules

First, load the necessary NVMe target and NVMe-TCP kernel modules.

sudo modprobe nvmet
sudo modprobe nvmet-tcp
                                

To make these modules persist across reboots, add them to /etc/modules:

echo "nvmet" | sudo tee -a /etc/modules
echo "nvmet-tcp" | sudo tee -a /etc/modules
                                

2.2 Identify the NVMe Drive

Find the block device name of the NVMe drive you want to share.

lsblk
                                

Let's assume the drive you want to share is /dev/nvme1n1. Warning: Ensure this drive does not contain your operating system, as we will be giving block-level access to the Initiator.

2.3 Install NVMe Configuration Tools

Install the nvmetcli tool, which provides a user-friendly way to configure the NVMe target subsystem.

sudo apt install nvmetcli -y
                                

2.4 Configure the Subsystem

We will use nvmetcli to create a configuration. You can do this interactively, but it is much safer and reproducible to use a JSON configuration file.

Create a file named nvmet-config.json:

{
  "ports": [
    {
      "addr": {
        "adrfam": "ipv4",
        "traddr": "10.0.0.10",
        "treq": "not specified",
        "trsvcid": "4420",
        "trtype": "tcp"
      },
      "portid": 1,
      "referrals": [],
      "subsystems": [
        "nqn.2026-08.com.idatam:storage.target01"
      ]
    }
  ],
  "hosts": [],
  "subsystems": [
    {
      "allowed_hosts": [],
      "attr": {
        "allow_any_host": "1",
        "serial": "IDATAM001",
        "version": "1.3"
      },
      "namespaces": [
        {
          "device": {
            "nguid": "00000000-0000-0000-0000-000000000000",
            "path": "/dev/nvme1n1",
            "uuid": "00000000-0000-0000-0000-000000000000"
          },
          "enable": 1,
          "nsid": 1
        }
      ],
      "subnqn": "nqn.2026-08.com.idatam:storage.target01"
    }
  ]
}
                                

Explanation of parameters:

• traddr: The private IP of your storage server (10.0.0.10).

• trsvcid: The default NVMe-oF TCP port is 4420.

• subnqn: The NVMe Qualified Name. This acts as the unique identifier for your storage subsystem.

• path: The block device we are sharing (/dev/nvme1n1).

• allow_any_host: Set to 1 for simplicity in this tutorial, but in production, you should restrict this to specific Initiator NQNs for security.

Apply the configuration:

sudo nvmetcli restore nvmet-config.json
                                

Verify that the target is listening on port 4420:

ss -tlnp | grep 4420
                                

3.1 Load Client Kernel Modules

Load the NVMe TCP client module:

sudo modprobe nvme-tcp
echo "nvme-tcp" | sudo tee -a /etc/modules
                                

3.2 Install NVMe-CLI

Install the standard nvme-cli package, which is used to discover and connect to NVMe-oF targets.

sudo apt install nvme-cli -y
                                

3.3 Discover the Target

Run a discovery command to ensure the Initiator can see the subsystem exported by the Target.

sudo nvme discover -t tcp -a 10.0.0.10 -s 4420
                                

You should see output similar to this:

Discovery Log Number of Records 1, Generation counter 2
=====Discovery Log Entry 0======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified
portid:  1
trsvcid: 4420
subnqn:  nqn.2026-08.com.idatam:storage.target01
traddr:  10.0.0.10
                                

3.4 Connect to the Target

Now, issue the connect command using the NQN we defined earlier:

sudo nvme connect -t tcp -n nqn.2026-08.com.idatam:storage.target01 -a 10.0.0.10 -s 4420
                                

If successful, the command will return silently. Verify the connection by checking your block devices:

lsblk
                                

You should now see a new NVMe drive (e.g., nvme1n1 or nvme2n1) listed on your Initiator server, even though the physical drive is inside the Target server. To view detailed information about the remote NVMe drive:

sudo nvme list
                                

4.1 Create a File System

Let's format the new remote drive (assuming it mapped to /dev/nvme1n1 on the Initiator) with the XFS file system, which is highly optimized for large files and high I/O workloads like AI datasets.

sudo mkfs.xfs -f /dev/nvme1n1
                                

4.2 Mount the Drive

Create a mount point and mount the drive:

sudo mkdir -p /mnt/ai_datasets
sudo mount /dev/nvme1n1 /mnt/ai_datasets
                                

Check that the drive is mounted and available:

df -h /mnt/ai_datasets
                                

4.3 Persistent Mounting (fstab)

To ensure the drive reconnects and mounts automatically after a reboot of the compute node, we need to configure NVMe auto-connect and fstab.

First, generate the NVMe connection configuration automatically:

sudo nvme discover -t tcp -a 10.0.0.10 -s 4420 -p
                                

Next, enable the nvme-tcp auto-connect systemd service:

sudo systemctl enable nvme-auth.service
sudo systemctl enable nvmf-autoconnect.service
                                

Finally, add the drive to your /etc/fstab. It is crucial to use the _netdev option so the system knows this is a network drive and waits for the network to initialize before attempting to mount it. Find the UUID of the drive using blkid /dev/nvme1n1 and add this line to /etc/fstab:

UUID=your-uuid-here /mnt/ai_datasets xfs defaults,_netdev,x-systemd.requires=network-online.target 0 0
                                

5.1 Benchmarking with FIO

Install fio to test the raw I/O performance of the mounted drive.

sudo apt install fio -y
                                

Run a sequential read test (common for loading AI checkpoints):

sudo fio --name=seqread --ioengine=libaio --iodepth=64 --rw=read --bs=1M --direct=1 --size=10G --numjobs=4 --runtime=60 --group_reporting --filename=/mnt/ai_datasets/testfile
                                

You should see throughput numbers closely matching the maximum bandwidth of your private network connection (e.g., ~1100 MB/s on a 10Gbps link).

5.2 Jumbo Frames (MTU 9000)

If you want to squeeze every last drop of performance out of NVMe over TCP, you must configure Jumbo Frames. Standard Ethernet frames have an MTU of 1500 bytes. By increasing this to 9000 bytes, you drastically reduce the packet header overhead and CPU interruptions on both the Target and Initiator servers.

To do this temporarily (test this before making it permanent):

sudo ip link set dev eth1 mtu 9000
                                

(Replace eth1 with the name of your private network interface on both servers).

After enabling Jumbo Frames on both servers (and ensuring your private network switch supports it), re-run the fio benchmark. You should see a noticeable drop in CPU utilization and an increase in total throughput, ensuring your GPUs are fed data as fast as they can process it.