The NUMA Bottleneck: Pinning Processes for Maximum RAM Speed on Bare Metal

When enterprises provision high-end dedicated servers featuring modern AMD EPYC or dual-socket Intel Xeon processors, they expect uncompromising performance. However, out of the box, standard Linux distributions can silently strangle the performance of databases, hypervisors, and AI workloads due to a hardware concept known as NUMA (Non-Uniform Memory Access).

Modern high-core servers do not treat all RAM equally. They divide their CPU cores and physical memory into separate zones called NUMA nodes. If an application (like Redis, PostgreSQL, or a virtual machine) is running on a CPU core in Node 0, but the Linux kernel allocates its memory from the RAM sticks physically attached to Node 1, the data must travel across a physical bridge on the motherboard (like Intel's UPI or AMD's Infinity Fabric).

This cross-node traffic introduces massive microsecond latency, silently crippling throughput. The solution is straightforward but often overlooked: explicitly pinning high-performance applications to specific CPU cores and their local memory banks using numactl and hwloc. In this tutorial, we will show you how to map your hardware and enforce strict NUMA policies for maximum speed.

What You'll Learn

Understanding the NUMA Node Bottleneck

In older SMP (Symmetric Multiprocessing) architectures, all CPUs accessed a single, shared pool of memory at the same speed. As core counts exploded, this shared bus became a massive traffic jam. Hardware manufacturers solved this with NUMA.

In a NUMA architecture, a processor and the memory slots closest to it form a "Node".

• Local Memory Access: When a CPU reads memory from its own node, it is extremely fast and high-bandwidth.

• Remote Memory Access: When a CPU needs to read memory from another node, the request must traverse the QPI/UPI (Intel) or Infinity Fabric (AMD) interconnect. This adds latency and reduces bandwidth.

By default, the Linux CPU scheduler moves processes around to balance the load. A Redis thread might start on Node 0, allocate 10GB of RAM on Node 0, and then get moved by the kernel to Node 1. Suddenly, every RAM access by Redis becomes a "remote" access. By pinning the process, we eliminate this scheduler jitter and guarantee 100% localized memory access.

Step 1: Installing hwloc and numactl

To fix the bottleneck, we need the right tools. We will use hwloc to visually and structurally map our hardware, and numactl to enforce the binding.

sudo apt update
sudo apt install numactl hwloc hwloc-nox -y
                                

sudo dnf install numactl hwloc -y
                                

Step 2: Analyzing Your Server's NUMA Topology

Before we pin a process, we must understand the physical layout of the server. Let's inspect the hardware.

lstopo-no-graphics
                                

numactl --hardware
                                

available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 64123 MB
node 0 free: 32000 MB
node 1 cpus: 8 9 10 11 12 13 14 15
node 1 size: 64511 MB
node 1 free: 41000 MB
node distances:
node   0   1
  0:  10  21
  1:  21  10
                                

Step 3: Pinning Processes to Specific NUMA Nodes

Now that we know which cores belong to which node, we can bind an application. The numactl command acts as a wrapper around the application you want to run.

numactl --cpunodebind=0 --membind=0 my_application --start
                                

numactl --physcpubind=2,3,4 --membind=0 redis-server /etc/redis/redis.conf
                                

Step 4: Automating NUMA Policies with Systemd

Running commands manually is great for testing, but in a production environment, services are managed by systemd. You must modify the service files to ensure your databases or web servers are pinned automatically on startup.

sudo systemctl edit redis-server.service
                                

[Service]
ExecStart=
ExecStart=/usr/bin/numactl --cpunodebind=0 --preferred=0 /usr/bin/redis-server /etc/redis/redis.conf
                                

sudo systemctl daemon-reload
sudo systemctl restart redis-server
                                

numastat -p $(pidof redis-server)
                                

Conclusion

By taking manual control over your CPU cache and memory allocations with numactl, you stop the Linux kernel from making inefficient scheduling guesses. Pinning your heavy workloads to specific NUMA nodes ensures that memory access remains local, latency remains low, and your bare-metal hardware performs exactly the way enterprise infrastructure should.