Stop Wasting GPU Cycles: Multi-Model Serving with SGLang

When you run multiple inference servers on a single GPU node (for instance, using standard Hugging Face transformers or even multiple instances of vLLM), each process reserves its own block of VRAM for model weights and KV caching.

Because these processes do not communicate, memory becomes fragmented. One model might be sitting idle while hoarding 20GB of reserved VRAM, while another model is crashing due to an OOM error because it needs 2GB more to process a massive context window. This isolation destroys your server's ROI. You are paying for 100% of the GPU, but utilizing perhaps 40% of its potential efficiency.

SGLang (Structured Generation Language) was engineered to solve the exact bottlenecks found in traditional inference servers. While vLLM introduced PagedAttention to reduce memory waste within a single model's generation, SGLang introduces RadixAttention.

RadixAttention automatically maintains an LRU (Least Recently Used) cache of the KV cache using a radix tree structure. This means if multiple users (or agents) are prompting your models with similar system prompts, few-shot examples, or context history, SGLang instantly reuses the cached attention states. It significantly reduces memory usage and massively accelerates TTFT. When paired with its lightweight router, SGLang allows you to multiplex queries across multiple models with minimal VRAM overhead.

To achieve the throughput we are aiming for, you cannot rely on shared VPS instances or hypervisor-throttled cloud instances. You need direct, unshared access to the PCIe lanes connecting your CPU, RAM, and GPUs.

For optimal multi-model serving, we recommend using iDatam dedicated servers, configured with at least:

• OS: Ubuntu 24.04 LTS (or 22.04 LTS).

• GPU: Minimum 1x NVIDIA RTX 3090/4090 (24GB) for smaller models, or multi-A100/H100 setups for enterprise workloads.

• Storage: PCIe Gen 4 or Gen 5 NVMe SSDs (Crucial for loading massive .safetensors files into VRAM quickly).

• RAM: At least 2x your total GPU VRAM (e.g., if you have 48GB VRAM, ensure 96GB+ of System RAM).

We will deploy SGLang using Docker. This prevents Python dependency hell and ensures our environment exactly matches the optimized CUDA runtimes provided by the SGLang developers.

First, update your bare metal server and install the fundamental dependencies:

sudo apt update && sudo apt upgrade -y
sudo apt install -y curl git jq build-essential
                                

Next, install Docker and the NVIDIA Container Toolkit. This toolkit bridges your physical GPUs to your Docker containers.

# Add Docker's official GPG key and repository
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io

# Install NVIDIA Container Toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
                                

Verify that Docker can see your GPUs:

sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
                                

If you see your GPU listed in the output table, your infrastructure is ready.

To serve models, we need to download them. We will create a local directory on our high-speed NVMe drive to store the Hugging Face models so that the Docker container can access them without re-downloading them every time it restarts.

mkdir -p /mnt/nvme/huggingface_cache
export HF_HOME=/mnt/nvme/huggingface_cache
                                

Now, let's spin up the SGLang container. For this tutorial, we will use meta-llama/Meta-Llama-3-8B-Instruct. Note: You will need to pass your Hugging Face token if you are using gated models.

# Replace YOUR_HF_TOKEN with your actual token
export HF_TOKEN="hf_your_token_here"

sudo docker run -d \
  --name sglang-server \
  --gpus all \
  --shm-size 16g \
  -p 30000:30000 \
  -v /mnt/nvme/huggingface_cache:/root/.cache/huggingface \
  -e HF_TOKEN=$HF_TOKEN \
  lmsysorg/sglang:latest \
  python3 -m sglang.launch_server \
  --model-path meta-llama/Meta-Llama-3-8B-Instruct \
  --port 30000 \
  --host 0.0.0.0
                                

Understanding the flags:

• --shm-size 16g: Shared memory allocation. LLM inference requires a large shared memory pool for tensor operations. Never leave this at the Docker default (64MB) or your server will crash.

• -p 30000:30000: Maps the internal SGLang API port to your server's external port.

• --model-path: The Hugging Face repo ID.

Check the logs to ensure the model loaded successfully into VRAM:

sudo docker logs -f sglang-server
                                

SGLang truly shines when you need to run complex logic, structured JSON generation, or multi-model queries. Let's write a Python script on our host machine to interact with the SGLang server.

First, install the SGLang client library locally:

pip install sglang
                                

Create a file named inference_test.py:

import sglang as sgl

# Connect to the SGLang server we just deployed
sgl.set_default_backend(sgl.RuntimeEndpoint("http://localhost:30000"))

@sgl.function
def text_summarizer(s, text):
    s += sgl.system("You are an expert summarizer. Be concise.")
    s += sgl.user(f"Summarize this text in 3 bullet points: {text}")
    s += sgl.assistant(sgl.gen("summary", max_tokens=150))

text_to_summarize = """
Large Language Models (LLMs) have revolutionized artificial intelligence. 
However, deploying them in production remains a challenge due to high hardware 
costs and memory bottlenecks. Frameworks like SGLang utilize advanced techniques 
like RadixAttention to manage the Key-Value (KV) cache more efficiently, 
allowing servers to handle higher concurrency and multiple models without 
running out of memory.
"""

# Execute the function
state = text_summarizer.run(text=text_to_summarize)

print("Output:")
print(state["summary"])
                                

Run the script:

python3 inference_test.py
                                

Because of RadixAttention, if you run this script multiple times with the exact same system prompt, SGLang will not recompute the attention weights for the system prompt. It retrieves them directly from the radix tree, drastically reducing latency for repetitive API calls.

To squeeze every drop of performance out of your bare-metal setup, you need to tune the KV cache and memory allocation. By default, inference engines are conservative with memory.

If you are running a dedicated GPU server solely for inference, you can instruct SGLang to utilize a larger percentage of your VRAM for the KV cache. This is done by appending the --mem-fraction-static flag during server launch.

# Stop the current container
sudo docker stop sglang-server && sudo docker rm sglang-server

# Relaunch with aggressive memory settings
sudo docker run -d \
  --name sglang-server \
  --gpus all \
  --shm-size 16g \
  -p 30000:30000 \
  -v /mnt/nvme/huggingface_cache:/root/.cache/huggingface \
  -e HF_TOKEN=$HF_TOKEN \
  lmsysorg/sglang:latest \
  python3 -m sglang.launch_server \
  --model-path meta-llama/Meta-Llama-3-8B-Instruct \
  --port 30000 \
  --host 0.0.0.0 \
  --mem-fraction-static 0.85
                                

By setting --mem-fraction-static 0.85, we explicitly tell SGLang to dedicate 85% of the total GPU VRAM to the model weights and the KV cache tree. This prevents random OOM crashes during sudden spikes in concurrent user requests, ensuring the server rejects requests gracefully rather than crashing the Linux kernel's OOM killer.