How to Deploy a Production-Ready LLM API (vLLM / Ollama) on a GPU Dedicated Server

To keep our AI environment clean and easily scalable, we will run our inference engine inside a Docker container. First, you need to ensure Docker can "see" your server's NVIDIA GPUs.

(Note: We assume you have already installed the proprietary NVIDIA drivers. If not, see our PyTorch setup guide first).

Install Docker:

sudo apt update
sudo apt install docker.io -y
sudo systemctl enable --now docker
                                

Next, configure the NVIDIA Container Toolkit repository and install it:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update
sudo apt install -y nvidia-container-toolkit
                                

Configure Docker to use the NVIDIA runtime and restart the service:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
                                

vLLM is the industry standard for production LLM serving because it utilizes PagedAttention to manage GPU memory, resulting in massively higher throughput than standard Hugging Face pipelines.

We will deploy a Llama-3 8B Instruct model. Since Llama-3 is a gated model, you will need a free Hugging Face token (hf_...) with access granted to the Llama-3 repository.

Run the following command to pull the vLLM container and start the API server on port 8000:

sudo docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 8000:8000 \
    --env "HUGGING_FACE_HUB_TOKEN=your_hugging_face_token_here" \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model meta-llama/Meta-Llama-3-8B-Instruct \
    --dtype bfloat16 \
    --max-model-len 8192
                                

What this command does:

• --gpus all: Gives the container access to all your NVIDIA H100s or A100s.

• -v ~/.cache/...: Maps your local storage so the model only downloads once.

• vllm/vllm-openai:latest: Uses the vLLM image that natively emulates the OpenAI API structure.

• --max-model-len: Sets the context window (adjust based on your available VRAM).

Once the model weights are loaded into VRAM (this takes a few minutes on the first run), your server will start listening on port 8000.

Because vLLM mimics the OpenAI API, you don't need to rewrite your application's frontend code. You simply change the base_url to point to your iDatam server.

Open a new terminal session and test your local API using a standard curl request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3-8B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a highly advanced AI coding assistant."},
      {"role": "user", "content": "Write a Python script to scrape a website."}
    ],
    "max_tokens": 512,
    "temperature": 0.7
  }'
                                

You will instantly receive a JSON response containing the model's generated text, running entirely on your own silicon.

If you don't need a high-throughput production API and just want to chat with a model in your terminal to test its capabilities, Ollama is the fastest alternative.

Install Ollama natively on your Linux server:

curl -fsSL https://ollama.com/install.sh | sh
                                

Once installed, simply run:

ollama run llama3
                                

Ollama will automatically download the quantized weights and drop you into a ChatGPT-style command-line interface.

By default, this API is exposed via HTTP on port 8000 without authentication. Before pointing your live applications to this server, you must secure it. We highly recommend installing Nginx as a reverse proxy to handle SSL/TLS encryption (HTTPS) and configuring an API Gateway (like Kong) or basic Nginx authentication to require an API key from your users.

Take Ownership of Your AI

You have now successfully built a self-hosted, scalable AI engine. By pairing open-source models with raw bare-metal hardware, you can offer your users intelligent features without sacrificing data sovereignty or worrying about runaway API costs.

To ensure your LLMs respond instantly—even under heavy concurrent user loads—deploy your production models on iDatam's GPU Dedicated Servers. With our unmetered bandwidth and physical infrastructure isolation, you own the entire pipeline from the network layer to the Tensor cores.