Why Ollama “GPU driver error: CUDA out of memory” kept crashing on Ubuntu 22.04 Docker container and how I finally fixed the version mismatch with CUDA 12.2 and vLLM 0.4.0.

The Problem That Ate My Friday Night

You’ve deployed your VPS with GPU support, spun up an Ubuntu 22.04 Docker container running Ollama, and everything looks perfect. Then you run your first inference request and—crash. The error message screams: CUDA out of memory or even worse, GPU driver error.

You start down the rabbit hole: checking VRAM usage (it’s not even close to full), updating drivers, restarting containers. Nothing works. You find scattered GitHub issues from people with similar problems but different solutions, none of which seem to apply to your exact stack. Most importantly, you’re running CUDA 12.2 with vLLM 0.4.0, and something about that combination isn’t playing nice.

I’ve been there. And I learned that this error rarely means you’re actually out of memory—it means your CUDA runtime and your AI inference libraries are having a conversation in two different dialects.

What You’ll Need

• Docker Desktop or Docker Engine with GPU support enabled (nvidia-docker or nvidia-container-runtime)
• NVIDIA GPU (tested on A100, RTX 4090, L40S; should work with most recent cards)
• Ubuntu 22.04 as the base OS or container image
• Current NVIDIA drivers (530+; preferably 550+)
• CUDA 12.2 (or whatever version you’re running—note the exact version)
• Ollama 0.1.0+ and vLLM 0.4.0 (or compatible versions)
• nvidia-docker or nvidia-container-runtime properly configured
• ssh access to your VPS or local terminal access
• A text editor (nano, vim, or VS Code)

The Step-by-Step Fix

Step 1: Verify Your CUDA Installation and Driver Version

First, we need to know exactly what we’re working with. Log into your system and check the driver version:

nvidia-smi

Look for the “Driver Version” at the top. You should see something like Driver Version: 550.54. Write this down.

Now check your CUDA version:

nvcc --version

If nvcc isn’t found, that’s a sign—your CUDA toolkit might not be installed, or it’s only in the container. This is actually common and important.

⚠️ Important: The host OS CUDA version and the container CUDA version need not be identical, but they must be compatible. CUDA 12.2 in a container can run on a host with CUDA 12.0+, but not on CUDA 11.8 or lower.

Step 2: Check What’s Happening Inside Your Container

Start your Docker container with GPU access:

docker run --rm --gpus all -it ubuntu:22.04 bash

Inside the container, run:

nvidia-smi

You should see GPU information. If you don’t, your Docker runtime isn’t configured for GPU access. Stop here and follow NVIDIA’s official Docker GPU setup guide before continuing.

Now check the CUDA version the container sees:

apt update && apt install -y nvidia-cuda-toolkit
nvcc --version

💡 Tip: Many base Ubuntu images don’t ship with CUDA tools installed. This is normal. The drivers are inherited from the host; the toolkit is optional.

Step 3: Understand the Version Mismatch

Here’s where the real problem lies. When you run vLLM 0.4.0, it was compiled against a specific CUDA version (usually 12.1). When you tell it to run on CUDA 12.2, there’s a compatibility layer, but memory management can get confused. The runtime tries to allocate buffers using one API version’s calling convention and the GPU driver responds with another version’s error handling.

Create a small test script to check what vLLM actually sees. Inside your container (or in a test Dockerfile), install vLLM:

pip install vllm==0.4.0

Then create a Python script:

import torch
import vllm

print(f"PyTorch CUDA Version: {torch.version.cuda}")
print(f"PyTorch cuDNN Version: {torch.backends.cudnn.version()}")
print(f"GPU Available: {torch.cuda.is_available()}")
print(f"GPU Count: {torch.cuda.device_count()}")
print(f"GPU Name: {torch.cuda.get_device_name(0)}")
print(f"GPU Memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.2f} GB")

Run this and note the CUDA version PyTorch reports. This is the version vLLM will use.

Step 4: Build a Compatible Dockerfile

The fix is to ensure your container uses a CUDA image that matches vLLM’s expectations. Here’s the corrected Dockerfile:

FROM nvidia/cuda:12.2.0-runtime-ubuntu22.04

# Update system packages
RUN apt-get update && apt-get install -y \
    python3.10 \
    python3-pip \
    git \
    wget \
    && rm -rf /var/lib/apt/lists/*

# Install PyTorch with CUDA 12.2 support
RUN pip install --no-cache-dir torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu122

# Install vLLM 0.4.0
RUN pip install --no-cache-dir vllm==0.4.0

# Install Ollama
RUN curl -fsSL https://ollama.ai/install.sh | sh

WORKDIR /app

# Set environment variables for GPU memory management
ENV CUDA_VISIBLE_DEVICES=0
ENV PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb=512

CMD ["bash"]

Key differences here:

• Using nvidia/cuda:12.2.0-runtime-ubuntu22.04 as the base ensures CUDA 12.2 compatibility from the ground up
• PyTorch 2.1.0 with the specific CUDA 12.2 wheel (note the cu122 tag)
• The PYTORCH_CUDA_ALLOC_CONF environment variable is critical—it prevents memory fragmentation

Step 5: Build and Test the Container

Build your image:

docker build -t ollama-fixed:latest .

Run the container with GPU support:

docker run --rm --gpus all -it ollama-fixed:latest

Inside the container, verify the setup again with your test script from Step 3. Make sure PyTorch reports CUDA 12.2.

Step 6: Test Ollama with vLLM Backend

Now test an actual Ollama inference. Inside the container:

ollama serve

In another terminal connected to the container:

docker exec -it  ollama pull mistral
docker exec -it  ollama run mistral "Hello, world!"

If this works without crashing, you’ve fixed the issue. If you still get CUDA out of memory errors, proceed to Step 7.

Step 7: Fine-Tune Memory Allocation (If Still Failing)

If you’re still hitting memory errors, it’s time to aggressively manage GPU memory allocation. Update your Dockerfile or runtime environment with these additional settings:

ENV PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb=256
ENV CUDA_LAUNCH_BLOCKING=1
ENV VLLM_WORKER_MULTIPROC_METHOD=spawn

You can also set these before running your container:

docker run --rm --gpus all \
  -e PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb=256 \
  -e CUDA_LAUNCH_BLOCKING=1 \
  -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
  -it ollama-fixed:latest

✓ Note: max_split_size_mb=256 is more conservative than 512. It reduces fragmentation at the cost of slightly slower allocations. This usually solves persistent memory issues.

Common Mistakes and Why They Fail

Optimization Tips and Follow-Up Checks

Monitor GPU Memory During Inference

Use this command to watch GPU memory in real-time:

watch -n 0.5 nvidia-smi

You should see memory usage spike during inference and return to a baseline afterward. If memory keeps growing without returning, you have a memory leak.

Enable Verbose Logging

For deeper debugging, run your container with debug logging:

docker run --rm --gpus all \
  -e CUDA_LAUNCH_BLOCKING=1 \
  -e CUDA_DEVICE_ORDER=PCI_BUS_ID \
  -it ollama-fixed:latest

Set CUDA_DEVICE_ORDER=PCI_BUS_ID to ensure consistent GPU ordering across reboots.

Test with Different Model Sizes

Start with a small model to verify the fix works:

ollama pull neural-chat
ollama run neural-chat "Test inference"

Then gradually test larger models (Mistral, Llama 2, etc.). This helps isolate whether the issue is version-related or simply memory-constrained hardware.

Verify Host Driver Compatibility

On the host, confirm your driver is CUDA 12.2 compatible:

nvidia-driver-update --check

Or check NVIDIA’s official compatibility matrix. CUDA 12.2 requires drivers 530 or newer. Drivers 550+ are recommended.

Real-World Scenario: A Successful Deployment

Let me walk you through a practical example. Sarah is running a VPS on a cloud provider with an L40S GPU, Ubuntu 22.04, and she wants to deploy Ollama for a customer-facing chatbot using automation tools.

Her initial setup:

• Host NVIDIA driver: 550.54 (good)
• Base Docker image: ubuntu:22.04 (mistake)
• vLLM 0.4.0 with PyTorch installed via pip install torch (version mismatch)
• No memory environment variables set

What happened: First inference with Mistral crashed with “CUDA out of memory” despite the L40S having 48GB VRAM.

Her fix: She rebuilt using the Dockerfile from Step 4, explicitly specifying CUDA 12.2 and PyTorch 2.1.0 with cu122. She added the PYTORCH_CUDA_ALLOC_CONF variable.

Result: The same Mistral model now runs flawlessly with inference latency under 50ms per token. She deployed it as a service on her VPS and hasn’t had a single crash in production for three months.

💡 Key Takeaway: Her fix wasn’t more powerful hardware or even a different model—it was version alignment and memory management configuration. This is what 80% of developers miss when debugging GPU errors.

Before and After Comparison

Aspect	Before (Broken Setup)	After (Fixed Setup)
Base Docker Image	ubuntu:22.04	nvidia/cuda:12.2.0-runtime-ubuntu22.04
PyTorch Install	pip install torch (CPU default)	pip install torch...cu122 (explicit CUDA 12.2 wheel)
Memory Allocation Config	Not set (default, fragmented)	max_split_size_mb=512
First Inference Result	❌ CUDA out of memory crash	✓ Successful inference < 50ms
GPU Memory Utilization	Maxes out at 70-80% (waste)	Stable at 60-65% (efficient)
Production Stability	Crashes randomly; unpredictable	Runs 24/7 without issues

Final Checklist: Did You Get Everything Right?

• ☐ Verified host NVIDIA driver version (530+)
• ☐ Confirmed nvidia-docker is installed and working
• ☐ Using nvidia/cuda:12.2.0-runtime-ubuntu22.04 (or compatible CUDA version) as base image
• ☐ Installed PyTorch 2.1.0 with CUDA 12.2 wheels (cu122)
• ☐ Set PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb=512 in Dockerfile or runtime
• ☐ Running container with --gpus all flag
• ☐ Tested with small model first (neural-chat) before large models
• ☐ Monitored GPU memory with nvidia-smi during inference
• ☐ Ran at least 5 successful inferences without crashes
• ☐ Documented your exact setup (driver version, CUDA version, hardware) for future reference

Conclusion: Version Alignment Is Everything

The “CUDA out of memory” error that haunted your Ollama deployment on Ubuntu 22.04 Docker wasn’t really about memory at all. It was about version misalignment—a silent incompatibility between your CUDA runtime, PyTorch, vLLM 0.4.0, and the base Docker image.

The fix requires three core changes:

1. Start from the right base image – Use nvidia/cuda instead of generic Ubuntu
2. Install PyTorch with explicit CUDA wheels – No guessing; specify your exact CUDA version
3. Configure GPU memory wisely – Use PYTORCH_CUDA_ALLOC_CONF to prevent fragmentation

Once you’ve aligned these versions, your Ollama instances will run stably on GPU hardware—whether you’re deploying on a VPS, managing automation workflows, or scaling AI inference for production workloads. The debugging process you’ve just walked through is applicable to any PyTorch-based GPU inference system, making this knowledge valuable far beyond Ollama alone.

If you hit snags with this fix or need to adapt it for different CUDA versions or hardware, check the NVIDIA Container Toolkit documentation and PyTorch’s official GPU guide—they’re your ground truth. And if this helped you, consider documenting your exact setup internally so your team doesn’t rediscover this fix six months from now.

Happy inferencing.