cuda

vLLM Docker container keeps crashing with “CUDA out of memory” on Ubuntu 22.04 (RTX 4090) – step‑by‑step fix for the GPU memory leak and version mismatch issue.

by

You’ve been running vLLM in Docker for LLM inference, everything seemed fine in development, and then BAM—your container crashes with “CUDA out of memory” after a few minutes. Your RTX 4090 has 24GB of VRAM, but it’s behaving like you’re running on a laptop with 2GB. This is one of the most frustrating debugging sessions … Read more

Fix “CUDA out of memory” error when launching Ollama Llama 2 via vLLM in a Docker container on Ubuntu 22.04 VPS with 8 GB GPU – step‑by‑step debugging guide

by

You’ve got Ollama and vLLM set up on your Ubuntu VPS. You spin up the Docker container, everything looks ready, and then it hits you: CUDA out of memory. Your 8 GB GPU isn’t even close to being maxed out, but the error won’t budge. If this sounds familiar, you’re not alone—and the solution is … Read more