How to Run LLMs on Your Own Hardware Without Sending Everything to OpenAI

Look, I've tried running LLMs on everything from a GTX 1060 to an RTX 4090. Here's what actually matters and what's just marketing bullshit.

VRAM: The One Thing That Actually Matters

VRAM is everything. Run out of VRAM and your model either won't load or will crawl slower than a dying browser tab. I learned this the hard way trying to run Llama 70B on my RTX 3080's 12GB - it just laughed and fell back to system RAM at 0.5 tokens per second.

Real-world VRAM needs I've actually tested:

• 7B models: Need 4-6GB minimum. My RTX 3060 with 12GB runs Llama 3.1 8B at about 45 tokens/second
• 13B models: 8-12GB if you want decent speed. Anything less and you're swapping to system RAM
• 34B+ models: Forget it unless you have 24GB+. My friend's RTX 4090 barely handles Llama 34B at 15 tokens/second

System RAM matters when VRAM runs out. I run 32GB because models spill over constantly. 16GB works if you're only doing one thing at a time, but who actually does that?

GPU Reality Check

NVIDIA just works. Every LLM framework supports CUDA out of the box. No setup hell, no driver conflicts, no mysterious crashes. RTX 4080 ($800) and 4090 ($1600) are the sweet spots if you can afford them - but factor in the 320W and 450W power draw. My electricity bill jumped $40/month running inference workloads.

AMD ROCm is... complicated. Spent 6 hours getting ROCm working on Ubuntu 22.04 with my RX 7900 XTX. Performance is decent once it's running - about 80% of equivalent NVIDIA speeds - but the setup process is a nightmare of conflicting documentation and kernel module hell.

Apple Silicon works better than expected. My M2 Mac Studio with 64GB unified memory runs 13B models at 25 tokens/second. Not blazing fast, but the fact that it uses system RAM as VRAM means you can actually run larger models than most gaming rigs. Plus it's dead silent and sips power.

Storage: Don't Use Hard Drives

Get an NVMe SSD or suffer. Learned this loading Llama 70B from a mechanical drive - took 8 minutes every time. Same model loads in 20 seconds from my Samsung 980 Pro. These models are massive:

• Llama 3.1 8B: ~4.7GB
• Llama 3.1 70B: ~40GB for 4-bit, 140GB unquantized
• Code Llama 34B: ~20GB

Plan for 500GB minimum if you want to try different models. I filled a 1TB drive in two weeks downloading every interesting model I found on Hugging Face. "Oh, I'll just try this one 30B model" turns into a model hoarding addiction fast.

Network bandwidth: You'll download a lot of models. Each one is several GB. Get decent internet or you'll be waiting hours for each download. Ollama's resume feature works sometimes - when it doesn't, just ctrl+c and restart the damn thing.

CPU Performance: Don't Count It Out Completely

Modern CPUs aren't hopeless. While GPU inference dominates performance, AMD Zen 4 and Intel 13th gen processors with AVX-512 support can push 3-8 tokens per second on quantized 7B models. Not fast, but usable for testing and development when your GPU is busy mining Bitcoin or whatever.

ARM64 is getting interesting. Apple's M3 processors and AWS Graviton4 instances show decent performance per watt. My M3 MacBook Pro runs Llama 3.1 8B at 12 tokens/second using only system RAM - slower than dedicated GPU, but I can run inference for 8 hours on battery without the laptop turning into a space heater.

I've installed these tools on more machines than I care to remember. Here's what actually works and what will waste your weekend.

Ollama: Actually Easy (Mostly)

Windows: Just download the exe from ollama.com and run it. Seriously, it's one of the few installers that actually works. Windows Defender might freak out and quarantine it - click "allow" and move on.

macOS: brew install ollama works perfectly. Don't download the .pkg unless you hate yourself - Homebrew handles everything cleanly. If you get "command not found" after installing, restart your terminal like a normal person.

Linux: curl -fsSL https://ollama.com/install.sh | sh works on Ubuntu and most Debian-based distros. On Arch, just use the AUR package. On CentOS/RHEL, you might need to manually add the binary to /usr/local/bin/ because their install script has issues with systemd service files.

First model: ollama pull llama3.1:8b downloads about 5GB. Don't panic when it seems stuck at 90% - it's verifying the download. Takes 2-10 minutes depending on your internet.

llama.cpp: Prepare for Compilation Hell

Building from source: git clone https://github.com/ggerganov/llama.cpp.git then make clean && make -j$(nproc) on Linux. On Windows, good luck with Visual Studio - I spent 3 hours fighting CUDA path issues. For CUDA support, use make LLAMA_CUBLAS=1 but make sure your CUDA toolkit version matches exactly what their GitHub README says or you'll get cryptic linking errors like "nvcc fatal: Unsupported gpu architecture 'compute_89'".

Models: Download GGUF files from Hugging Face. Don't use GGML files - that format is dead since 2023. I keep models in ~/llm-models/ because scattered files are chaos. Popular ones: mistral-7b-instruct-v0.3.Q4_K_M.gguf runs well on 8GB cards.

Running it: ./main -m model.gguf -p "Your prompt" -n 100 -t 8 where -t is your CPU cores. Without CUDA, it's CPU-only and will take forever - like watching paint dry.

API server: ./server -m model.gguf --host 0.0.0.0 --port 8080 creates OpenAI-compatible endpoints. Works with most tools that expect GPT-style APIs.

vLLM: Installation Nightmare Mode

vLLM installation is a pain in the ass. Expect to spend 2 hours dealing with CUDA version conflicts, PyTorch compatibility hell, and mysteriously missing dependencies.

Set up Python environment: python -m venv vllm-env && source vllm-env/bin/activate then pip install --upgrade pip setuptools wheel. Don't skip this - mixing system Python with vLLM is asking for dependency conflicts.

Install vLLM: pip install vllm for CUDA. On AMD, use pip install vllm[rocm] and pray to the ROCm gods. Half the time this fails with CUDA version mismatches. If you get "RuntimeError: CUDA error: no kernel image is available for execution", your PyTorch CUDA version doesn't match your driver CUDA version. Delete everything and start over.

Actually running it:

python -m vllm.entrypoints.openai.api_server \
  --model /path/to/model \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 2

This assumes you have 2 GPUs. vLLM will crash if it can't find enough VRAM across your GPUs.

Making It Actually Work

Memory stuff: Set vm.swappiness=10 on Linux so it doesn't swap to death. Monitor with nvidia-smi and htop. When VRAM fills up, performance dies and your fans sound like jet engines.

IDE integration: Most code editors support OpenAI APIs. Point them at http://localhost:8080 for llama.cpp or http://localhost:8000 for vLLM. Works with VSCode extensions, Cursor, and most Python notebooks. Some need fake API keys even for local models - just put "sk-local" or whatever.

Production Deployment Reality Check

Docker makes this easier. Skip the dependency hell and use containers. Ollama's official Docker image handles GPU passthrough automatically with --gpus all. For vLLM, the official containers save you hours of CUDA setup pain.

Load balancing multiple instances: If you're serving real traffic, run multiple vLLM instances behind NGINX or HAProxy. Each instance can handle 10-50 concurrent requests depending on your hardware. Don't expect Ollama to scale beyond 4 simultaneous users.

Monitoring is critical. Set up Prometheus to track GPU utilization, model loading times, and request queues. When VRAM fills up, everything becomes a crawling disaster. I use Grafana dashboards to catch memory leaks before they kill performance and wake me up with alerts at 3am.