Ollama Context Length Errors: The Silent Killer

I've wasted too many nights debugging Ollama's context bullshit. Here's what's actually broken and how to fix it.

What Are Context Length Errors?

Context length errors happen when your conversation gets too long for the model's tiny brain to handle. Unlike CUDA out of memory errors that crash your GPU, context limits just silently fuck everything up without telling you.

LLMs use context windows to track conversation history, but Ollama's garbage defaults hide the real limits from you

Here's the bullshit: Ollama fails silently when you hit context limits. No error messages, no warnings - it just silently truncates your conversation by throwing away the oldest tokens. This causes all kinds of problems:

Silent Truncation: The Hidden Problem

When context overflows, Ollama silently discards old tokens using FIFO (first-in-first-out) strategy - no warnings, just missing context that breaks everything

I've seen this shit break in production more times than I can count. Here's how silent truncation will screw you:

• Conversation amnesia: Your chatbot suddenly forgets it's supposed to be helpful
• Document analysis failures: 50-page PDF? Model only reads the last 3 pages
• Persona drift: Your customer service bot becomes a sarcastic asshole mid-conversation
• Incomplete reasoning: Multi-step problems lose their setup and give you garbage answers

This happens because Ollama uses FIFO token management - first tokens in, first tokens deleted when you run out of space. No warning, no error message, just silent failure that makes you question your sanity.

Token Counting: The Invisible Fuckery

You need to understand how Ollama counts tokens or you'll never figure out why your context breaks. Token estimation is completely different between models, and tokenization approaches vary like crazy:

• Llama models: Average ~4 characters per token for English text
• Code-focused models: Different tokenization for programming languages
• Multilingual models: Varying token ratios for different languages
• Special tokens: System prompts and formatting add overhead

Here's where people fuck up: thinking words equal tokens. Wrong. "Hello world!" = 2-3 tokens. "Optimization" = 3 tokens. "Anthropomorphization" = 6 tokens. Your carefully crafted 500-word prompt might be 800 tokens, and you won't know until your context window explodes.

Your perfectly crafted prompt gets shredded into tokens - "optimization" becomes 3 tokens, "Anthropomorphization" becomes 6. No wonder you run out of space.

Default Context Limits: The 2048 Problem

Here's the bullshit: Ollama defaults to 2048 tokens for EVERY model. Doesn't matter if your model can handle 128K tokens - Ollama says "fuck you, you get 2K." This causes:

• Artificial limitations: Your Llama 3.1 can do 128K tokens but Ollama caps it at 2K
• Silent failures: No warnings, no errors, just broken shit
• User confusion: Spend hours wondering why your 10-page document analysis only covers page 9-10

The 2048 token limit is equivalent to roughly 1,500-2,000 words of English text, including the model's response. For any substantial conversation or document analysis, this limit is reached quickly. From what I've seen, 4K is the minimum for anything real. 2K is a joke. Aider documentation confirms this causes major problems for coding applications, and BrainSoup optimization guides recommend 8K+ tokens for complex tasks.

Real-World War Stories

Production Horror Story - The Customer Service Meltdown:
We had this support bot, worked fine for weeks until it didn't. Thing started being rude to customers out of nowhere. Karen from customer service is calling me at 8pm screaming about the bot telling someone to "figure it out yourself." Took me three fucking days to realize the context was eating our 'be nice' instructions. No errors, no warnings, just our bot slowly becoming a sarcastic asshole because Ollama decided the system prompt was expendable.

The Document Demo Disaster:
CEO demo, of course. "Watch our AI analyze this 40-page compliance document." Guess what? It only read the last 3 pages because context limit. Made up shit about the first 37 pages. CEO asks about section 2, AI confidently bullshits an answer. I'm sitting there knowing it never saw section 2. Good times.

Model-Specific Context Capabilities

Different model families have vastly different context capabilities that aren't reflected in Ollama's defaults. Ollama models often understate actual context capabilities, and community testing reveals significantly higher limits than officially documented:

Llama Family Context Windows

Most Llama 3.1 models handle up to 128K tokens, but Ollama defaults to 2K like an asshole. Code models usually do 16K. Check the docs for your specific model because they're all different and the defaults are garbage.

The disconnect between model capabilities and Ollama defaults is a major source of user confusion. OpenAI's research on context lengths shows that most users need 8-16K tokens for practical applications, but Ollama defaults to 2K across all models.

Context Length vs. Memory Issues: Critical Differences

People always confuse context length errors with memory problems, but they're completely different beasts:

Context Length Issues:

• Cause: Token limit exceeded in model architecture
• Symptoms: Silent truncation, conversation amnesia, incomplete analysis
• Solution: Bump num_ctx parameter or set up sliding window
• Resource usage: Affects computational complexity, not RAM/VRAM directly

Memory Issues:

• Cause: Insufficient RAM/VRAM for model loading
• Symptoms: CUDA out of memory errors, system crashes, allocation failures
• Solution: Free memory, reduce model size, optimize GPU allocation
• Resource usage: Directly related to hardware limitations

The Performance Trade-off

Increasing context length has significant performance implications that many users don't anticipate:

• Quadratic complexity: Attention calculation scales O(n²) with sequence length
• Memory usage increases: Longer contexts require more GPU memory for attention matrices
• Slower inference: Each token generation takes longer with larger contexts
• Hardware requirements: Large contexts may require high-end GPUs

According to performance benchmarks, increasing context from 2K to 32K tokens can reduce inference speed by 4-8x on typical consumer hardware. Context window management techniques become crucial for overcoming LLM token limits in production applications.

Why your 70B model crawls when you bump context - attention math is quadratic so doubling context quadruples the work

When Context Length Errors Occur

Here's exactly where this breaks in production:

Long Conversations

Multi-turn conversations accumulate context over time. A typical pattern:

• Turn 1-5: Normal performance, full context retained
• Turn 6-10: Subtle degradation as early context starts getting truncated
• Turn 11+: Significant personality drift and context loss

Document Processing

Large document analysis fails predictably:

• PDF summarization: Only final pages get processed
• Code review: Incomplete file analysis
• Research analysis: Missing introductions and methodology

Batch Processing

Applications processing multiple items sequentially hit limits when context accumulates across batch items.

Template-Heavy Applications

System prompts with extensive templates consume significant context budget before user input is processed.

The OLLAMA_NUM_CTX Environment Variable

The fix that actually works (when it works) is `OLLAMA_NUM_CTX`, but the docs are shit:

## Set context length to 8192 tokens
export OLLAMA_NUM_CTX=8192
ollama serve

## Or use the Ollama CLI directly
ollama run llama3.1:70b --num-ctx 8192 \"Your prompt here\"

However, many users report that setting OLLAMA_NUM_CTX doesn't always work as expected, particularly with systemd services or when running in containers.

Context Management Strategies

Instead of just cranking up context length, smart applications use context management tricks:

Sliding Window Approach

Keep only the most recent N tokens, discarding older context gradually rather than all-at-once.

Context Summarization

Periodically summarize older conversation history into compressed context, preserving key information while reducing token count.

Hierarchical Context

Separate system instructions, recent conversation, and background knowledge into different context tiers with different retention policies.

Retrieval-Augmented Context

Store context externally and retrieve relevant portions based on current input, rather than maintaining everything in the context window.

These strategies are essential for production applications that need to maintain coherent, long-term conversations without running into context limits.

The big problem with Ollama is that it gives you fuck-all for context management tools, so you have to build your own solutions to handle context smartly.

Now that you understand what's broken and why, let's fix this shit with step-by-step solutions that actually work in production.

After wasting months debugging this crap, here are the fixes that actually work in production.

Diagnosing Context Length Issues

Before fixing anything, make sure you're actually dealing with context length problems and not just memory issues:

Detection Methods

Try logging first - see what's actually happening

## Enable verbose logging to see token counts
export OLLAMA_DEBUG=1
ollama run llama3.1:70b "Your test prompt here"

Look for log entries showing token counts. If you see truncation warnings or the response quality degrades suddenly during long conversations, you're hitting context limits. Debugging Ollama issues requires understanding systemctl logs and environment variable inheritance in different deployment scenarios.

Feed it something big and see what breaks

## Create a test file with known token count
echo "This is a test document with approximately 3000 tokens..." > large_test.txt
## (add enough content to exceed 2048 tokens)

## Test with default settings
ollama run llama3.1:70b "Summarize this entire document: $(cat large_test.txt)"

If the model only references the end of the document, you've confirmed silent truncation.

Chat until it forgets who it is

Start a conversation with detailed system instructions, then have a long conversation (15+ exchanges). If the model forgets its initial instructions or persona, you're hitting the context limit.

Solution 1: Configure OLLAMA_NUM_CTX Properly

The primary solution is increasing the context window, but configuration can be tricky:

Basic Configuration

## Try the environment variable first
export OLLAMA_NUM_CTX=8192
ollama serve

## Method 2: Per-request via CLI
ollama run llama3.1:70b --num-ctx 8192 "Your prompt"

## Method 3: Via modelfile
echo "FROM llama3.1:70b" > Modelfile
echo "PARAMETER num_ctx 8192" >> Modelfile
ollama create custom-model -f Modelfile

Systemd Service Configuration

Many users report that environment variables don't work with systemd. This is because systemd service isolation prevents services from inheriting user environment variables. Systemd security features and service configuration best practices require explicit environment variable declaration. Here's the reliable fix:

## Edit the systemd service file
sudo systemctl edit ollama.service

## Add this content:
[Service]
Environment="OLLAMA_NUM_CTX=8192"

## Reload and restart
sudo systemctl daemon-reload
sudo systemctl restart ollama.service

## Verify the setting took effect
systemctl show ollama.service | grep OLLAMA_NUM_CTX

Docker Configuration

For Docker deployments, you need to pass environment variables explicitly because Docker containers are isolated by default. Docker security model and container isolation mechanisms prevent automatic environment inheritance from the host system. Docker Compose environment variables provide consistent configuration management across deployment scenarios:

Docker deployment with proper context configuration - environment variables must be explicitly passed because Docker containers don't inherit your shell settings

## Run with environment variable
docker run -d \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  -e OLLAMA_NUM_CTX=8192 \
  --name ollama \
  ollama/ollama

## Or use docker-compose
version: '3.8'
services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    environment:
      - OLLAMA_NUM_CTX=8192

Solution 2: Choosing Optimal Context Lengths

Not all context lengths are equal. Pick your context size based on how much pain you can tolerate:

Conservative Approach (4K-8K tokens)

export OLLAMA_NUM_CTX=4096  # Good for most conversations
export OLLAMA_NUM_CTX=8192  # Better for document analysis

Use cases:

• General chatbots
• Code assistance
• Short document summarization
• Customer support

Performance impact: Minimal on most hardware

Moderate Approach (16K-32K tokens)

export OLLAMA_NUM_CTX=16384  # Long conversations
export OLLAMA_NUM_CTX=32768  # Large document processing

Use cases:

• Long-form writing assistance
• Complex technical discussions
• Multi-page document analysis
• Educational tutoring

Performance impact: Your GPU will wheeze, responses take forever, and you'll need 8GB+ VRAM

Aggressive Approach (64K+ tokens)

export OLLAMA_NUM_CTX=65536   # Very long contexts
export OLLAMA_NUM_CTX=131072  # Maximum for Llama 3.1/3.3

Use cases:

• Research paper analysis
• Entire codebase review
• Book summarization
• Long-term conversation memory

Performance impact: Responses take forever, your GPU will melt, needs 16GB+ VRAM

Context eats VRAM like crazy - 32K contexts need 16GB+ VRAM and make your GPU wheeze. Monitor with nvidia-smi to watch your VRAM disappear as context grows.

Solution 3: Model-Specific Optimization

Different models have different optimal context configurations:

For most models: 8K is safe, 16K if you have decent hardware, 32K if you hate waiting for responses. Code models usually need 16K+ to analyze full files. Adjust based on your VRAM and patience level.

Solution 4: Application-Level Context Management

For production apps, use smart context management instead of just bumping limits:

Sliding Window Implementation

class SlidingWindowContext:
    def __init__(self, max_tokens=8192, keep_system=True):
        self.max_tokens = max_tokens
        self.keep_system = keep_system
        self.system_prompt = ""
        self.messages = []
    
    def add_message(self, role, content):
        # Estimate tokens (rough approximation)
        tokens = len(content.split()) * 1.3
        
        # Add new message
        self.messages.append({"role": role, "content": content, "tokens": tokens})
        
        # Truncate if needed
        total_tokens = sum(msg["tokens"] for msg in self.messages)
        if total_tokens > self.max_tokens:
            # Keep system prompt and recent messages
            keep_recent = []
            current_tokens = 0
            
            for msg in reversed(self.messages):
                if current_tokens + msg["tokens"] < self.max_tokens * 0.8:
                    keep_recent.insert(0, msg)
                    current_tokens += msg["tokens"]
                else:
                    break
            
            self.messages = keep_recent

Context Summarization Strategy

def summarize_old_context(messages, keep_recent=5):
    """Summarize older messages to compress context"""
    if len(messages) <= keep_recent * 2:
        return messages
    
    old_messages = messages[:-keep_recent]
    recent_messages = messages[-keep_recent:]
    
    # Create summary of old messages
    old_text = "
".join([f"{msg['role']}: {msg['content']}" 
                         for msg in old_messages])
    
    summary_prompt = f"Summarize this conversation history in 2-3 sentences:
{old_text}"
    summary = ollama_generate(summary_prompt)
    
    # Return summary + recent messages
    return [{"role": "system", "content": f"Previous context: {summary}"}] + recent_messages

Solution 5: When Nothing Works

You followed the instructions and it's still broken. Here's the weird shit that breaks context settings:

Issue: OLLAMA_NUM_CTX Not Taking Effect

You set the environment variable but still getting 2048 token behavior like nothing happened. Classic Ollama weirdness.

• Docker containers: Container can't see your host settings - Docker doesn't magically inherit your shell
• WSL2 weirdness: Windows loses your environment variables somewhere between the layers
• Systemd isolation: Service runs in its own world, can't see your shell variables
• Version-specific bugs: I've seen this break differently across Ollama versions - check the GitHub issues for yours

Solutions That Actually Work:

## 1. Nuclear option - kill everything and restart
pkill -9 ollama  # -9 because ollama sometimes hangs on SIGTERM
export OLLAMA_NUM_CTX=8192
ollama serve

## 2. Check if systemd is fucking with you
systemctl show ollama.service | grep Environment
## If empty, systemd is ignoring your env vars

## 3. Bypass the whole mess with CLI params  
ollama run llama3.1:70b --num-ctx 8192 "test prompt"

## 4. Create a custom model (works on all versions)
echo "FROM llama3.1:70b" > Modelfile  
echo "PARAMETER num_ctx 8192" >> Modelfile
ollama create llama3.1-8k -f Modelfile

Issue: Performance Goes to Shit with Large Contexts

You bumped context to 32K and now responses take 30 seconds. Welcome to quadratic attention - it's not a bug, it's math.

Damage Control:

## Reduce GPU layers to free VRAM for context processing
export OLLAMA_NUM_CTX=8192
export OLLAMA_NUM_GPU_LAYERS=20  # Down from 35, saves 4GB VRAM
export OLLAMA_NUM_PARALLEL=1     # Don't run multiple models

## CPU inference sometimes wins with huge contexts
export OLLAMA_NUM_THREADS=8      
export OLLAMA_NUM_CTX=4096       

## Watch your VRAM die in real-time
watch -n1 'nvidia-smi; echo "---"; free -h'

Issue: Out of Memory with Large Contexts

Oh great, now you're getting CUDA OOM errors. Your GPU can't handle the context size you threw at it.

## Free up some VRAM for context processing
export OLLAMA_NUM_GPU_LAYERS=28  # Down from 35
export OLLAMA_NUM_CTX=8192

## Hybrid approach when GPU is full
export OLLAMA_NUM_GPU_LAYERS=20  
export OLLAMA_NUM_CTX=16384

## Watch the carnage
watch -n1 'nvidia-smi; echo "---"; free -h'

Solution 6: Actually Test That It Works

OK, you think you fixed it. Don't trust Ollama to tell you - test that your context settings actually work:

Context Length Test Script

#!/bin/bash
## Test if your context actually works (spoiler: it probably doesn't)

echo "Testing context length - fingers crossed..."

## Check what you think you set
echo "OLLAMA_NUM_CTX: ${OLLAMA_NUM_CTX:-not set, so you're fucked}"

## Feed it something massive and see if it chokes
echo "Throwing big input at it..."
LARGE_TEXT=""
for i in {1..100}; do
    LARGE_TEXT+="Line $i blah blah filling up tokens. "
done

echo "If this only mentions the last few lines, your context is broken:"
ollama run llama3.1:70b "Summarize this and tell me the first line number: $LARGE_TEXT" || echo "Failed completely, nice"

## Memory test - will it remember 5 seconds ago?
echo "Testing if it has the memory of a goldfish..."
ollama run llama3.1:70b "My favorite color is purple" > /dev/null 2>&1
sleep 2
echo "Does it remember what I just said?"
ollama run llama3.1:70b "What's my favorite color?" || echo "Can't even remember colors, good luck with documents"

API-Based Context Testing

import requests
import json

def test_context_length(model="llama3.1:70b", num_ctx=8192):
    """Test if context length setting is working"""
    
    # Create a large prompt
    large_prompt = "Remember these numbers: " + " ".join([str(i) for i in range(1, 500)])
    large_prompt += " Now, what were the first 5 numbers I mentioned?"
    
    response = requests.post("http://localhost:11434/api/generate", 
                           json={
                               "model": model,
                               "prompt": large_prompt,
                               "stream": False,
                               "options": {"num_ctx": num_ctx}
                           })
    
    result = response.json()
    print(f"Context {num_ctx}: {result['response']}")
    
    # Check if it remembered the beginning
    if "1" in result['response'] and "5" in result['response']:
        print(f"✅ Context {num_ctx} working - remembered beginning")
    else:
        print(f"❌ Context {num_ctx} failed - truncated beginning")

## Test different context lengths
for ctx in [2048, 4096, 8192]:
    test_context_length(num_ctx=ctx)

Solution 7: Monitoring Context Usage

Implement monitoring to track context usage and prevent future issues:

Simple Context Monitor

#!/bin/bash
## Monitor context usage - this breaks half the time but whatever

while true; do
    echo "=== Context Usage $(date) ==="
    echo "OLLAMA_NUM_CTX: ${OLLAMA_NUM_CTX:-probably 2048}"
    
    # Check if ollama is even running (spoiler: it's not)
    if pgrep ollama > /dev/null; then
        echo "✅ Ollama actually running for once"
        
        # Test response time - this hangs sometimes
        START_TIME=$(date +%s.%N)
        timeout 30 ollama run llama3.1:70b "test" > /dev/null 2>&1 || echo "Timed out, ollama is stuck again"
        END_TIME=$(date +%s.%N) 
        RESPONSE_TIME=$(echo "$END_TIME - $START_TIME" | bc 2>/dev/null || echo "bc failed")
        echo "Response time: ${RESPONSE_TIME}s (if it worked)"
        
        # GPU memory - assuming you have nvidia-smi installed
        if command -v nvidia-smi &> /dev/null; then
            GPU_MEM=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits 2>/dev/null || echo "?")
            echo "GPU memory: ${GPU_MEM}MB (might be wrong)"
        else
            echo "No nvidia-smi, probably using CPU like a caveman"
        fi
    else
        echo "❌ Ollama dead again, surprise surprise"
    fi
    
    echo "---"
    sleep 30  # wait and try again, maybe it'll work this time
done

That covers the main ways context length fucks up and how to fix it. I've debugged this exact problem about 50 times across different Ollama versions, and these solutions work on everything I've tested.

Version warning: Newer versions handle context better than older ones, but each version breaks in its own special way. Check the GitHub issues for your specific version if nothing works.

Lost a weekend debugging this once. Customer demo Monday, RAG system hallucinating because it could only read the last 2 pages of documents. Ollama defaults are trash.

When bumping context doesn't work, you need hierarchical memory management. I've implemented these after simple OLLAMA_NUM_CTX increases crashed production apps. Here's what works beyond just making the window bigger. Advanced memory management patterns and context engineering techniques from recent research show hierarchical attention mechanisms significantly improve long context performance in production environments.

Production systems need smarter memory - you can't just increase context forever without breaking something

Hierarchical Context Management

Tier 1: System prompts (never truncated) → Tier 2: Key facts (preserved when possible) → Tier 3: Recent conversation (fills remaining space) → Tier 4: Old messages (compressed or discarded)

OK, enough ranting. Here's the technical breakdown - you need a tiered system that prioritizes different types of information:

Tier 1: System Instructions (Always Preserved)

class HierarchicalContext:
    def __init__(self, max_tokens=8192):
        self.max_tokens = max_tokens
        self.system_prompt = ""  # Never truncated (hopefully)
        self.key_facts = []      # Important shit 
        self.conversation = []   # Gets nuked first
        
        # Needs 4GB+ VRAM or it crashes
        
    def add_system_prompt(self, prompt):
        # Keep system prompt or everything goes to hell
        self.system_prompt = prompt
        
    def add_key_fact(self, fact):
        # Critical shit that better not disappear
        self.key_facts.append({
            "content": fact,
            "timestamp": datetime.now(),
            "tokens": self.estimate_tokens(fact)
        })
        
    def add_conversation_turn(self, role, content):
        # Regular chat - gets truncated when we run out of space
        self.conversation.append({
            "role": role, 
            "content": content,
            "timestamp": datetime.now(),
            "tokens": self.estimate_tokens(content)
        })
        
    def get_context(self):
        """Assemble context respecting hierarchy and token limits"""
        context_parts = []
        used_tokens = 0
        
        # Tier 1: Always include system prompt
        if self.system_prompt:
            context_parts.append(f"System: {self.system_prompt}")
            used_tokens += self.estimate_tokens(self.system_prompt)
        
        # Tier 2: Include key facts if space allows
        for fact in reversed(self.key_facts):  # Most recent first
            if used_tokens + fact["tokens"] < self.max_tokens * 0.3:
                context_parts.append(f"Key: {fact['content']}")
                used_tokens += fact["tokens"]
        
        # Tier 3: Fill remaining space with recent conversation
        remaining_tokens = self.max_tokens - used_tokens
        recent_conversation = []
        
        for turn in reversed(self.conversation):
            if used_tokens + turn["tokens"] < self.max_tokens:
                recent_conversation.insert(0, f"{turn['role']}: {turn['content']}")
                used_tokens += turn["tokens"]
            else:
                break
                
        context_parts.extend(recent_conversation)
        
        # Breaks if token math is wrong (which happens)
        return "
".join(context_parts)

Tier Benefits

• System instructions remain intact throughout long conversations
• Important facts are preserved even when conversation history is truncated
• Recent context is prioritized over older exchanges
• Predictable behavior under varying conversation lengths

Context Compression Techniques

Instead of discarding old context, compress it intelligently to preserve essential information:

Summarization-Based Compression

class ContextCompressor:
    def __init__(self, ollama_client, compression_ratio=0.3):
        self.client = ollama_client
        self.compression_ratio = compression_ratio  # Target 30% of original tokens
        
    def compress_context_block(self, messages, target_tokens):
        """Compress a block of messages to target token count"""
        
        # Combine messages into text
        text_block = "
".join([f"{msg['role']}: {msg['content']}" 
                               for msg in messages])
        
        # Create compression prompt
        compression_prompt = f"""
        Summarize this conversation block, preserving key information:
        - Important decisions or agreements
        - Specific facts or data mentioned  
        - User preferences or requirements
        - Technical details or constraints
        
        Original conversation:
        {text_block}
        
        Provide a concise summary in approximately {target_tokens} tokens:
        """
        
        # Generate compressed version
        response = self.client.generate(
            model="llama3.1:70b",
            prompt=compression_prompt,
            options={"num_ctx": 4096}
        )
        
        return {
            "role": "system",
            "content": f"Context summary: {response['response']}",
            "tokens": self.estimate_tokens(response['response']),
            "compressed_from": len(messages)
        }
    
    def compress_conversation(self, conversation_history, max_tokens):
        """Compress old messages so we don't run out of space"""
        
        if self.total_tokens(conversation_history) <= max_tokens:
            return conversation_history
            
        # Keep recent messages uncompressed
        keep_recent = 5
        recent = conversation_history[-keep_recent:] 
        older = conversation_history[:-keep_recent]
        
        # Compress older messages in blocks
        compressed_blocks = []
        current_block = []
        current_tokens = 0
        target_block_size = max_tokens * 0.2  # 20% of total for old context
        
        for msg in older:
            current_block.append(msg)
            current_tokens += msg['tokens']
            
            # When block is large enough, compress it
            if current_tokens >= target_block_size:
                compressed = self.compress_context_block(
                    current_block, 
                    int(current_tokens * self.compression_ratio)
                )
                compressed_blocks.append(compressed)
                current_block = []
                current_tokens = 0
                
        # Compress remaining messages
        if current_block:
            compressed = self.compress_context_block(
                current_block,
                int(current_tokens * self.compression_ratio)
            )
            compressed_blocks.append(compressed)
            
        # Return compressed old + recent uncompressed
        return compressed_blocks + recent

Template-Based Compression

For structured conversations, use templates to compress repetitive information:

class TemplateCompressor:
    def __init__(self):
        self.patterns = {
            "troubleshooting": {
                "pattern": r"User: (.+)
Assistant: (.+solution.+)
User: (tried|done|working)",
                "template": "Solved: {problem} → {solution}"
            },
            "q_and_a": {
                "pattern": r"User: (.+\?)
Assistant: (.+)",
                "template": "Q: {question} A: {answer}"
            },
            "configuration": {
                "pattern": r"set (.+) to (.+)",
                "template": "Config: {param}={value}"
            }
        }
    
    def compress_by_patterns(self, conversation_text):
        """Use templates to compress common conversation patterns"""
        compressed = conversation_text
        
        for pattern_name, pattern_info in self.patterns.items():
            regex = re.compile(pattern_info["pattern"], re.IGNORECASE | re.DOTALL)
            template = pattern_info["template"]
            
            # Find and replace matches
            matches = regex.findall(compressed)
            for match in matches:
                if isinstance(match, tuple):
                    replacement = template.format(*match)
                else:
                    replacement = template.format(match)
                compressed = regex.sub(replacement, compressed, count=1)
                
        return compressed

Retrieval-Augmented Context (RAC)

For applications requiring access to large knowledge bases, implement external context storage. Retrieval-augmented generation (RAG) using vector databases like ChromaDB, Pinecone, or Weaviate enables scalable context management for enterprise applications requiring long-term memory:

Vector-Based Context Retrieval

Vector database storage for context retrieval using ChromaDB - manages long-term memory by storing conversation chunks as embeddings

import chromadb
from sentence_transformers import SentenceTransformer

class RetrievalAugmentedContext:
    def __init__(self, max_context_tokens=8192, max_retrieved=2048):
        self.max_context_tokens = max_context_tokens
        self.max_retrieved = max_retrieved
        
        # Initialize vector database
        self.chroma_client = chromadb.Client()
        self.collection = self.chroma_client.create_collection("context_memory")
        
        # Initialize embedding model
        self.encoder = SentenceTransformer('all-MiniLM-L6-v2')
        
    def store_context(self, context_id, content, metadata=None):
        """Store context chunk in vector database"""
        embedding = self.encoder.encode(content).tolist()
        
        self.collection.add(
            documents=[content],
            embeddings=[embedding],
            ids=[context_id],
            metadatas=[metadata or {}]
        )
        
    def retrieve_relevant_context(self, query, n_results=5):
        """Retrieve most relevant context for current query"""
        query_embedding = self.encoder.encode(query).tolist()
        
        results = self.collection.query(
            query_embeddings=[query_embedding],
            n_results=n_results,
            include=["documents", "distances", "metadatas"]
        )
        
        # Filter and rank results
        relevant_context = []
        used_tokens = 0
        
        for doc, distance, metadata in zip(
            results["documents"][0], 
            results["distances"][0], 
            results["metadatas"][0]
        ):
            # Only include if similarity is high enough
            if distance < 0.7:  # Adjust threshold as needed
                doc_tokens = self.estimate_tokens(doc)
                if used_tokens + doc_tokens < self.max_retrieved:
                    relevant_context.append({
                        "content": doc,
                        "relevance": 1 - distance,
                        "tokens": doc_tokens,
                        "metadata": metadata
                    })
                    used_tokens += doc_tokens
                    
        return relevant_context
    
    def build_context_with_retrieval(self, current_conversation, user_query):
        """Build context combining conversation + retrieved information"""
        
        # Get relevant background context
        retrieved = self.retrieve_relevant_context(user_query)
        
        # Calculate token budget
        retrieved_tokens = sum(item["tokens"] for item in retrieved)
        conversation_budget = self.max_context_tokens - retrieved_tokens
        
        # Truncate conversation to fit budget
        truncated_conversation = self.truncate_conversation(
            current_conversation, 
            conversation_budget
        )
        
        # Assemble final context
        context_parts = []
        
        # Add retrieved context
        if retrieved:
            context_parts.append("Relevant background information:")
            for item in retrieved:
                context_parts.append(f"- {item['content']}")
            context_parts.append("
Current conversation:")
            
        # Add conversation
        for turn in truncated_conversation:
            context_parts.append(f"{turn['role']}: {turn['content']}")
            
        return "
".join(context_parts)

Dynamic Context Allocation

Adjust context allocation based on conversation type and user needs:

Adaptive Context Manager

class AdaptiveContextManager:
    def __init__(self, max_tokens=8192):
        self.max_tokens = max_tokens
        self.conversation_types = {
            "short_qa": {"recent_turns": 3, "system_ratio": 0.1},
            "technical_support": {"recent_turns": 10, "system_ratio": 0.3},
            "creative_writing": {"recent_turns": 15, "system_ratio": 0.2},
            "document_analysis": {"recent_turns": 5, "system_ratio": 0.1},
            "code_review": {"recent_turns": 8, "system_ratio": 0.2}
        }
        
    def detect_conversation_type(self, recent_messages):
        """Analyze conversation to determine type"""
        
        combined_text = " ".join([msg["content"] for msg in recent_messages[-3:]])
        
        # Simple heuristics (could be replaced with ML classifier)
        if any(keyword in combined_text.lower() for keyword in 
               ["error", "bug", "fix", "troubleshoot", "not working"]):
            return "technical_support"
        elif any(keyword in combined_text.lower() for keyword in 
                ["write", "story", "creative", "character", "plot"]):
            return "creative_writing"
        elif any(keyword in combined_text.lower() for keyword in 
                ["code", "function", "class", "import", "def", "var"]):
            return "code_review"
        elif any(keyword in combined_text.lower() for keyword in 
                ["analyze", "summarize", "document", "review"]):
            return "document_analysis"
        else:
            return "short_qa"
            
    def allocate_context(self, conversation_history, system_prompt=""):
        """Dynamically allocate context based on conversation type"""
        
        conv_type = self.detect_conversation_type(conversation_history)
        config = self.conversation_types[conv_type]
        
        # Calculate allocations
        system_tokens = int(self.max_tokens * config["system_ratio"])
        conversation_tokens = self.max_tokens - system_tokens
        
        # Get recent turns based on type
        recent_turns = conversation_history[-config["recent_turns"]:]
        
        # Truncate to fit budget
        final_conversation = []
        used_tokens = 0
        
        for turn in reversed(recent_turns):
            turn_tokens = self.estimate_tokens(turn["content"])
            if used_tokens + turn_tokens < conversation_tokens:
                final_conversation.insert(0, turn)
                used_tokens += turn_tokens
            else:
                break
                
        # Build final context
        context = []
        if system_prompt:
            context.append(f"System: {system_prompt}")
        
        context.append(f"Conversation type: {conv_type}")
        
        for turn in final_conversation:
            context.append(f"{turn['role']}: {turn['content']}")
            
        return "
".join(context)

Context Streaming and Checkpointing

For very long conversations, implement context checkpointing:

Context Checkpointing System

Memory management and checkpointing strategies for long conversations use periodic compression to avoid hitting token limits

class ContextCheckpointer:
    def __init__(self, checkpoint_interval=20, max_checkpoints=5):
        self.checkpoint_interval = checkpoint_interval
        self.max_checkpoints = max_checkpoints
        self.checkpoints = []
        self.current_conversation = []
        
    def add_turn(self, role, content):
        """Add conversation turn and checkpoint if needed"""
        self.current_conversation.append({
            "role": role,
            "content": content,
            "timestamp": datetime.now()
        })
        
        # Create checkpoint every N turns
        if len(self.current_conversation) % self.checkpoint_interval == 0:
            self.create_checkpoint()
            
    def create_checkpoint(self):
        """Create a compressed checkpoint of current conversation"""
        
        if not self.current_conversation:
            return
            
        # Compress conversation block
        checkpoint_summary = self.compress_conversation_block(
            self.current_conversation
        )
        
        checkpoint = {
            "id": len(self.checkpoints),
            "timestamp": datetime.now(),
            "summary": checkpoint_summary,
            "turn_count": len(self.current_conversation),
            "tokens": self.estimate_tokens(checkpoint_summary)
        }
        
        self.checkpoints.append(checkpoint)
        
        # Keep only recent checkpoints
        if len(self.checkpoints) > self.max_checkpoints:
            self.checkpoints.pop(0)
            
        # Clear old conversation (keep last few turns for continuity)
        self.current_conversation = self.current_conversation[-5:]
        
    def build_context_from_checkpoints(self, max_tokens):
        """Build context from checkpoints + current conversation"""
        
        context_parts = []
        used_tokens = 0
        
        # Add checkpoints (oldest first, most recent last)
        for checkpoint in self.checkpoints:
            if used_tokens + checkpoint["tokens"] < max_tokens * 0.4:
                context_parts.append(f"Previous context: {checkpoint['summary']}")
                used_tokens += checkpoint["tokens"]
                
        # Add current conversation
        for turn in self.current_conversation:
            turn_tokens = self.estimate_tokens(turn["content"])
            if used_tokens + turn_tokens < max_tokens:
                context_parts.append(f"{turn['role']}: {turn['content']}")
                used_tokens += turn_tokens
                
        return "
".join(context_parts)

Performance Optimization for Large Contexts

When using large context windows, optimize for performance:

Context Caching Strategy

class ContextCache:
    def __init__(self, cache_size=10):
        self.cache = {}
        self.cache_size = cache_size
        self.access_order = []
        
    def get_context_hash(self, context_parts):
        """Create hash for context to enable caching"""
        context_str = "|".join([part["content"] for part in context_parts])
        return hashlib.md5(context_str.encode()).hexdigest()
        
    def get_cached_response(self, context_hash):
        """Get cached response if available"""
        if context_hash in self.cache:
            # Update access order
            self.access_order.remove(context_hash)
            self.access_order.append(context_hash)
            return self.cache[context_hash]
        return None
        
    def cache_response(self, context_hash, response):
        """Cache response with LRU eviction"""
        
        # Remove oldest if cache full
        if len(self.cache) >= self.cache_size:
            oldest = self.access_order.pop(0)
            del self.cache[oldest]
            
        # Add new response
        self.cache[context_hash] = response
        self.access_order.append(context_hash)

Implemented these after a client's support bot crashed from 32K context eating 18GB VRAM. Two days debugging why responses took 45 seconds. Hierarchical context fixed it.

Version warning: I've seen context compression break in some Ollama versions due to token estimation bugs. Test your token math carefully or roll your own counting. ChromaDB eats memory if you don't clean old embeddings.

These work but they're complex. Start with OLLAMA_NUM_CTX=8192, only go advanced if simple fixes don't work.

Why I built this shit: Client's support bot was using 32K context and eating 18GB VRAM. Responses took 45 seconds, customers were pissed. Two days debugging why the GPU was melting. Hierarchical context management fixed everything - now it runs on 8GB and responds instantly.