Tabby: Self-Hosted AI Code Completion - Technical Reference

Core Value Proposition

• Primary Function: Self-hosted GitHub Copilot alternative that keeps code local
• Key Differentiator: Zero data transmission to external servers vs. cloud alternatives
• Community Validation: 32k GitHub stars with active bug fixes vs. feature requests

Configuration That Actually Works

Docker Deployment (Recommended Path)

# NVIDIA GPU (Production Path)
docker run -it --gpus all -p 8080:8080 -v $HOME/.tabby:/data \
  registry.tabbyml.com/tabbyml/tabby serve --model StarCoder-1B --device cuda

# CPU-Only (Emergency Fallback)
docker run -it -p 8080:8080 -v $HOME/.tabby:/data \
  registry.tabbyml.com/tabbyml/tabby serve --model StarCoder-1B --device cpu

Hardware Requirements (Real-World)

Model Size	Listed VRAM	Actual VRAM	Performance	Use Case
1B (StarCoder-1B)	2-4GB	8GB minimum	Better than nothing	Testing setup
7B (CodeLlama-7B)	8GB	14GB minimum	Comparable to early Copilot	Production viable
13B+	16GB	24GB+ (RTX 4090/A100)	Approaches current cloud tools	Enterprise

Critical Warning: Documentation understates VRAM requirements by 50-100% due to CUDA overhead

Failure Scenarios and Solutions

Docker GPU Integration Failures

• Error: docker: Error response from daemon: could not select device driver
• Root Cause: Missing NVIDIA Container Toolkit
• Solution: Install NVIDIA Container Toolkit
• Time Investment: 30 minutes to 2 hours depending on system state

Memory Exhaustion Patterns

• Symptom: Cryptic CUDA out-of-memory errors
• Cause: Model overhead + OS + other applications exceeding available VRAM
• Mitigation: Add 4GB buffer to all listed requirements
• Prevention: Monitor GPU memory before deployment

Platform-Specific Breaking Points

• Windows: Docker Desktop WSL2 integration randomly fails requiring full reinstall
• CUDA Mismatches: Container expects CUDA 11.x but drivers are 12.x (or vice versa)
• Port Conflicts: Default port 8080 often occupied, change to 8081+

IDE Integration Quality Matrix

IDE	Extension Quality	Setup Complexity	Maintenance Burden
VS Code	Excellent	2 minutes	Minimal
JetBrains	Functional but janky	5 minutes	Moderate
Neovim	Requires lua configuration	30+ minutes	High
Eclipse	Minimal support	Variable	High

Recommendation: Use VS Code for primary development, treat others as secondary

Performance Reality vs. Marketing

Actual Speed Improvements

• Marketing Claim: 55% faster coding
• Real-World Result: 10-20% improvement maximum
• Quality Threshold: Requires 7B+ models for meaningful assistance
• Hardware Dependency: RTX 4070+ for acceptable response times

Codebase Integration

• Initial Indexing: 2-3 hours for large repositories
• Context Understanding: Actually parses internal APIs and patterns
• Advantage Over Generic Tools: Knows project-specific function names and conventions

Cost Analysis vs. Alternatives

Break-Even Analysis

• Tabby: Free software + hardware costs
• GitHub Copilot: $10-19/month/user
• Break-Even Point: 10-20 team members (hardware vs. subscription costs)

Hidden Costs

• Setup Time: 30 minutes to 3 hours initial configuration
• Maintenance Burden: No 24/7 support, requires in-house GPU troubleshooting expertise
• Cloud GPU Alternative: $1-3/hour for adequate performance
• Expertise Requirement: Docker + GPU drivers + CUDA knowledge mandatory

Decision Criteria Matrix

Factor	Use Tabby	Use Cloud Alternative
Legal IP Restrictions	✓ Required	✗ Blocked
Team Size	10+ members	<10 members
Technical Expertise	High (Docker/GPU)	Low (install extension)
Budget Preference	High upfront, low ongoing	Low upfront, recurring
Internet Dependency	Offline capable	Requires connectivity

Production Deployment Considerations

Enterprise Requirements

• Monitoring: Prometheus integration needed
• Authentication: LDAP integration for SSO
• Load Balancing: Nginx for teams >20 developers
• Backup Strategy: Docker volume management
• Security Hardening: Kubernetes deployment recommended

Operational Costs

• GPU Infrastructure: $500-2000/month cloud costs
• Maintenance Overhead: Dedicated DevOps resources required
• Scaling Complexity: Manual capacity planning vs. automatic cloud scaling

Critical Warnings

What Documentation Doesn't Tell You

1. Memory Requirements: Official specs are 50-100% understated
2. Windows Compatibility: WSL2 integration breaks unpredictably
3. Model Performance: 1B models produce poor completions, 7B minimum for production
4. Support Reality: Community support only, no enterprise SLA

Breaking Points

• UI Failure: Interface becomes unusable with large distributed transactions
• CUDA Version Lock-in: Version mismatches cause complete failure
• Docker Desktop: Random WSL2 failures require full reinstall cycle

Migration Considerations

From Cloud Solutions

• Data Migration: No cloud data to migrate (privacy benefit)
• Workflow Disruption: 1-2 week team adaptation period
• Feature Parity: Always behind bleeding-edge cloud models
• Infrastructure Burden: Shifts from vendor to internal team

Alternative Self-Hosted Solutions

• Continue.dev: More LLM provider options, less setup
• Codeium On-Premises: Enterprise-focused, higher cost
• Tabby Advantages: Better documentation, more active development, stronger community

Resource Requirements

Time Investment

• Initial Setup: 30 minutes (ideal) to 3 hours (typical)
• Team Training: 1-2 weeks adaptation period
• Maintenance: Ongoing GPU troubleshooting expertise required

Expertise Requirements

• Mandatory: Docker, GPU drivers, CUDA basics
• Recommended: Kubernetes for production, monitoring setup
• Optional: Model fine-tuning, custom integrations

Success Criteria

Technical Metrics

• Response Time: <2 seconds for completions (7B+ models)
• Uptime: 99%+ (requires proper monitoring)
• Memory Utilization: <80% peak VRAM usage

Business Metrics

• Cost Efficiency: Break-even at 10-20 team members
• Legal Compliance: Zero external data transmission
• Developer Adoption: >80% daily usage rate indicates success