Tabby Enterprise Deployment: Production Technical Reference

Hardware Requirements - Production Reality

7B Models (CodeLlama-7B, DeepSeek-Coder-7B)

Minimum Production Configuration:

• GPU: RTX 4070 Ti or better (16GB VRAM minimum)
• RAM: 32GB system RAM
• CPU: 8 cores minimum
• Storage: NVMe SSD with 500GB+ free space

Critical Warning: Official documentation claims 8GB VRAM works - this causes CUDA out of memory crashes every 2 hours in production.

1B Models (StarCoder-1B)

Functional Configuration:

• GPU: RTX 3060 with 12GB VRAM
• RAM: 16GB system RAM sufficient
• Performance trade-off: Fast but mediocre suggestion quality

Failure Scenarios

• RTX 3070 (8GB VRAM) with 7B models = guaranteed crashes every few hours
• Default 4GB RAM allocation = OOMKilled during peak hours
• Model weights + OS overhead consume significantly more memory than documented

Kubernetes Production Configuration

Resource Requirements That Work

resources:
  requests:
    memory: "16Gi"  # Not 4Gi default
    cpu: "8"        # Not 2 default
    nvidia.com/gpu: 1
  limits:
    memory: "24Gi"  # Leave headroom for memory leaks
    cpu: "12" 
    nvidia.com/gpu: 1

Storage Requirements

• Default 20GB persistent volume = insufficient
• Production requirement: 100GB minimum
• Breakdown: 14GB compressed model + multiple models + git caches + indexed code + logs

Critical Deployment Issues

Pod Anti-Affinity Required:

podAntiAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
  - labelSelector:
      matchLabels:
        app: tabby
    topologyKey: kubernetes.io/hostname

Ingress Configuration:

• Standard nginx configurations fail with Istio/Traefik
• WebSocket support required for chat interface
• Extended timeouts needed for model loading

Memory Leak Management

Root Cause

• Indexing jobs don't clean up properly after completion
• Memory usage increases from 8GB baseline to 28GB+ over one week
• Container stays "healthy" while consuming excessive memory

Production Solution

Automated Rolling Restarts:

# Cron job for proactive memory management
kubectl get pods -n tabby --no-headers | while read pod _; do
  memory=$(kubectl top pod $pod -n tabby --no-headers | awk '{print $3}' | sed 's/Mi//')
  if [ "$memory" -gt 20000 ]; then  # 20GB threshold
    kubectl delete pod $pod -n tabby
  fi
done

Schedule: Every 24 hours during off-hours
Impact: 30-second interruption prevents production crashes

Authentication Configuration

LDAP Integration Issues

Common Failures:

• Cryptic "authentication failed" errors without root cause indication
• Nested group membership not detected (most enterprises use nested groups)
• Certificate validation failures with internal CAs

Required Configuration:

# Enable debug logging for troubleshooting
RUST_LOG=debug

# Certificate handling for internal CAs
# Mount CA bundle to /etc/ssl/certs/ in container

Group Search Requirements:

• Manual search filter configuration required for Active Directory vs OpenLDAP
• Nested groups require custom filter configuration

Network Configuration

Air-Gapped Deployment

Model Pre-loading Process:

# Internet-connected machine
docker run -v $(pwd)/models:/data tabbyml/tabby download --model CodeLlama-7B-Instruct

# Transfer models directory to production
# Run with --model /data/CodeLlama-7B-Instruct

Proxy Configuration

Required Environment Variables:

HTTP_PROXY=http://proxy.company.com:8080
HTTPS_PROXY=http://proxy.company.com:8080
NO_PROXY=localhost,127.0.0.1,.company.com

Known Issue: HuggingFace model downloads hang despite correct proxy settings
Detection: Custom monitoring required - downloads taking >10 minutes indicate failure

Production Failure Modes

IDE Extension Connectivity

Failure Pattern: "Not connected" status in VS Code extension
Root Cause: Extension doesn't retry connections after server restarts
Solution: Enable auto-reconnect in extension settings (not foolproof)

Repository Indexing Failures

Failure Threshold: 500k+ lines of code
Symptoms: Initial indexing times out, completions become generic
Solutions:

• Split large repos into multiple smaller indexes
• Increase indexing timeout to 4+ hours for initial run
• Incremental updates much faster after initial completion

CUDA Version Mismatches

Failure Pattern: Container starts, loads model, crashes on first completion request
Root Cause: Different NVIDIA driver versions across Kubernetes nodes
Solution:

nodeSelector:
  nvidia.com/gpu.driver-version: "535.86.10"

Monitoring Configuration

Critical Metrics

Essential Prometheus Metrics:

• tabby_memory_usage_bytes - Track memory leak progression
• tabby_model_load_duration_seconds - Detect stuck downloads
• tabby_completion_requests_total - Usage patterns
• tabby_git_sync_last_success_timestamp - Indexing health

Health Check Requirements

Built-in /health endpoint limitations: Returns 200 OK during failures
Custom health checks must verify:

• Model loaded and responding to test prompts
• Git repositories indexed within 4 hours
• Memory usage under operational thresholds
• GPU temperature within safe ranges

Scaling Patterns

Multi-Node Architecture

Pattern: Active-Passive with GPU Affinity

• Two instances on separate GPU nodes
• One active, one warm standby
• Failover time: <30 seconds vs 5-minute cold start

Load Balancing Configuration

For 20+ developers:

nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/session-cookie-name: "tabby-server"
nginx.ingress.kubernetes.io/session-cookie-expires: "7200"

Repository Sharding Strategy

Large monorepo handling:

• Split by service boundaries rather than single massive repo
• 3-5 related repositories per Tabby instance
• Prevents indexing timeouts and improves suggestion relevance

Model Selection Matrix

Model	VRAM Required	RAM Required	Use Case	Quality
CodeLlama-7B-Instruct	16GB	32GB	General enterprise	Best balance
DeepSeek-Coder-7B	16GB	32GB	Math/ML heavy	Good for algorithms
StarCoder-1B	12GB	16GB	CI/CD, staging	Fast but mediocre

Enterprise Standardization: Choose one model per organization to avoid operational complexity

Cost Optimization

Instance Selection Strategy

• A100: Expensive but handles larger models and more concurrent users
• RTX 4090: Cheaper per VRAM GB but limited cloud availability
• H100: Overkill unless running 13B+ models

Development Environment Cost Reduction

• Spot instances: 60-70% cost savings for non-production
• Time-based scaling: Scale down overnight, up before work hours
• Model caching: Shared persistent volumes prevent repeated downloads

Security Implementation

Network Segmentation

- to:
  - namespaceSelector:
      matchLabels:
        name: monitoring
  - namespaceSelector:
      matchLabels:
        name: developer-tools

Audit Requirements

• Log user IDs, request types, response times, error rates
• Do NOT log code snippets (defeats privacy purpose)
• Store audit logs separate from application logs
• Enable detailed request logging for compliance

Disaster Recovery

Recovery Testing Requirements

Quarterly testing protocol:

1. Intentionally kill entire Tabby deployment during low usage
2. Verify failover works within SLA
3. Confirm IDE auto-reconnection
4. Validate model loading completes within timeframes
5. Test persistent volume and config restoration

Recovery Time Objectives

• Failover: <30 seconds with warm standby
• Cold start: 5-10 minutes depending on model size
• Full disaster recovery: <1 hour with proper automation

Critical Warning Indicators

Immediate Action Required

• Memory usage >20GB per container
• Model loading duration >10 minutes
• Git sync timestamp >4 hours old
• GPU temperature >80°C
• CUDA out of memory errors in logs

Upgrade Risk Factors

• Config format changes between versions without migration scripts
• Container crashes during model loading with insufficient resources
• Single-node deployments have guaranteed downtime during updates

Alternative Comparison Matrix

Solution	Setup Time	Maintenance	Data Privacy	Cost (50 devs/month)
Tabby Self-Hosted	1-2 weeks	High	Complete	$5-15K
GitHub Copilot Enterprise	30 minutes	Zero	Microsoft servers	$1,950
Sourcegraph Cody	3-5 days	Medium	Sourcegraph servers	$10-20K
Amazon CodeWhisperer	1-2 hours	Low	AWS processing	$950

Production Readiness Checklist

Pre-Deployment Requirements

• GPU nodes with consistent driver versions
• 100GB+ persistent storage per instance
• Network policies configured
• LDAP/SSO integration tested
• Custom health checks implemented
• Monitoring and alerting configured

Post-Deployment Validation

• Memory leak monitoring active
• Rolling restart automation configured
• IDE extension connectivity verified across team
• Repository indexing completed successfully
• Failover procedures tested
• Audit logging compliance verified

Ongoing Maintenance Tasks

• Weekly memory usage review
• Monthly model update evaluation
• Quarterly disaster recovery testing
• Semi-annual security review
• Annual hardware capacity planning