Tabby Enterprise Deployment - Production Troubleshooting Guide

I've deployed Tabby for three different companies over the past year or so. One was a mid-size fintech with strict compliance requirements, one was a defense contractor that couldn't send code to the cloud, and one was a healthcare startup that needed HIPAA compliance. Here's what actually happens when you try to run this thing in production.

Hardware Requirements That Actually Work

The official docs say you need "8GB GPU memory minimum." That's bullshit. Here's what you actually need based on painful experience:

For 7B models (CodeLlama-7B, DeepSeek-Coder-7B):

• GPU: RTX 4070 Ti or better (around 16GB VRAM minimum, not the 8GB they claim)
• RAM: 32GB system RAM, not 16GB - the model weights plus OS overhead eat memory
• CPU: 8 cores minimum for indexing large codebases without timing out
• Storage: NVMe SSD with 500GB+ free space for model files and git repo caches

For 1B models (StarCoder-1B):

• GPU: RTX 3060 with 12GB VRAM works fine
• RAM: 16GB system RAM is actually sufficient here
• The suggestions are mediocre but it won't crash every few hours

I learned the hard way that running a 7B model on an RTX 3070 (8GB VRAM) crashes with CUDA out of memory errors every couple hours. The Docker container just dies, takes your IDE completions with it, and your developers start filing tickets about "the AI being down again."

This is why proper GPU monitoring is essential, along with understanding CUDA memory management and Docker resource limits. Many teams underestimate the importance of proper cooling and power delivery for production GPU workloads.

The Kubernetes Deployment Hell

The official Kubernetes deployment looks simple until you actually try it. Here's what the YAML doesn't tell you:

Resource limits are too low. The default config requests 4GB RAM and 2 CPU cores. That's enough to start the container, not enough to actually serve completions under load. Bump it to 16GB RAM minimum, 8 CPU cores, or watch it get OOMKilled during peak hours.

resources:
  requests:
    memory: "16Gi"  # Not 4Gi
    cpu: "8"        # Not 2
    nvidia.com/gpu: 1
  limits:
    memory: "24Gi"  # Leave headroom
    cpu: "12" 
    nvidia.com/gpu: 1

Persistent volumes need way more space. They allocate 20GB for model storage. A 7B model is 14GB compressed, but Tabby downloads multiple models, keeps git repo caches, stores indexed code, and logs everything. I hit the 20GB limit within a week. Use 100GB minimum.

The ingress configuration assumes you're running vanilla nginx. If you're using Istio, Traefik, or any other service mesh, you'll need custom annotations for WebSocket support (the chat interface needs this) and longer timeouts for model loading.

Authentication Nightmares

Enterprise means SSO integration. Tabby supports LDAP as of v0.24.0, but the setup is painful.

LDAP binding fails with cryptic errors. The logs just say "authentication failed" without telling you if it's a connection issue, wrong DN, or certificate problem. Enable debug logging (RUST_LOG=debug) to see what's actually happening.

Group membership doesn't work like you think. If your LDAP uses nested groups (most enterprises do), Tabby won't find users who are in subgroups. You need to configure the search filter manually, and it's different for Active Directory vs OpenLDAP.

Certificate validation breaks in Docker. If your LDAP server uses internal Certs, mount your CA bundle into the container at /etc/ssl/certs/ or LDAPS connections will fail with SSL errors.

Enterprise LDAP integration often requires custom certificate authorities, DNS configuration, and network policies. Consider HashiCorp Vault for secret management, and implement RBAC policies for proper access control. Many organizations also need audit logging for compliance requirements.

Memory Leaks That Kill Production

The biggest operational issue: Tabby leaks memory over time. Not badly, but enough to matter in production.

Indexing jobs don't clean up. When Tabby indexes your git repos (it re-indexes every few hours to stay current), the memory usage spikes and doesn't fully return to baseline. After a week of normal operation, I've seen containers using 28GB when they started with 8GB.

The fix is ugly but works: Set up a cron job to restart the Tabby pods every 24 hours during low usage periods (like middle of the night). Kubernetes rolling restarts work fine - your developers won't notice a 30-second interruption when they're not working, but they will notice when the whole thing crashes during standup because it ran out of memory.

Network and Proxy Hell

Most enterprises have complex network setups. Tabby needs to make outbound HTTP requests to download models from HuggingFace, but many corporate networks block this.

Model pre-loading for air-gapped environments. Download models on a machine with internet access, then transfer them:

## On internet-connected machine
docker run -v $(pwd)/models:/data tabbyml/tabby download --model CodeLlama-7B-Instruct

## Transfer the ./models directory to your production environment
## Then run with --model /data/CodeLlama-7B-Instruct

Proxy configuration is inconsistent. Set these environment variables if you're behind a corporate proxy:

HTTP_PROXY=http://proxy.company.com:8080
HTTPS_PROXY=http://proxy.company.com:8080
NO_PROXY=localhost,127.0.0.1,.company.com

But even with correct proxy settings, HuggingFace model downloads sometimes hang. The Docker healthcheck doesn't catch this - the container stays "healthy" while model loading is stuck. You need custom monitoring to detect when model loading takes longer than 10 minutes.

Monitoring and Alerting That Actually Helps

The built-in health endpoints (/health) are useless. They return 200 OK even when model loading is stuck or memory usage is through the roof.

Custom monitoring endpoints I've built:

• Memory usage over time (catch the leaks before they kill the container)
• Model loading status and duration
• Active completion requests (queue up during model loading)
• Git repository sync status (indexing can get stuck on large repos)

Prometheus metrics that matter:

• tabby_memory_usage_bytes - Track the memory leak
• tabby_model_load_duration_seconds - Catch stuck downloads
• tabby_completion_requests_total - Usage patterns
• tabby_git_sync_last_success_timestamp - Indexing health

The Shit That Actually Breaks in Production

IDE extensions randomly stop working. The VS Code extension loses connection to the Tabby server and shows "Not connected" in the status bar. Restarting VS Code fixes it, but your developers shouldn't need to restart their IDE twice a day.

This happens when the Tabby server is restarted (normal maintenance) but the extension doesn't retry the connection properly. Enable auto-reconnect in the extension settings, but it's not foolproof.

Git repository indexing gets stuck on large repos. If you have a monorepo with 500k+ lines of code, the initial indexing takes hours and sometimes times out. The container stays running but completions are garbage because it doesn't have code context.

Split large repos into multiple smaller indexes, or increase the indexing timeout to 4+ hours for the initial run. Subsequent incremental updates are much faster.

CUDA version mismatches crash silently. Your container starts fine, loads the model, then crashes with cryptic CUDA errors when someone requests a completion. This happens when your Kubernetes nodes have different NVIDIA driver versions.

Use nvidia-smi to check driver compatibility, and pin your Tabby deployment to nodes with consistent GPU drivers. NodeSelectors work for this:

nodeSelector:
  nvidia.com/gpu.driver-version: "535.86.10"

The hardest lesson learned: enterprise deployment isn't about getting it working once, it's about keeping it working when real developers use it for real work. The next section covers the specific production issues you'll hit and how to actually fix them.

After deploying Tabby for three enterprise clients, here are the deployment patterns that survive contact with real developers and real workloads.

Multi-Node Architecture for Resilience

Don't run Tabby as a single pod. I learned this the hard way when a memory leak killed the only Tabby instance at 2PM during a sprint demo. The entire engineering team lost AI completions while the PM was showing off productivity improvements to the board. Awkward as hell.

Pattern: Active-Passive with GPU Affinity

Deploy two Tabby instances on separate GPU-enabled nodes. Use Kubernetes pod anti-affinity to ensure they never run on the same node:

podAntiAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
  - labelSelector:
      matchLabels:
        app: tabby
    topologyKey: kubernetes.io/hostname

One instance handles traffic, the other stays warm as a hot spare. When the active instance crashes (it will), traffic fails over in under 30 seconds. GPU model loading is expensive, so keeping both instances warm saves you from 5-minute failover times.

Pattern: Load Balancing Multiple Instances

For teams with 20+ developers, run multiple active instances behind a load balancer. Each developer's IDE connects to a specific instance (sticky sessions), so model context stays consistent. Use nginx ingress with session affinity:

nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/session-cookie-name: "tabby-server"
nginx.ingress.kubernetes.io/session-cookie-expires: "7200"

This prevents a developer from getting completions from one model and chat responses from another, which confuses users and breaks context.

Model Selection Strategy for Enterprise

Different teams need different models based on their codebase size and performance requirements.

CodeLlama-7B-Instruct: Best balance of quality and resource usage. Handles most enterprise codebases well, understands context across large repos, decent at explaining legacy code. Needs 16GB+ VRAM but worth it for the suggestion quality.

DeepSeek-Coder-7B: Better at math-heavy code (financial calculations, data science) but weaker at web development patterns. Good choice if your team writes a lot of algorithms or ML code.

StarCoder-1B: Useful for CI/CD environments where you need fast suggestions on lightweight hardware. Quality is mediocre but it runs on older GPUs and uses minimal resources. Good for dev staging environments.

Avoiding Model Hell: Don't let different teams run different models. Pick one and standardize. I've seen companies with 5 different Tabby deployments because each team wanted their preferred model. The operational overhead absolutely isn't worth the marginal quality differences. Trust me on this one.

Model selection should be based on systematic benchmarking, hardware compatibility, and licensing requirements. Consider model quantization techniques for reducing memory usage, and evaluate fine-tuning options for domain-specific code patterns. The HuggingFace model hub provides performance metrics and community feedback to guide selection decisions.

Scaling Patterns for Large Development Teams

Pattern: Horizontal Pod Autoscaling Based on GPU Utilization

CPU-based autoscaling doesn't work for Tabby - GPU utilization is what matters. Use NVIDIA's GPU metrics for HPA:

metrics:
- type: Pods
  pods:
    metric:
      name: nvidia.com/gpu.memory.used
    target:
      type: AverageValue
      averageValue: "12Gi"  # Scale up when GPU memory hits 12GB

Pattern: Repository Sharding for Code Indexing

Large monorepos (1M+ lines) break Tabby's indexing. Split repos by service boundaries:

• Frontend team gets React/TypeScript repos indexed
• Backend team gets Go/Python service repos
• Data team gets Python/SQL analytics repos

Each Tabby instance indexes 3-5 related repositories instead of one massive monorepo. Suggestions stay relevant, indexing completes faster, and you can scale each team independently.

Pattern: Time-Based Scaling for Development Hours

Most development happens 9-5 in your timezone. Scale down to minimal instances overnight, scale up before developers start work:

## Scale to 4 replicas at 8AM weekdays
- schedule: "0 8 * * 1-5"
  replicas: 4

## Scale to 1 replica at 7PM weekdays  
- schedule: "0 19 * * 1-5"  
  replicas: 1

## Minimal weekend scaling
- schedule: "0 9 * * 6-7"  
  replicas: 1

This saves GPU costs (GPU instances are expensive) while ensuring developers have good performance during work hours.

Security Patterns for Enterprise

Network Segmentation: Run Tabby in its own namespace with NetworkPolicies that only allow traffic from developer machines and monitoring systems. Block all outbound internet access except for model downloads during maintenance windows.

- to:
  - namespaceSelector:
      matchLabels:
        name: monitoring
  - namespaceSelector:
      matchLabels:
        name: developer-tools

Secret Management: Don't put LDAP passwords or API keys in ConfigMaps. Use external secret management (HashiCorp Vault, AWS Secrets Manager) and rotate credentials every 90 days.

Audit Logging: Enable detailed request logging but don't log code snippets (defeats the privacy purpose). Log user IDs, request types, response times, and error rates. Store logs in a separate system from your main application logs.

Operational Patterns That Prevent Late Night Pages

Pattern: Proactive Memory Management

Memory leaks are inevitable. Instead of waiting for OOM kills, restart pods proactively based on memory usage:

## Cron job that restarts high-memory pods
kubectl get pods -n tabby --no-headers | while read pod _; do
  memory=$(kubectl top pod $pod -n tabby --no-headers | awk '{print $3}' | sed 's/Mi//')
  if [ "$memory" -gt 20000 ]; then  # 20GB threshold
    kubectl delete pod $pod -n tabby
  fi
done

Pattern: Health Checks That Actually Work

The default /health endpoint is useless. Create custom health checks that verify:

• Model is loaded and responding to test prompts
• Git repositories are indexed within the last 4 hours
• Memory usage is under operational thresholds
• GPU temperature is reasonable (overheating causes crashes)

Pattern: Gradual Model Updates

Don't update all Tabby instances simultaneously. Update one instance, let it run for 24 hours, monitor for issues, then update the rest. Model updates can break unexpectedly, especially with custom configurations.

Pattern: Disaster Recovery Testing

Quarterly, intentionally kill your entire Tabby deployment during low-usage hours. Verify that:

• Failover works as expected
• Developer IDEs reconnect automatically
• Model loading completes within SLA timeframes
• Persistent volumes and configs restore correctly

Document the recovery procedure because the person who deployed Tabby probably won't be available during the outage.

Effective disaster recovery requires proper backup strategies, configuration management, and automated testing. Use GitOps practices for infrastructure as code, implement chaos engineering for resilience testing, and maintain runbooks for common failure scenarios.

Cost Optimization Without Sacrificing Performance

GPU Instance Selection: A100 instances are expensive but handle larger models and more concurrent users. RTX 4090s are cheaper per VRAM GB but limited availability in cloud providers. H100s are overkill unless you're running 13B+ models.

Spot Instances for Development: Use spot instances for non-production Tabby deployments. Save 60-70% on costs for developer staging environments where occasional interruptions are acceptable.

Model Caching Strategy: Cache downloaded models in persistent volumes shared across multiple deployments. Don't download the same 7GB model file every time you recreate a pod.

Implement container registry mirroring for model artifacts, use CDN distribution for faster downloads, and consider multi-region replication for global teams. Monitor storage costs carefully as model files can consume significant space, and implement lifecycle policies for automated cleanup of outdated model versions.

The next section covers specific error scenarios and their solutions - basically the debugging guide I wish I'd had when things broke during the worst possible moments.