MCP Production Troubleshooting - AI-Optimized Reference

Critical Failure Patterns

Transport Layer Failures (60% of Production Issues)

• STDIO Buffering Hangs: Server starts, health checks pass, dies silently under real traffic
• Breaking Point: Windows WSL2 STDIO randomly hangs during log rotation
• Breaking Point: Docker containers hit stdout buffer limits causing silent failures
• Nuclear Fix: Switch to HTTP/SSE for production (STDIO is unreliable)
• Mitigation Requirements:
  • Force line buffering: export PYTHONUNBUFFERED=1
  • 30 second max timeouts for operations
  • Process supervisors (systemd, supervisor) mandatory

Authentication Failures (25% of Production Issues)

• OAuth Breaking Points: Token refresh flows break between client/server updates
• Real Impact: Asana MCP server leaked customer data across organizations due to auth boundary failures
• Migration Pain: OAuth 2.1 spec migration required but causes compatibility issues
• Time Investment: 4-6 hours for complete OAuth flow debugging

Dependency Hell (15% of Production Issues)

• Node.js Version Mismatches: Dev/prod environment differences cause runtime failures
• Container Fragmentation: 7,000+ MCP server instances with no quality control
• TypeScript SDK Conflicts: Specific library combinations cause total system failure

Security Threat Landscape (September 2025)

Active Exploitation Vectors

• SQL Injection in Anthropic SQLite Server: Reference implementation (forked 5,000+ times) has critical SQLi vulnerability
• GitHub MCP Prompt Injection: Malicious support tickets hijack AI responses through prompt injection
• Mass Scanning: 7,000+ exposed servers, half without authentication, actively being compromised
• Configuration Exposure: 50% of public MCP servers misconfigured and externally accessible

Incident Response Timeline

• Hour 1: Emergency shutdown - kubectl scale deployment mcp-server --replicas=0
• Hour 2: Check indicators of compromise (unusual tool patterns, off-hours queries)
• Hour 6: Preserve forensic evidence before cleanup
• 72-96 Hours: Complete infrastructure rebuild timeline for major breaches

Security Hardening Requirements

• Network Segmentation: Block all outbound internet connections except approved services
• Least Privilege: Database read-only where possible, non-root containers with explicit UID/GID
• Input Validation: Never trust LLM output as safe input to other systems
• Monitoring: Alert on 100+ same tool calls per minute, off-hours access, cross-tenant queries

Connection Errors (-32000 "Could Not Connect")

Root Cause Distribution

1. Server Command Path Wrong (40%): npx or python not in PATH in production
2. Port Already In Use (20%): Another process grabbed the port
3. Permission Denied (15%): User can't execute server binary
4. Dependency Missing (15%): Package not installed in production
5. Environment Variable Missing (8%): Database URLs, API keys missing
6. Firewall Blocking (2%): Network policy blocking port

5-Minute Debug Process

# Test server command directly
npx @your-org/mcp-server --transport stdio
# Check port availability
netstat -tulpn | grep :8000
# Verify environment variables
env | grep -E "(DATABASE|API|MCP)"
# Test network connectivity
curl -v https://api.github.com/zen

Container Deployment Breaking Points

Memory Limits Kill Servers Silently

• Breaking Point: MCP servers balloon to 2GB+ processing large resources
• Impact: OOMKill without warning in Kubernetes
• Solution: Set memory limits AND implement resource cleanup

Health Check Lies

• Problem: Server responds to /health but MCP protocol completely broken
• Root Cause: Health check uses different code path than actual requests
• Fix: Test actual MCP endpoints in health checks, not just HTTP responses

Log Aggregation Breaks STDIO

• Problem: Centralized logging systems don't handle STDIO transport properly
• Impact: Lose critical debugging info when most needed
• Solution: Use HTTP/SSE transport with structured logging

Nuclear Recovery Options

Emergency Shutdown Procedures

# Kubernetes Nuclear Option
kubectl delete deployment mcp-server --force --grace-period=0
kubectl delete pods -l app=mcp-server --force --grace-period=0

# Docker Nuclear Option
docker kill $(docker ps -q --filter ancestor=mcp-server)
docker system prune -a --volumes

Database Connection Reset

-- PostgreSQL: Kill all connections
SELECT pg_terminate_backend(pg_stat_activity.pid)
FROM pg_stat_activity
WHERE pg_stat_activity.datname = 'your_mcp_db'
  AND pid <> pg_backend_pid();

Scorched Earth Rebuild (4-6 Hour Timeline)

1. Evidence Preservation: Export all logs, database snapshots, configs
2. Infrastructure Destruction: terraform destroy, kubectl delete namespace
3. Clean Rebuild: Deploy from infrastructure-as-code definitions
4. Gradual Restoration: Start with single replica, add complexity incrementally

Production War Stories & Solutions

Random 503 Errors in Kubernetes

• Root Cause: Readiness probe succeeds before MCP protocol handler ready
• Impact: Kubernetes routes traffic to broken pods
• Fix: Make readiness probe test actual MCP functionality, not just HTTP

SSL Certificate Failures in Production

• Root Cause: Corporate CAs and proxy servers in production vs local
• Quick Fix: Add corporate CA certificates to container
• Never Do: Disable certificate validation (creates security holes)

Memory Leaks Under Production Load

• Root Cause: Long-lived connections accumulate state, caching never expires
• Debug Tools: heapdump for Node.js, tracemalloc for Python
• Solution: Compare heap dumps during normal vs high memory usage

"Connection Reset by Peer" Intermittent Errors

• Root Cause: Load balancer health checks killing long-lived MCP connections
• Fix: Configure load balancer timeout to match longest tool execution time
• Temp Fix: Connection retry logic with exponential backoff

Resource Requirements & Time Estimates

Nuclear Option Timelines

• Service restart: 2-5 minutes
• Configuration reset: 5-15 minutes
• Database connection reset: 10-30 minutes
• Container rebuild: 15-45 minutes
• Full infrastructure rebuild: 2-6 hours

Debug Time Investments

• STDIO transport debugging: 6+ hours (switch to HTTP/SSE instead)
• OAuth flow migration: 4-6 hours
• Kubernetes networking issues: 2-4 hours
• Memory leak investigation: 4-8 hours with proper tools

Critical Breaking Points

Performance Thresholds

• UI Breaking Point: 1000 spans makes debugging distributed transactions impossible
• Memory Limit: 2GB+ memory usage triggers OOMKill without warning
• Connection Limits: Default 1024 file descriptors too low for production load
• Query Timeout: 30+ second database queries require pagination implementation

Version Compatibility

• June 2025 Spec Change: Streamable HTTP replaced SSE, breaks backward compatibility
• SDK Version Requirements: TypeScript SDK 0.4+, Python SDK 0.3+ for new transport
• Migration Requirement: Update server first, then clients (never mix versions)

Monitoring Requirements

Alert Thresholds

• Tool Execution Anomalies: 100+ same tool calls per minute (data exfiltration)
• Error Rate Spike: >5% tool error rate indicates system problems
• Off-Hours Access: Any tool execution outside business hours
• Memory Growth: >50% memory increase over baseline

Compliance Timeline Requirements

• GDPR: 72 hours to report breach to authorities
• CCPA: Variable by state notification periods
• Documentation: Audit trails for data access, breach timeline, containment actions

Critical Resource Links

Emergency Debugging Tools

• MCP Inspector: Protocol-level debugging, isolates server vs client issues
• MCP Probe: Alternative debugging when Inspector insufficient
• Network Debug: kubectl run debug --image=nicolaka/netshoot

Production Deployment References

• AWS Lambda/ECS/EKS: AWS MCP deployment patterns
• Google Cloud Run: Containerized HTTP transport deployments
• Azure Container Instances: AKS deployment best practices
• Block Engineering: 60+ MCP servers production lessons

Security Resources

• OWASP LLM Security: LLM-specific vulnerabilities including prompt injection
• MCP Security Checklist: Community validation before production
• Trend Micro Research: SQLite server vulnerability deep dive