Moving Legacy Apps to Containers Without Breaking Everything

Let me be blunt: if you think you're going to move your 10-year-old Java monolith to Kubernetes without any downtime, you're delusional. I've been through this 50 times now, and "zero downtime" is what CTOs promise to the board while engineering deals with reality.

The Real Cost of Fucking Up

Our first major migration attempt took down checkout for 6 hours on Black Friday 2023. That cost us $2.3 million in lost sales. The vendor promised "seamless migration" - turns out their demo environment had 3 users, not 50,000 concurrent shoppers hitting the database.

Here's what actually breaks: your legacy app probably has 20 hardcoded configuration files, connects to 5 databases you forgot about, writes to /tmp, and depends on some cron job that runs every 3 weeks. None of this shows up in your "application inventory."

Things That Will Go Wrong (Not If, When)

Your load balancer will work perfectly in staging and shit the bed in production. We discovered our F5 had a 30-second timeout that only triggered under real load. Six months of testing, missed it completely.

Database connections are the worst. Your connection pooling that worked for years suddenly becomes a bottleneck when containers start spinning up and down. Plan on rewriting half your database interaction code.

That "stateless" application? It's not. It's writing session data to local files, caching user preferences in memory, and probably storing uploaded files on the local disk. I guarantee it.

The Migration Reality Check

Week 1: "This looks straightforward, should take 2-3 weeks"
Month 3: Still debugging why the containerized app uses 4x more memory
Month 6: Finally figured out the Java garbage collector settings that work in containers
Month 8: Production deployment, everything breaks, emergency rollback
Month 12: Successfully running in production, but costs 40% more than predicted

What Actually Works

Start with your newest, simplest applications first. Not because they're more important, but because you need wins to justify the budget when everything else takes 3x longer.

Never migrate databases and applications simultaneously. Pick one, get it stable, then tackle the other. We tried to be clever and do both - spent 4 months debugging synchronization issues that didn't exist when we did them separately.

Learn from others' fuckups: the Kubernetes failure stories site is basically a support group for engineers who've been burned. Browse r/kubernetes for the war stories vendors don't want you to hear. The CNCF case studies are sanitized marketing fluff, but sometimes contain useful technical breadcrumbs.

Real incident reports tell you what actually breaks: Monzo's autoscaling challenges shows how Kubernetes resource management fails under load, Spotify's migration strategies reveals the platform team pain, and Shopify's container adoption covers the database connection pool disasters everyone encounters.

Blue-green deployments are great in theory. In practice, you need double the infrastructure, which means double the costs. Most companies do it once for the demo then switch to rolling updates because nobody wants to pay for idle servers.

Your monitoring will lie to you. Kubernetes says everything's healthy while your users are getting 500 errors. Build real synthetic transactions that actually test your business logic, not just HTTP 200 responses.

The truth? Most successful migrations take 6-18 months and cost 2-3x the initial estimate. But when it works, your ops team stops getting paged at 3am, deployments become boring, and you can actually scale without buying more hardware.

Resources that actually help:

• Kubernetes the Hard Way - Skip the managed services, learn what breaks
• kubectl cheat sheet - You'll use this daily
• Helm charts - Don't write YAML from scratch like a masochist
• Official Docker docs - Actually well-written for once
• k8s troubleshooting guide - For when everything's on fire
• Container security guide - Because you'll need it eventually
• 12-Factor App methodology - Design patterns that work in containers
• Docker best practices - Avoid common Docker mistakes
• Kubernetes patterns book - Design patterns for container orchestration
• Database Migration Service - Google's comprehensive migration guide

Just don't believe anyone who promises you zero downtime on the first try.

Forget the "automated discovery tools" - they'll miss half your dependencies. Spend 2 weeks manually documenting everything:

Walk through your servers and write down:

• Every database connection (including that MySQL instance running on port 3307 for some reason)
• All the environment variables your app reads
• Where it writes logs, temp files, uploads, cache files
• Every cron job, background process, and scheduled task
• All the external APIs it calls (including that one that only works on Tuesdays)

Pro tip: Grep your codebase for hardcoded IPs, file paths, and hostnames. There are always more than you think. Use tools like ripgrep or just basic grep -r "192\.168\|localhost\|\/var\/\|\/tmp\/" . to find problems.

Run your app on a fresh VM with minimal permissions. Whatever breaks is what you need to containerize properly. This step alone will save you weeks of debugging later.

Useful inventory tools: docker-slim can analyze your containers, dive shows you what's actually in your Docker layers, and hadolint catches common Dockerfile mistakes.

Additional discovery tools: syft generates software bills of materials, grype scans for vulnerabilities, trivy provides comprehensive security scanning, and cosign handles container signing for supply chain security.

Step 2: Containerize One Thing at a Time

Don't try to containerize your entire stack at once. Pick your simplest, newest application first. You need a win to build confidence (and budget) for the harder stuff.

Start with this Dockerfile pattern:

FROM node:20-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
EXPOSE 3000
USER node
CMD ["npm", "start"]

Things that will break:

• File permissions (add RUN chown -R node:node /app)
• Missing environment variables (check your .env files)
• Can't write to /app (use /tmp for temporary files)
• Node process dies with no logs (add proper signal handling)

Test locally first. If docker run doesn't work on your laptop, it definitely won't work in production.

Step 3: Get Kubernetes Working (Good Luck)

Set up a development cluster first. Don't use production for experiments. k3s is easier than full Kubernetes for testing. kind runs Kubernetes in Docker containers on your laptop, and minikube is the classic local development option.

Minimum Kubernetes resources you need:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: your-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: your-app
  template:
    metadata:
      labels:
        app: your-app
    spec:
      containers:
      - name: app
        image: your-registry/your-app:latest
        ports:
        - containerPort: 3000
        env:
        - name: DATABASE_URL
          value: "postgresql://user:pass@db:5432/app"
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /health
            port: 3000
          initialDelaySeconds: 30
        readinessProbe:
          httpGet:
            path: /ready
            port: 3000
          initialDelaySeconds: 5

That readiness probe better actually work. If your app says it's ready but can't handle traffic, Kubernetes will send requests to broken pods and your users will get 500 errors.

Step 4: Handle the Database (This Is Where It Gets Ugly)

Database migration is where dreams die. Never migrate the database and application simultaneously unless you enjoy 3am emergency calls.

Option A: Keep the database where it is
Point your containerized app to the existing database. This works great until you need to scale and hit connection limits.

Option B: Migrate database first
Move your data to a managed database service (RDS, Cloud SQL), then containerize the app. Safer but more expensive.

Option C: Do both at once
You'll spend 4 months debugging synchronization issues that wouldn't exist if you did them separately. Don't do this.

For connection pooling, use PgBouncer for PostgreSQL, ProxySQL for MySQL, or pgpool-II for more advanced PostgreSQL features. Your database will thank you when Kubernetes starts spinning up 20 instances during deployments.

Database migration guides: Postgres migration best practices, MySQL replication setup, and MongoDB replica sets for the NoSQL crowd.

Database-as-a-Service options: AWS RDS for managed relational databases, Google Cloud SQL for PostgreSQL and MySQL, Azure Database for Microsoft environments, and PlanetScale for serverless MySQL with branching.

Step 5: Deploy to Production (Prepare for Disappointment)

Rolling updates sound great until you try them. Half your pods will be running the old version, half the new version, and somehow both will be broken differently.

Blue-green deployment reality:

• Works great for demos
• Costs 2x your infrastructure budget
• Requires duplicate databases ($$$$)
• Most companies do it once then switch to rolling updates

What actually works:

1. Deploy during maintenance windows for the first few migrations
2. Use feature flags to control new functionality
3. Have a rollback plan that you've actually tested
4. Monitor real user transactions, not just HTTP response codes

Your monitoring will lie to you. Kubernetes thinks everything is healthy while users are getting timeout errors. Build synthetic transactions that actually test your business logic.

The Hard Truth

Most migrations take 3x longer than estimated and cost 2x more than budgeted. But when it works:

• Deployments become boring (in a good way)
• Scaling doesn't require buying servers
• Your ops team stops hating you
• Recovery from failures is measured in seconds, not hours

Just don't expect it to be painless. And definitely don't promise your CEO zero downtime on the first try.

The Strangler Fig Pattern (Or: How to Slowly Strangle Yourself)

The Strangler Fig pattern sounds great in theory. You gradually replace your monolith piece by piece while keeping everything running. In practice, you'll spend 8 months building an API gateway router that becomes more complex than your original monolith.

What actually happens:

1. Identify boundaries - Turns out your 10-year-old codebase has no boundaries, everything calls everything
2. Build new services - Each "simple" service needs authentication, logging, monitoring, deployment pipelines
3. Route requests - Your router becomes a 5,000-line configuration nightmare that nobody understands
4. Debug distributed failures - Error tracking across 12 services is harder than debugging one big app
5. Legacy never dies - That "temporary" legacy code will be running 3 years from now

The reality check: We tried strangling our monolith for 18 months. Ended up with a distributed monolith that was harder to debug, impossible to test locally, and cost 3x more to run. Sometimes burning it down and starting over is actually faster than slowly strangling yourself.

Monitoring That Actually Works (Not the Pretty Dashboards)

Your monitoring strategy needs to survive the migration, not just look good in vendor demos. Focus on user-facing metrics, not internal Kubernetes health checks.

Monitoring that actually catches problems:

• Synthetic transactions that exercise your actual business logic, not just HTTP 200s
• Real user monitoring (RUM) because synthetic tests miss half the edge cases
• Database query performance - your app works, but queries take 10x longer
• Error budgets based on user impact, not technical metrics

Tools that work in real environments:

• Datadog - Expensive but comprehensive, actually correlates problems across services
• New Relic - Good APM, terrible alerting, prepare for alert fatigue
• Prometheus + Grafana - Free, flexible, requires dedicated platform team
• ELK Stack - Works great until you need to search logs during an outage and it's down too
• Jaeger for distributed tracing across microservices
• Zipkin as an alternative distributed tracing system
• OpenTelemetry for vendor-agnostic observability
• Honeycomb for high-cardinality observability data
• Sentry for error tracking and application monitoring

Alert fatigue is real. You'll get 50 alerts about pod restarts while users can't log in. Focus on business impact metrics, not infrastructure health.

Security in the Real World (Spoiler: It's Terrible)

Container security is like regular security, but with more YAML files to misconfigure. Every security scan will find 47 "critical" vulnerabilities in base images that can't be fixed.

Security reality:

• Image scanning finds problems, provides no solutions - that Alpine Linux CVE from 2019? Still not fixed
• Runtime security tools generate false positives constantly
• Network policies break everything initially, get disabled "temporarily" for 6 months
• RBAC is configured by trial and error until something works
• Secret management everyone knows the database password is in the environment variables

What actually secures your system:

1. Regular updates of base images (automate this or you'll never do it)
2. Least privilege for service accounts (not humans, those need admin)
3. Network segmentation at the cloud provider level, not just Kubernetes
4. Backup your secrets because when the secret store dies, you're fucked

Post-Migration: When the Bills Come Due

"Cloud native" doesn't automatically mean cheaper. Your AWS bill will double in the first 6 months as you figure out rightsizing.

Cost optimization reality:

• Right-sizing takes 6 months of production data to get right
• Auto-scaling scales up fast, down slow, usually costs more than fixed capacity
• Spot instances work great until your batch jobs disappear mid-processing
• Resource quotas prevent your staging environment from costing more than production

Hidden costs of "success":

• Managed databases cost 4x self-hosted
• Load balancers are $20/month each, you'll have 12 of them
• Container registry costs scale with your Docker image addiction
• Data transfer between availability zones adds up fast

The 80/20 rule: 80% of your costs come from 20% of your resources. Find that 20% first.

Disaster Recovery: When Your Cloud Goes Down

Multi-region deployments sound great until you realize your database doesn't replicate across regions and your users' data is stuck in us-east-1 when it dies.

Real disaster scenarios:

• Region failure - Your app works, your database doesn't
• Cluster upgrade fails - Kubernetes 1.28 breaks your ingress controller
• Certificate expiration - Everything dies at 3am on a Saturday
• Vendor lock-in - Can't migrate off AWS because of 47 managed services
• Human error - Someone deleted the wrong namespace (yes, this happens)

What actually works for DR:

1. Test your backups monthly, not when you need them
2. Document the nuclear option - how to recreate everything from scratch
3. Practice failover during business hours when people are available
4. Have a rollback plan that doesn't depend on the system you're fixing

Truth: Most "disaster recovery" is just "restore from backup and hope it works." Plan accordingly.

The Hard Truth About Migration Success

Your migration isn't done when you turn off the old servers. It's done when your team stops being constantly paged about container orchestration issues and can focus on actual features again.

Budget 18 months from "working in containers" to "stable in production." The first 6 months after go-live will be the hardest of your career.

Additional survival resources:

• SRE Workbook - Google's lessons learned, skip to the failure stories
• AWS Well-Architected Framework - Boring but comprehensive
• 12-Factor App - Still relevant, especially for containerization
• CNCF Landscape - Tool overwhelm in visual form
• Kubernetes operators - Because managing databases manually is for masochists