Currently viewing the human version
Switch to AI version

AWS Compute Optimizer: Finally, RDS Sizing That Doesn't Require Guesswork

AWS Compute Optimizer Dashboard

AWS finally released Compute Optimizer support for databases. About fucking time. I've been sizing RDS instances based on "double whatever we think we need and pray" for years. This tool actually analyzes your workload patterns instead of making you guess.

How Compute Optimizer Actually Works (And Why You Should Care)

I tested this on a production PostgreSQL instance that was costing us $800/month because we over-provisioned it after getting burned by a slow query incident in 2023. Compute Optimizer analyzed our CloudWatch Database Insights metrics for two weeks and told us we could drop to a smaller instance and save $300/month without performance impact.

The key thing Compute Optimizer checks is memory usage because that's usually what kills database performance. If your RDS instance is swapping to disk (os.swap.in > 0), your database is about to turn into an expensive paperweight. I learned this the hard way when our main app database started hitting swap during traffic spikes and response times went from 50ms to 8 seconds.

It also looks at connection patterns over time. Our e-commerce site has predictable spikes every Monday morning when people place orders after the weekend, but the tool caught that pattern and recommended sizing for the baseline load, not the peaks.

What Compute Optimizer Actually Tells You

Optimized Instances: Rare. Usually means you actually sized it correctly, which never happens on the first try.

Over-Provisioned Instances: The expensive mistakes. I had a MySQL instance running on r6i.2xlarge ($450/month) that was using 15% CPU and 30% memory. Compute Optimizer recommended dropping to r6i.large and saved us $200/month. Common findings include "CPU over-provisioned" which basically means "you're paying for cores you're not using."

Under-Provisioned Instances: The 3am pager alerts. These show "Memory under-provisioned" which means your database is struggling and probably hitting swap. Fix these immediately or enjoy explaining to your manager why the site was slow during peak hours.

Graviton Migration: Cheaper Instances That Actually Work

Compute Optimizer also identifies AWS Graviton migration opportunities. I was skeptical of ARM-based instances because changing architectures usually breaks something, but RDS abstracts away the complexity.

The numbers are real: we migrated a PostgreSQL instance from r6i.xlarge to r6g.xlarge and saved 25% on the monthly bill with the same performance. The migration took 5 minutes during a maintenance window and our application didn't notice the difference.

Important: Only works for MySQL, PostgreSQL, and Aurora. SQL Server and Oracle are still stuck on x86. The migration is literally just changing the instance type - your connection strings, drivers, and application code stay exactly the same.

Storage Recommendations That Don't Suck

Compute Optimizer also analyzes your storage usage, which matters because AWS storage pricing is designed to confuse you into overpaying.

gp2 to gp3 Migration: This is a no-brainer. gp3 storage gives you 3,000 IOPS baseline regardless of volume size, while gp2 gives you 3 IOPS per GB. So a 100GB gp2 volume gets 300 IOPS, while gp3 gets 3,000 IOPS for 20% less money. I've never found a reason not to migrate.

Aurora I/O Costs: Aurora billing is fucking complicated. You pay per I/O operation on standard Aurora, but Aurora I/O-Optimized gives you predictable hourly pricing. If you're doing more than 1.5M I/O operations per hour, I/O-Optimized saves money.

Provisioned IOPS Waste: If you're using io1/io2 storage, Compute Optimizer will tell you if you're paying for 10,000 IOPS but only using 2,000. I had one instance provisioned for 8,000 IOPS because we thought we needed it for peak load, but we were averaging 1,200 IOPS. Dropped it to 2,000 IOPS and saved $200/month.

Finding Your Expensive Mistake Databases

The idle detection actually works, which surprised me. It found a MySQL instance I forgot about from a project that got cancelled 6 months ago. The instance was just sitting there burning $180/month with zero connections.

It checks for:

  • Zero connections for weeks - Pretty obvious sign nobody needs this database
  • No read/write activity - Even connection pools usually generate some I/O
  • Read replica confusion - Sometimes you forget which replica is actually being used

For Aurora: Compute Optimizer suggests Aurora Serverless v2 for idle databases. This pauses the compute when there's no activity, so you only pay for storage. Saved us $300/month on a staging Aurora cluster that only gets used during business hours.

The AWS Cost Dashboards

Cost Optimization Hub shows all your Compute Optimizer recommendations in one place across multiple accounts. Useful if you have dozens of RDS instances and can't remember which ones are expensive.

The CORA Dashboard is better for executives who want pretty charts to show how much money they're saving. I prefer the raw recommendations in Cost Explorer.

How to Actually Use This Without Getting Fired

Don't Ignore Reserved Instances: If you turn on savings estimation mode, Compute Optimizer won't recommend changes that waste your existing RIs. I made this mistake once and got yelled at for suggesting we move off an instance type we had 3 years of reservations for.

Start Small: Test recommendations on dev/staging first. I don't care how confident AWS is about "Low risk" - changing production instance types during business hours is how you end up in a very uncomfortable meeting with leadership.

Plan for Maintenance Windows: Instance type changes require reboots. Plan for 5-15 minutes of downtime depending on your setup. Multi-AZ helps but doesn't eliminate the reboot.

Compute Optimizer isn't perfect, but it's the first AWS tool that actually analyzes real usage patterns instead of just showing you graphs. After 6 months of using it, I've saved about $1200/month across our RDS instances without any performance issues.

The recommendations are usually conservative, which I appreciate. It's better to save 20% with confidence than risk a production incident trying to save 40%.

But here's the thing: rightsizing your instances only gets you so far. You can have the perfectly sized database server and still watch it collapse under connection pressure during traffic spikes. That's where connection management becomes the make-or-break factor in RDS performance.

Why Your App Dies During Traffic Spikes: Connection Management Reality

RDS Proxy Connection Pooling Architecture

Connection issues will ruin your day faster than a bad deployment. I've seen apps that run fine in dev completely shit the bed in production because nobody thought about connection pooling. Here's what actually works instead of what the AWS docs pretend is simple.

How Connection Limits Will Fuck You Over

PostgreSQL on RDS defaults to 100 connections on smaller instances. Sounds like plenty until you deploy a microservice architecture with Lambda functions that each grab their own connection. I learned this the hard way when our Black Friday traffic took down the entire checkout flow with FATAL: remaining connection slots are reserved for non-replication superuser connections errors.

The real killer isn't running out of connections - it's the overhead of constantly creating new ones:

  • Authentication takes forever: Every new connection has to verify credentials, establish SSL, check permissions
  • Memory gets eaten alive: Each connection reserves buffers and caches
  • CPU burns on housekeeping: Creating and destroying connections instead of processing queries
  • Connection storms: When one service restarts, 50 instances all try to reconnect simultaneously

I've watched perfectly good databases melt down because the app was spending more time managing connections than running SQL.

RDS Proxy: Actually Useful Connection Bouncer

Amazon RDS Proxy is basically a connection bouncer for your database. It's not magic, but it'll save your ass when Lambda functions try to open 1,000 connections simultaneously. I was skeptical until I tested it on a production workload that was burning through connections like crazy.

Smart Pool Scaling: RDS Proxy tries to be smart about connection pooling. Most of the time it works - it keeps more connections open during busy periods and scales down when things are quiet. Sometimes it doesn't figure out your traffic patterns and you'll spend hours debugging why connections are getting dropped.

Failover That Doesn't Suck: Normal RDS failovers take 1-2 minutes while your app figures out the database is dead and reconnects. With RDS Proxy, failovers drop to 10-15 seconds because the proxy already knows which database is healthy. This alone paid for itself when our primary RDS instance decided to shit the bed during peak hours.

Connection Sharing Magic: Here's the clever part - RDS Proxy can run multiple database transactions over the same physical connection. Your app thinks it has its own connection, but the proxy is secretly sharing connections behind the scenes. Works great until your app does something weird that breaks the sharing.

RDS Proxy Configuration: Where Things Get Weird

Connection Pinning Will Ruin Your Day: RDS Proxy automatically pins connections when your app does anything that breaks connection sharing. This defeats the whole point of using a proxy:

  • Prepared statements that change session variables
  • Creating temp tables (huge performance killer)
  • Setting session-level variables
  • Long-running transactions that lock tables

I spent a week debugging why RDS Proxy wasn't helping our performance, only to find our ORM was creating temp tables for every complex query. Switched to a different query pattern and suddenly connection sharing started working.

Multi-AZ Target Groups: Set up target groups across multiple AZs so when one AZ dies, your proxy doesn't take everything down with it. Learned this during the great us-east-1 outage of 2023 when our proxy was only configured for a single AZ.

Secrets Manager Integration: RDS Proxy works with AWS Secrets Manager to rotate database passwords automatically. Nice feature until you realize it adds 10-20ms latency to every connection because it has to fetch credentials. Fine for most apps, but if you're doing high-frequency trading, that latency will cost you money.

When RDS Proxy Isn't Worth the Hassle

Connection Pooling Comparison Architecture

Application-Level Pooling: Sometimes old-school client-side connection pools work better than RDS Proxy, especially if your app has predictable patterns:

  • PostgreSQL: pgbouncer is bulletproof. I've run it in production for years without issues. Set it to transaction mode and watch your connection problems disappear.
  • MySQL: MySQL Connector pools built into most frameworks. Works fine if you configure it properly instead of using defaults.
  • Java: HikariCP is the gold standard. Fast, reliable, and the maintainer actually responds to bug reports.
  • Python: SQLAlchemy pooling does the job, but you need to tune the pool size or it'll bite you during traffic spikes.

Sidecar Container Pooling: If you're using Kubernetes, run a connection pooler as a sidecar container. Adds complexity but gives you more control than RDS Proxy. Plus you can debug it when things go wrong instead of waiting for AWS support to tell you "it should work."

Database Parameter Tuning: The Settings That Actually Matter

Stop Using Default Parameters: The default RDS parameter groups are garbage for production workloads. I've seen too many databases struggle because nobody bothered to create a custom parameter group.

-- PostgreSQL: What I actually use in production
max_connections = 200  -- Don't go crazy, monitor your actual usage first
shared_preload_libraries = 'pg_stat_statements'  -- Essential for finding slow queries
log_connections = off  -- Turn this off unless you enjoy paying for CloudWatch logs
log_disconnections = off  -- Same deal

-- MySQL: Settings that won't kill your database
max_connections = 500  -- Start conservative, scale up based on monitoring
max_user_connections = 400  -- Leave headroom for admin connections
interactive_timeout = 300  -- Kill idle connections after 5 minutes
wait_timeout = 300  -- Same timeout for non-interactive connections

Connection Monitoring That Actually Helps: CloudWatch has the metrics you need, but you have to know what to look for:

  • DatabaseConnections: Watch this like a hawk. If you're consistently above 80% of max_connections, you're fucked during the next traffic spike.
  • Connection creation rate: High rates mean your app is churning connections instead of reusing them. Fix your connection pooling.
  • Connection saturation: When you hit max_connections, everything stops working. Set alerts at 85% or prepare for weekend emergency debugging sessions.
  • RDS Proxy metrics: Track connection pool utilization and pinning frequency. High pinning means your app is doing something that breaks connection sharing.

Connection Pool Sizing: Don't Just Guess

Start Small and Test: Begin with 10-20% of your database's max_connections. For a PostgreSQL instance with 100 max connections, start with 10-20 connections total across all your app instances. I know it sounds low, but you'd be surprised how little you actually need.

Load Testing Is Your Friend: Test with realistic traffic patterns to find the sweet spot:

  • Too small: Your app times out waiting for connections during traffic spikes. Users get angry.
  • Too large: You're wasting database memory and probably have other performance bottlenecks you haven't found yet.
  • Just right: Consistent response times even when traffic doubles unexpectedly.

Auto-Scaling Will Bite You: If your app auto-scales, your connection pools need to scale too. I've seen setups where 5 app instances with 4 connections each suddenly become 20 instances with 4 connections each, overwhelming the database. Plan for this or enjoy explaining to your boss why the site went down during your biggest traffic day.

Lambda: The Connection Nightmare Generator

Lambda Hates Your Database: Every Lambda invocation wants its own database connection. Scale to 1000 concurrent Lambda functions and watch your RDS instance collapse under connection pressure. I've debugged this exact scenario more times than I care to remember.

RDS Proxy Saves Lambda: RDS Proxy actually works well with Lambda because it maintains a persistent connection pool that Lambda functions can borrow from. Without it, your Lambda functions spend more time connecting to the database than actually running queries.

Lambda Connection Best Practices: Design your Lambda functions to play nice with connection pooling:

  • Keep database operations simple - one query per Lambda invocation if possible
  • Don't start long-running transactions that hold connections hostage
  • Handle database errors properly so connections get returned to the pool instead of leaking

Debugging Connection Problems: Where to Look When Everything's On Fire

Performance Insights is Your Friend: Performance Insights actually shows you what's happening with your connections:

  • Top SQL statements: Find the queries that are hogging connections. Usually it's some developer who wrote a query that takes 30 seconds to run.
  • Wait events: Shows you what's blocking your database. Connection waits mean you're hitting connection limits.
  • Database load: High load with low connection count means your queries suck. High load with high connection count means you have connection pooling problems.

RDS Proxy Monitoring: Keep an eye on these metrics or RDS Proxy will fail you when you need it most:

  • Proxy CPU utilization: If the proxy itself is pegged at 100% CPU, you've found your bottleneck.
  • Target health: Make sure the proxy can actually reach your database. Sounds obvious but I've seen proxy health checks fail for hours.
  • Query duration: Compare performance with and without proxy. If proxy makes things slower, you're probably hitting connection pinning issues.

Connection optimization is one of those things that either saves your ass or you never think about it. Get it right and your database can handle traffic spikes. Get it wrong and your site goes down at the worst possible moment. Choose wisely.

The truth is, even with perfect connection management and proper instance sizing, you'll still run into performance problems. That's because database optimization is like peeling an onion—fix one layer and you discover the next bottleneck. Here are the questions I get asked most often when people hit these next-level performance issues.

Performance Troubleshooting: Questions Every RDS User Asks

Q

Why is my RDS instance pegged at 100% CPU but my queries are simple as fuck?

A

It's almost always connection churn. Your app is probably creating and destroying database connections like it's going out of style. I spent 3 hours debugging "slow queries" once before realizing our Lambda functions were opening new connections for every request. The database was spending 80% of its CPU on connection setup instead of actual queries.

How to diagnose: Check CloudWatch DatabaseConnections metric. If you see it spiking every few seconds while CPU usage mirrors it, your app is burning CPU on connection overhead instead of actual work.

The fix: Use RDS Proxy for serverless workloads or pgbouncer for traditional apps. Start with 10-20 connections per app instance and adjust from there. Yes, it's another thing to manage, but it's better than explaining to your boss why checkout is timing out.

Q

My database is fine during the day but turns to shit at night during backups. What gives?

A

Backups are stealing your I/O. RDS backups read through your entire database, and if you're on gp2 storage with limited IOPS, it competes with everything else trying to use the disk.

We had this exact problem on a 500GB PostgreSQL instance. During the day: sub-50ms query times. At 3am during backup: 2-second query timeouts and angry customers in Europe. The backup window was reading 500GB while our monitoring queries were trying to run.

How to fix it:

  • Migrate to gp3 storage - gives you 3,000 IOPS baseline instead of 3 per GB
  • Move your backup window to when you actually have low traffic (not AWS's default)
  • If you're on a tiny instance, consider io2 storage but it's expensive
Q

How do I know if I should upgrade my RDS instance size?

A

Use AWS Compute Optimizer first: The recent Compute Optimizer updates provide ML-driven recommendations based on actual workload patterns, not guesses.

Manual evaluation: Check these CloudWatch metrics over 2-4 weeks:

  • CPU utilization > 80% sustained: Consider CPU upgrade
  • FreeableMemory < 15% of total: Memory constraint
  • ReadIOPS + WriteIOPS approaching instance limits: I/O bottleneck
  • DatabaseConnections near max_connections: Connection scaling issue

Right-sizing strategy: Upgrade one constraint at a time. If CPU is high but memory is fine, try compute-optimized instances (c6i) before memory-optimized (r6i).

Q

Why is my Aurora cluster slower than regular RDS for write-heavy workloads?

A

Aurora's storage architecture trade-off: Aurora's distributed storage layer optimizes for durability and read scaling, but adds latency to write operations due to the 6-way replication across AZs.

Write optimization strategies:

  • Use Aurora Optimized Writes for reduced write latency
  • Batch write operations when possible to amortize commit overhead
  • Consider regular RDS for write-intensive, OLTP workloads where write latency is critical

When Aurora makes sense: Read-heavy workloads, auto-scaling storage requirements, global database replication needs.

Q

My queries were fast yesterday but slow today. What changed?

A

Statistics staleness: Database query planners rely on table statistics to choose optimal execution plans. Stale statistics lead to poor query plans.

Immediate fixes:

  • PostgreSQL: Run ANALYZE on affected tables or enable autovacuum in parameter groups
  • MySQL: Run ANALYZE TABLE table_name or enable innodb_stats_auto_recalc

Prevention: Ensure autovacuum (PostgreSQL) or automatic statistics updates (MySQL) are enabled in parameter groups.

Q

Should I use RDS Proxy for everything or is it just more AWS bullshit?

A

RDS Proxy is actually useful, but not everywhere. It adds about 1-2ms latency and costs an extra $0.015 per hour per 1000 connections. Do the math before you enable it everywhere.

Use it for:

  • Lambda functions - This is where it shines. Lambda + RDS without proxy = connection exhaustion hell
  • Microservices that can't be bothered to implement proper connection pooling
  • Legacy apps where you can't modify the connection logic

Skip it for:

  • Apps that already use pgbouncer or similar (you're just adding another layer)
  • High-frequency trading or anything where 1-2ms actually matters
  • Single monolithic apps with predictable load patterns

I tried using RDS Proxy for everything once and our AWS bill went up $200/month. It's useful but not free.

Q

How can I stop AWS from bleeding me dry on storage costs?

A

Start with the obvious shit:

  1. gp2 → gp3: This is a no-brainer. 20% cheaper for better performance. I've never found a reason not to migrate.
  2. io1 → io2: Same features, better price. Just do it.
  3. Question if you need io2: Most people don't need 16,000+ IOPS. We had one instance provisioned for 10,000 IOPS that was averaging 800. That's $650/month of wasted IOPS.

Stop the surprise bills:

  • Turn on storage autoscaling unless you enjoy explaining to your boss why the database is down at 2am
  • Monitor actual storage usage. We had a 1TB instance that was using 200GB for 18 months because nobody bothered to check
  • Aurora I/O billing is confusing as hell. I/O-Optimized costs more upfront but can save money if you're doing lots of I/O operations
Q

My RDS instance memory usage keeps growing. Is this normal?

A

Yes, for the most part. Database engines cache frequently accessed data in memory for performance. High memory usage often indicates effective caching, not a problem.

When to worry:

  • Swap usage > 0: Database is using disk for memory operations (performance killer)
  • Memory growth + performance degradation: Potential memory leak in application connections
  • Out of memory errors: Instance is genuinely undersized

Investigation steps:

  1. Check CloudWatch SwapUsage metric
  2. Review Performance Insights for memory-intensive queries
  3. Analyze connection patterns for potential leaks
Q

Multi-AZ vs read replicas: which one actually helps performance?

A

Multi-AZ does fuck-all for performance. It's pure disaster recovery. The standby instance just sits there burning money until your primary dies, then it takes over. Your app never talks to it during normal operation.

Read replicas can help performance if you can split your reads and writes properly in your application code. But here's the shit nobody tells you:

  • Replication lag is real: Usually 100-500ms, but I've seen it spike to 30+ seconds during heavy write periods
  • Your data might be stale: User updates their profile, immediately views it, sees old data because the read hit the replica
  • Your code gets complex: Now you need logic to decide which queries go where

Reality check: We tried read replicas for our main app and spent 2 weeks debugging weird data inconsistencies before going back to vertical scaling. Multi-AZ + bigger instance is often simpler than read/write splitting.

Q

Should I enable Performance Insights or is AWS about to kill it?

A

Yes, enable it immediately. Performance Insights is incredibly useful for finding slow queries and has minimal performance impact. It's one of the few AWS tools that actually helps you fix problems instead of just showing you fancy graphs.

The catch? Like all useful AWS tools, they keep fucking with it. They've been pushing CloudWatch Database Insights integration, which means more complexity and probably higher costs eventually.

How to use it effectively:

  • Turn it on from day one (you can't add it to encrypted instances later without a migration)
  • Stick with 7-day retention unless you're debugging complex performance issues
  • Export important queries and metrics because AWS loves to change interfaces and break your workflows

I've found more production performance issues with Performance Insights than any other AWS tool. It's actually useful, which means they'll probably fuck with it eventually.

Performance Optimization Options: What Actually Works vs. What Wastes Your Time

Strategy

Implementation Effort

Performance Impact

Cost Impact

Best For

Potential Drawbacks

AWS Compute Optimizer

Low (automated recommendations)

15-40% improvement

20-50% cost reduction

All RDS workloads

Requires 2+ weeks of data for accurate recommendations

gp2 → gp3 Storage Migration

Low (simple storage modification)

25-50% IOPS improvement

20% cost reduction

I/O bound workloads

Minimal downtime during migration

RDS Proxy Implementation

Medium (infrastructure changes)

30-60% connection efficiency

Adds $0.015/hour + data transfer

Lambda, microservices, connection-heavy apps

1-2ms latency overhead, additional cost

Graviton Instance Migration

Low (instance type change)

20-40% price-performance

20-40% cost reduction

Compute-intensive workloads

ARM compatibility validation needed

Connection Pooling (Application)

High (code changes required)

40-70% connection efficiency

No additional AWS costs

Predictable workload patterns

Development effort, testing complexity

Read Replica Scaling

Medium (read/write logic changes)

50-200% read capacity

$$ (doubles read infrastructure)

Read-heavy workloads

Replication lag, eventual consistency

Parameter Group Optimization

Medium (database expertise needed)

10-30% query performance

No cost

Database-specific bottlenecks

Risk of misconfiguration, requires testing

Query Optimization + Indexing

High (database analysis required)

100-1000% for specific queries

No additional cost

Poorly performing specific queries

Requires database expertise, ongoing maintenance

Aurora Optimized Reads

Low (instance type selection)

30-75% for temp object workloads

10-15% premium over standard

Large temporary object workloads

Limited to specific instance types

Multi-AZ → Aurora Migration

High (significant architectural change)

Variable (depends on workload)

50-100% cost increase

Auto-scaling, global replication needs

Vendor lock-in, cost complexity

Storage and Query Optimization: Where You Either Save Your Ass or Make Everything Worse

Aurora Serverless v2 Architecture

Alright, you've fixed the obvious stuff. Now comes the hard part - storage and queries. This is where you either become a hero or accidentally nuke performance for everyone. I've done both, and I can tell you which one gets you called into more meetings.

EBS Storage Types: Choose Wrong and Your Database Will Perform Like a Potato

EBS storage types matter more than AWS lets on. I've seen databases crawl because someone picked gp2 for a high-IOPS workload, then spent weeks debugging "performance issues" that were actually storage bottlenecks.

gp3 is Usually the Right Answer: General Purpose SSD (gp3) gives you 3,000 IOPS baseline regardless of volume size. gp2 gives you 3 IOPS per GB, which means a 100GB volume gets a pathetic 300 IOPS. That's why your small test databases run like garbage.

Storage Performance Reality Check:

  • gp3: 3,000 baseline IOPS, scales to 16,000. Costs 20% less than gp2. There's literally no reason to use gp2 anymore.
  • gp2: 3 IOPS per GB with burst credits that run out. When credits are gone, performance becomes unpredictable. I've seen production databases randomly slow down because someone ran a backup that ate all the burst credits.
  • io2: Up to 64,000 IOPS with predictable latency. Expensive as hell but worth it if you actually need consistent sub-millisecond response times.

The gp2 Burst Credit Trap That Will Ruin Your Day

If You're Still on gp2, You're Playing Russian Roulette: Monitor BurstBalance in CloudWatch like your job depends on it, because it does. When burst credits hit zero, your database performance makes waiting for Windows updates seem fast:

  • Queries that fly during quiet hours suddenly take 10x longer during busy periods
  • Nightly backups eat your burst credits and make the next day's performance garbage
  • Response times become completely random with no correlation to actual query complexity

I once spent three days debugging "intermittent performance issues" only to discover our gp2 volume was running out of burst credits every afternoon when the batch jobs ran. The CloudWatch graphs showed perfect CPU and memory usage, but queries would randomly time out with ERROR: canceling statement due to user request because everything was waiting for disk I/O.

IOPS vs Throughput: Know the Difference: Most people obsess over IOPS but actually hit throughput limits first. gp3 lets you tune both independently, so you can optimize for your actual workload:

  • OLTP workloads: You need IOPS for all those small, frequent transactions
  • Analytics workloads: You need throughput for those huge table scans
  • Mixed workloads: Use Performance Insights to figure out which one is actually killing you

Storage Auto-Scaling Will Save Your Ass: RDS Storage Auto Scaling prevents the 3am call that your database ran out of space. But configure it wrong and you'll get a surprise AWS bill:

-- Storage auto-scaling configuration that won't bankrupt you
-- Free storage threshold: 20% (gives you time to react, prevents frequent scaling)
-- Maximum storage limit: Set 2-3x current usage, not unlimited
-- Auto-scaling cooldown: 300 seconds minimum to prevent thrashing

-- Example for 500GB production database:
-- Current: 500GB
-- Threshold: 20% (scales when <100GB free)
-- Maximum: 1500GB (room to grow without runaway costs)
-- Cooldown: 300 seconds (prevents rapid successive scaling events)

-- Monitor: "FreeStorageSpace" CloudWatch metric
-- Alert: When free space drops below 25% manually

Aurora Storage: Pay-Per-IO vs I/O-Optimized (Choose Wisely)

Aurora Billing is More Complex Than Quantum Mechanics: Aurora I/O-Optimized changes how you pay for storage I/O. Standard Aurora charges per I/O operation, I/O-Optimized gives you predictable hourly pricing.

The Math That Matters:

  • Standard Aurora: $0.20 per million I/O operations. Cheap until it's not.
  • I/O-Optimized: 35% higher instance cost, but no I/O charges. Predictable billing.
  • Break-even point: Around 4 million I/Os per hour. Do the math for your workload or guess wrong and pay extra.

Workload Evaluation for I/O Optimization:

-- Analyze I/O patterns using Performance Insights
-- High I/O workloads: Analytics, real-time processing, frequent backups
-- Low I/O workloads: Simple CRUD operations, read-heavy applications
-- Variable I/O: Consider cost predictability vs. optimization

RDS Performance Monitoring Dashboard

RDS Performance Insights Query Analysis

Query Optimization: Fix Your Worst Queries First

RDS Optimized Reads for Complex Queries: RDS Optimized Reads helps when your queries create massive temporary objects. If you've ever run a complex JOIN that brings your database to its knees, this might save you.

When Queries Create Huge Temp Objects:

  • Complex JOINs: When your query joins 5 tables with no indexes and creates giant intermediate results
  • Sorting without indexes: ORDER BY on millions of rows without a proper index
  • High-cardinality GROUP BY: Grouping by columns with millions of unique values
  • Window functions: Analytics queries that partition and rank huge datasets

How Optimized Reads Works: Uses local NVMe storage for temp objects instead of your main EBS storage:

  • 5x faster temp object processing - but only for specific workload patterns
  • Less I/O contention on your main storage while queries run
  • Better concurrency when multiple expensive queries run simultaneously

The catch? It only works on specific instance types and costs 10-15% more. Worth it if you actually have these problems.

Instance Type Considerations: Optimized Reads is available on specific instance families:

  • r6id, r6idn: Memory-optimized instances with local NVMe storage
  • m6id, m6idn: General-purpose instances with local NVMe storage
  • Cost premium: 10-15% higher than standard instances

Parameter Groups: Stop Using Defaults and Start Using Your Brain

Default Parameters Are Garbage for Production: AWS RDS parameter recommendations exist for a reason, but AWS ships with settings optimized for 2003 hardware. I've seen too many slow databases that could be fixed by spending 10 minutes configuring a custom parameter group instead of just accepting whatever Amazon gives you.

PostgreSQL Parameters That Actually Matter: Use the PostgreSQL Performance Tuning Guide and AWS PostgreSQL Best Practices, but here's what I actually change in production:

-- Memory settings that won't kill your instance (for r6i.xlarge: 32GB RAM)
shared_buffers = '8GB'           -- 25% of instance memory for buffer cache
work_mem = '32MB'                -- Higher for complex queries, monitor total usage
maintenance_work_mem = '2GB'     -- For VACUUM, CREATE INDEX operations
effective_cache_size = '24GB'    -- 75% of instance memory, tells optimizer about cache
random_page_cost = 1.1           -- SSD storage is fast, tell Postgres to use indexes
seq_page_cost = 1.0              -- Sequential scan cost on SSD

-- Connection management that works
max_connections = 200            -- Don't go crazy, each connection uses ~2.5MB base memory
shared_preload_libraries = 'pg_stat_statements'  -- Essential for finding slow queries

-- Autovacuum tuning (critical for performance)
autovacuum = on                  -- Turn this off and your database will slowly die
autovacuum_max_workers = 6       -- More workers for busy systems
autovacuum_work_mem = '256MB'    -- Larger work memory for vacuum operations
track_counts = on                -- Required for autovacuum to know what to clean

-- Write-ahead log optimization
wal_buffers = '64MB'             -- Larger WAL buffers for write-heavy workloads
checkpoint_completion_target = 0.7  -- Spread checkpoint I/O over longer period

MySQL Performance Parameters: Follow the MySQL Performance Schema Guidelines and AWS MySQL Best Practices:

-- InnoDB memory optimization (for r6i.xlarge: 32GB RAM)
innodb_buffer_pool_size = '24GB'         -- 75% of available memory for cache
innodb_buffer_pool_instances = 8         -- Multiple instances for concurrency
innodb_log_buffer_size = '64MB'          -- Larger log buffer for write-heavy loads
innodb_sort_buffer_size = '2MB'          -- Memory for index creation/rebuilding

-- Transaction and durability settings
innodb_flush_log_at_trx_commit = 1       -- Full durability (AWS recommendation)
innodb_flush_method = O_DIRECT           -- Avoid double buffering on Linux
innodb_file_per_table = ON               -- Individual tablespace files
innodb_stats_persistent = ON             -- Consistent query plans across restarts

-- I/O and concurrency optimization
innodb_io_capacity = 2000                -- Match your storage IOPS capability
innodb_io_capacity_max = 4000            -- Maximum I/O for background operations
innodb_read_io_threads = 8               -- Increase for read-heavy workloads
innodb_write_io_threads = 8              -- Increase for write-heavy workloads
innodb_purge_threads = 4                 -- Background cleanup threads

-- Connection and query optimization
max_connections = 500                    -- Higher than PostgreSQL (lighter per-connection)
max_user_connections = 450               -- Leave headroom for admin connections
interactive_timeout = 300               -- Kill idle connections after 5 minutes
wait_timeout = 300                       -- Same timeout for non-interactive connections
table_open_cache = 4000                  -- Cache for frequently accessed tables

-- Disable features that hurt SSD performance
innodb_change_buffering = NONE           -- Optimal for modern SSD storage
query_cache_type = 0                     -- Disable (removed in MySQL 8.0+, causes contention)

Index Strategy and Maintenance

Index Optimization Framework: Effective indexing requires understanding query patterns, not just creating indexes for every WHERE clause:

Covering Index Strategy: Design indexes that include all columns needed for query execution, eliminating table lookups:

-- Instead of separate indexes on (user_id) and (created_date)
-- Create covering index for common query patterns
CREATE INDEX idx_user_activities_covering
ON user_activities (user_id, created_date)
INCLUDE (activity_type, metadata);

Partial Index Optimization: For large tables with query patterns that filter on specific values:

-- Index only active records for frequently filtered queries
CREATE INDEX idx_active_users
ON users (email)
WHERE status = 'active';

Index Maintenance Automation: Configure automatic statistics updates to prevent query plan degradation:

  • PostgreSQL: Ensure autovacuum is properly configured and running
  • MySQL: Enable innodb_stats_auto_recalc for automatic statistics updates
  • Monitor: Use Performance Insights to identify queries with changing execution plans

Advanced Monitoring and Alerting

Custom CloudWatch Metrics: Create composite metrics that combine multiple performance indicators:

-- Storage performance composite metric
-- Combine IOPS utilization + queue depth + latency
-- Alert when multiple indicators show stress simultaneously

-- Connection efficiency metric
-- DatabaseConnections / max_connections ratio
-- Connection creation rate vs. steady state
-- Connection pool efficiency calculations

Performance Insights Query Analysis: Use Performance Insights and Enhanced Monitoring to identify optimization opportunities:

  • Top SQL by execution time: Focus optimization efforts on highest-impact queries
  • Wait event analysis: Understand what database operations are blocking performance
  • Dimension analysis: Correlate performance issues with specific users, hosts, or databases

Proactive Performance Monitoring: Configure alerts for leading indicators, not just problems:

  • Storage utilization trends: Alert before reaching capacity limits
  • Query performance degradation: Track execution time increases for critical queries
  • Connection pattern changes: Detect application behavior changes that impact database

Cost-Performance Optimization Strategy

Systematic Optimization Approach:

  1. Baseline establishment: Use Compute Optimizer and Performance Insights to understand current performance characteristics
  2. Storage optimization: Migrate to appropriate storage types based on workload analysis
  3. Query optimization: Address highest-impact performance bottlenecks identified through monitoring
  4. Connection optimization: Implement appropriate pooling strategies for workload patterns
  5. Ongoing monitoring: Establish automated monitoring and alerting for performance regression detection

ROI Tracking: Measure optimization impact through:

  • Query execution time improvements: Track specific query performance gains
  • Resource utilization efficiency: Monitor CPU, memory, and storage utilization improvements
  • Cost reduction: Calculate infrastructure cost savings from rightsizing and efficiency gains
  • Application performance: Measure end-user experience improvements through APM tools

Storage and query optimization isn't about implementing every fancy feature AWS offers. It's about fixing the actual bottlenecks that are killing your performance. Start with storage (it's usually the easiest win), then tackle the slow queries that Performance Insights is screaming about. Don't optimize for the sake of optimization - fix what's actually broken.

Your RDS Performance Optimization Roadmap

Here's your battle plan, in order of impact:

Week 1: The Foundation (Low effort, high impact)

  1. Enable Compute Optimizer and wait 2 weeks for recommendations
  2. Migrate gp2 → gp3 storage (immediate 20% cost reduction + better performance)
  3. Create custom parameter groups with production-appropriate settings

Week 3-4: Connection Management (Medium effort, eliminates outages)

  1. Implement RDS Proxy for Lambda/microservice workloads
  2. Configure application-level connection pooling with proper sizing
  3. Set up connection monitoring alerts before you hit limits

Month 2: Advanced Optimization (High effort, substantial gains)

  1. Apply Compute Optimizer recommendations systematically
  2. Implement query optimization for Performance Insights top offenders
  3. Consider Graviton migration for cost-sensitive workloads

Ongoing: Monitoring and Maintenance (Prevents regression)

  1. Configure proactive alerting for leading indicators
  2. Regular query performance reviews using Performance Insights
  3. Quarterly cost optimization reviews using Cost Explorer

The difference between databases that struggle and databases that scale isn't rocket science—it's methodically addressing each performance layer without getting distracted by shiny features you don't need. Start with the fundamentals, measure the impact, then move to the next optimization. Your future self (and your AWS bill) will thank you.

Essential Performance Optimization Resources

Related Tools & Recommendations

pricing
Recommended

MongoDB Atlas vs PlanetScale 料金比較 - どっちが安いか、どっちがクソなのか

2025年9月版:PlanetScaleの無料プラン廃止でマジで焦った人向け

MongoDB Atlas
/ja:pricing/mongodb-atlas-vs-planetscale/cost-comparison-analysis
100%
tool
Similar content

Google Cloud SQL - Database Hosting That Doesn't Require a DBA

MySQL, PostgreSQL, and SQL Server hosting where Google handles the maintenance bullshit

Google Cloud SQL
/tool/google-cloud-sql/overview
95%
integration
Similar content

Lambda + DynamoDB Integration - What Actually Works in Production

The good, the bad, and the shit AWS doesn't tell you about serverless data processing

AWS Lambda
/integration/aws-lambda-dynamodb/serverless-architecture-guide
89%
tool
Recommended

Azure SQL Database - Microsoft's Managed SQL Server

Azure's hosted SQL Server. Handles the patching hell so you don't have to, but your wallet will feel it.

Azure SQL Database
/tool/azure-sql-database/overview
66%
tool
Recommended

PlanetScale - まともにスケールするMySQLプラットフォーム

YouTubeと同じ技術でデータベースの悪夢から解放される

PlanetScale
/ja:tool/planetscale/overview
61%
tool
Recommended

PlanetScale 本番障害対応 - 午前3時のサバイバルガイド

実際のエラーメッセージと血と汗で覚えた解決法

PlanetScale
/ja:tool/planetscale/production-troubleshooting
61%
integration
Recommended

GitHub Actions + AWS Lambda: Deploy Shit Without Desktop Boomer Energy

AWS finally stopped breaking lambda deployments every 3 weeks

GitHub Actions
/brainrot:integration/github-actions-aws/serverless-lambda-deployment-automation
60%
tool
Recommended

Lambda Has B200s, AWS Doesn't (Finally, GPUs That Actually Exist)

integrates with Lambda Labs

Lambda Labs
/tool/lambda-labs/blackwell-b200-rollout
60%
integration
Recommended

Stop Clicking Through 50 AWS Consoles Every Week

Managing Security Across Multiple AWS Accounts is Hell - Here's How We Automated the Pain Away

Terraform
/integration/terraform-aws-multi-account-gitops-security/gitops-security-automation
60%
integration
Recommended

Terraform AWS CI/CD Integration - Stop Breaking Prod Manually

integrates with Terraform

Terraform
/brainrot:integration/terraform-aws/cicd-pipeline-integration
60%
integration
Recommended

How We Stopped Breaking Production Every Week

Multi-Account DevOps with Terraform and GitOps - What Actually Works

Terraform
/integration/terraform-aws-multiaccount-gitops/devops-pipeline-automation
60%
tool
Recommended

AWS DMSで企業DB移行 - VPC設定で3日間ほとんど寝れずに上司にブチ切れられた話

DMS使ったら案の定クソハマり。ネットワーク設定で眠れない日々が続いて、上司に「営業は簡単って言ったじゃないか!いつ終わるんだ!」ってキレられた

AWS Database Migration Service
/ja:tool/aws-dms/enterprise-deployment
60%
pricing
Recommended

MongoDB Atlas pricing makes no fucking sense. I've been managing production clusters for 3 years and still get surprised by bills.

competes with MongoDB Atlas

MongoDB Atlas
/pricing/mongodb-atlas-vs-competitors/cluster-tier-optimization
55%
tool
Recommended

MongoDB Atlas Enterprise Deployment Guide

competes with MongoDB Atlas

MongoDB Atlas
/tool/mongodb-atlas/enterprise-deployment
55%
tool
Recommended

Datadog Security Monitoring - Is It Actually Good or Just Marketing Hype?

integrates with Datadog

Datadog
/tool/datadog/security-monitoring-guide
55%
integration
Recommended

Why Your Monitoring Bill Tripled (And How I Fixed Mine)

Four Tools That Actually Work + The Real Cost of Making Them Play Nice

Sentry
/integration/sentry-datadog-newrelic-prometheus/unified-observability-architecture
55%
pricing
Recommended

Datadog vs New Relic vs Sentry: Real Pricing Breakdown (From Someone Who's Actually Paid These Bills)

Observability pricing is a shitshow. Here's what it actually costs.

Datadog
/pricing/datadog-newrelic-sentry-enterprise/enterprise-pricing-comparison
55%
tool
Recommended

Snowflake - Cloud Data Warehouse That Doesn't Suck

Finally, a database that scales without the usual database admin bullshit

Snowflake
/tool/snowflake/overview
49%
news
Recommended

Snowflake und Salesforce definieren neuen AI-Data-Standard

Unified AI Data Layer - endlich ein Standard für Enterprise AI-Pipelines?

snowflake
/de:news/2025-09-24/snowflake-salesforce-ai-standard
49%
pricing
Recommended

Your Snowflake Bill is Out of Control - Here's Why

What you'll actually pay (hint: way more than they tell you)

Snowflake
/pricing/snowflake/cost-optimization-guide
49%

Recommendations combine user behavior, content similarity, research intelligence, and SEO optimization