Arbitrum Production Debugging - Fix Shit That Breaks at 3AM

Your contract works perfectly in Remix. Your tests pass. You deploy to mainnet, and suddenly users are getting intrinsic gas too low errors while you're desperately refreshing Etherscan trying to figure out what the fuck went wrong.

The Real Problem: Gas estimation on Arbitrum is 2-dimensional. You've got L2 execution costs and L1 data posting costs. When the network gets busy during NFT drops or token launches, that L1 component can spike 3-5x in minutes.

The L1 Gas Price Gotcha

The official troubleshooting docs explain the theory, but here's what actually happens in production:

1. Your frontend estimates gas at 200,000 units during low traffic
2. User clicks "confirm" 30 seconds later
3. L1 gas price jumps from 20 gwei to 80 gwei
4. Transaction fails because your gas limit only covered the old L1 costs
5. User thinks your app is broken

Real Fix: Always buffer gas estimates by 20-30%. We learned this after our DeFi app failed during the PEPE launch when L1 costs spiked 400%.

// Don't do this - exact gas estimation 
const gasLimit = await contract.estimateGas.mint(tokenId);

// Do this - buffer for L1 price spikes
const gasEstimate = await contract.estimateGas.mint(tokenId);
const gasLimit = Math.floor(gasEstimate.toNumber() * 1.25);

Pro Tip: Monitor L1 gas prices in real-time. When they spike above 50 gwei, increase your buffer to 40-50%. We use a simple webhook from ETH Gas Station API that alerts us when L1 gets expensive.

RPC Provider Failures at Scale

Every RPC provider lies about their rate limits. Here's what actually happens when you hit production traffic:

Alchemy claims "no rate limits" on paid plans but starts throttling at ~1000 requests/10 seconds. Error you'll see: 429 Too Many Requests but only after your users already saw failed transactions.

QuickNode advertises "unlimited requests" but has burst limits. You'll hit walls during high-traffic events. Their error messages are cryptic: Request failed with status 503.

Infura just times out during network congestion. No error message, just hanging requests that timeout after 30 seconds.

Our Solution: Load balancing across 3 providers with automatic failover. Cost us an extra $200/month but saved us during the March 2025 MEV bot spam attacks.

// Multi-provider setup with fallbacks
const providers = [
  new ethers.providers.JsonRpcProvider(process.env.ALCHEMY_URL),
  new ethers.providers.JsonRpcProvider(process.env.QUICKNODE_URL),
  new ethers.providers.JsonRpcProvider(process.env.INFURA_URL)
];

let currentProvider = 0;
const getProvider = () => {
  if (providers[currentProvider].ready) return providers[currentProvider];
  currentProvider = (currentProvider + 1) % providers.length;
  return providers[currentProvider];
};

Bridge Transaction UX Nightmare

Bridge transactions take 10-45 minutes and users lose their fucking minds. Your support will get flooded with "where's my money" tickets.

The Reality: Users deposit ETH, see "pending" for 30 minutes, assume it's broken, submit 5 more transactions, then everything arrives at once.

What We Built:

• Real-time bridge status tracking using the Arbitrum SDK
• Email notifications when deposits complete
• Big scary warnings about withdrawal times (7 days to L1)
• Automatic retry logic for failed bridge calls

// Track bridge transaction status
import { L1TransactionReceipt } from '@arbitrum/sdk';

const trackDeposit = async (l1TxHash: string) => {
  const l1Receipt = new L1TransactionReceipt(l1TxHash);
  const l2Result = await l1Receipt.waitForL2(l2Provider);
  
  if (l2Result.complete) {
    // Notify user - money arrived
    sendNotification('Your deposit completed!');
  } else {
    // Still pending - show status
    updateUI('Bridge transaction processing...');
  }
};

Most important: Build a transaction tracker or prepare for support hell. We had 400 support tickets in our first month before building proper bridge monitoring.

Your Rust contract compiled fine. Tests passed locally. Then users hit a complex edge case, and suddenly you're staring at:

WASM trap: unreachable executed at wasm offset 0x1a2b

Good luck debugging that shit at 3 AM.

The Nuclear Option: cargo stylus replay

When your WASM contract panics with cryptic errors, cargo stylus replay is your only salvation. This tool lets you replay failed transactions with GDB/LLDB attached to the actual source code.

Prerequisites:

• Linux with GDB installed (sudo apt-get install gdb)
• RPC provider with tracing enabled (most paid providers support this)
• The exact transaction hash that's failing
• Your contract's source code and debug symbols

Setup Process:

## Install the replay tool
cargo install cargo-stylus

## Set your environment
export RPC_URL=your-rpc-endpoint
export TX_HASH=0xfailure-transaction-hash

## Replay with debugger attached
cargo stylus replay --tx=$TX_HASH --endpoint=$RPC_URL --use-native-tracer

The debugger automatically breaks at the Stylus entry point. From there, you can:

1. Set breakpoints in your actual Rust code: (gdb) b my_contract::problematic_function
2. Step through execution: (gdb) step or (gdb) next
3. Inspect variables: (gdb) p variable_name
4. View the call stack: (gdb) backtrace

WASM Memory Debugging Hell

WASM debugging is like debugging assembly code, but worse. When your contract panics, you usually get one of these useless errors:

"unreachable executed": Usually bounds check failure or integer overflow

// This will panic with \"unreachable\" in WASM
let arr = [1, 2, 3];
let idx = 5;
let value = arr[idx]; // Bounds check failure

"invalid function type": Wrong function signature or ABI mismatch

// Check your function signatures match the ABI
#[external]
impl Contract {
    // Make sure parameter types match exactly
    pub fn transfer(&mut self, to: Address, amount: U256) -> bool {
        // Your implementation
    }
}

"out of bounds memory access": Buffer overflow or uninitialized memory

// Common cause: string manipulation without proper bounds
let mut buffer = vec![0u8; 32];
// If input.len() > 32, this will panic
buffer[..input.len()].copy_from_slice(&input);

The println! Debugging Strategy

When GDB isn't available or tracing is broken, fall back to 2005-era debugging:

#[external]
impl Contract {
    pub fn problematic_function(&mut self, input: U256) -> bool {
        println!(\"Entering function with input: {:?}\", input);
        
        let intermediate = self.calculate_something(input);
        println!(\"Calculated intermediate: {:?}\", intermediate);
        
        if intermediate > U256::from(1000) {
            println!(\"Taking branch A\");
            self.branch_a(intermediate)
        } else {
            println!(\"Taking branch B\");
            self.branch_b(intermediate)
        }
    }
}

The process:

1. Add prints to narrow down the failing section
2. Deploy to testnet and reproduce the issue
3. Check transaction logs for your debug output
4. Move the prints closer to the actual failure
5. Repeat until you find the exact line that breaks

It's barbaric but effective when WASM debugging tools fail you.

Stack Overflow in WASM

WASM has a limited call stack. Deep recursion or large local variables will blow the stack with no useful error message:

// This will eventually overflow the WASM stack
fn recursive_nightmare(&self, n: u32) -> u32 {
    if n == 0 { 1 }
    else { n * self.recursive_nightmare(n - 1) }
}

Debugging stack issues:

• Look for recursive functions without proper termination
• Check for large local arrays or structs
• Use heap allocation (Vec, Box) instead of stack variables for large data

Common WASM Gotchas in Production

Integer Overflow Differences: Debug mode panics on overflow, release mode wraps. Your contract might work in testing but fail silently in production:

#[cfg(debug_assertions)]
let result = a.checked_add(b).expect(\"Addition overflow\");
#[cfg(not(debug_assertions))]
let result = a.wrapping_add(b); // Silent overflow in release

Memory Layout Issues: WASM memory layout differs from native. Pointer arithmetic that works locally might fail on-chain:

// Avoid raw pointer manipulation in WASM
// Use Vec and other safe containers instead

Missing Host Function Calls: Some Rust std library functions don't work in WASM. You'll get runtime panics instead of compile-time errors:

// These will panic in WASM context
std::thread::spawn(|| {}); // No threading
std::fs::File::open(\"file\"); // No filesystem
println!(\"Debug\"); // Works but goes to transaction logs

The Hard Truth: WASM debugging fucking sucks. Budget 3-5x longer for debugging compared to native Rust development. Your debugging workflow will be mostly printf debugging and GDB sessions on replayed transactions.