Hardhat Production Deployment - Don't Use This in Production Unless You Enjoy 2am Phone Calls

Moving from npx hardhat run scripts/deploy.js to actual mainnet deployments is where most teams lose their minds. I've seen $50M protocols fail because someone deployed the wrong contract version, and DevOps engineers wake up to Slack notifications about failed upgrades at 3am. This isn't a tutorial - it's a survival guide.

The Pre-Flight Checklist That Saves Careers

Version pinning isn't optional anymore. Pin EVERYTHING. I mean everything. Even the linter. One wrong dependency update has bricked more deployments than you'd believe:

// hardhat.config.ts - Lock down the world
module.exports = {
  solidity: {
    version: "0.8.19", // Specific version, not ^0.8.0
    settings: {
      optimizer: { 
        enabled: true, 
        runs: 200 // Not 1000, not 100 - exactly what you tested with
      },
      viaIR: true // For complex contracts hitting stack limits
    }
  }
};

Environment isolation saves companies. I watched a team accidentally deploy their dev contracts to mainnet because someone had MAINNET_RPC in their .env file. Use Hardhat's config variables for anything that matters:

## .env.example - What everyone should copy
SEPOLIA_RPC_URL=https://sepolia.infura.io/v3/YOUR_INFURA_KEY
DEPLOYER_PRIVATE_KEY=your_testnet_private_key_here
ETHERSCAN_API_KEY=your_etherscan_api_key

## What production actually needs
MAINNET_RPC_URL=https://mainnet.infura.io/v3/PRODUCTION_KEY
LEDGER_ACCOUNT=0x... # Hardware wallet address
MAINNET_ETHERSCAN_KEY=production_etherscan_key

Gas Estimation is Lying to You

Gas estimation in Hardhat assumes perfect network conditions that don't exist. Real networks have MEV bots, congestion, and gas price volatility that'll make your deployment fail halfway through.

The 1.5x rule that prevents disasters: Always multiply estimated gas by 1.5x for mainnet. Yes, you'll overpay sometimes. No, it's not worth the risk of a failed deployment eating $500 in gas fees:

// Production-ready gas configuration
const gasPrice = await ethers.provider.getGasPrice();
const deploymentTx = await factory.deploy(initArgs, {
  gasLimit: estimatedGas.mul(150).div(100), // 1.5x estimated
  gasPrice: gasPrice.mul(110).div(100),     // 10% above current
  maxFeePerGas: gasPrice.mul(150).div(100), // EIP-1559 buffer
});

Gas limit gotchas that bite everyone:

• Contract creation uses different gas calculations than calls
• Proxy deployments need gas for both proxy AND implementation
• OpenZeppelin upgrades add 20-30% overhead you don't see in tests

Network Configuration That Won't Fail You

Multi-network deployments are where teams realize their localhost setup means nothing. RPC providers have different behaviors, rate limits, and failure modes:

// networks.ts - Real network config that works
networks: {
  mainnet: {
    url: process.env.MAINNET_RPC_URL,
    accounts: process.env.LEDGER_ACCOUNT ? [] : [process.env.DEPLOYER_KEY],
    gasPrice: "auto",
    timeout: 60000, // 1 minute timeout for congested periods
    confirmations: 2,   // Wait for 2 confirmations minimum
  },
  polygon: {
    url: process.env.POLYGON_RPC_URL,
    accounts: [process.env.DEPLOYER_KEY],
    gasPrice: 35000000000, // 35 gwei - Polygon gas quirks
    confirmations: 5,      // Polygon needs more confirmations
  },
  arbitrum: {
    url: "https://arb1.arbitrum.io/rpc",
    accounts: [process.env.DEPLOYER_KEY], 
    gasPrice: "auto",
    verify: {
      etherscan: {
        apiKey: process.env.ARBISCAN_API_KEY,
        apiUrl: "https://api.arbiscan.io/"
      }
    }
  }
}

RPC provider reality check:

• Infura rate limits at 100k requests/day on free tier
• Alchemy has better WebSocket support but different error handling
• QuickNode costs more but doesn't randomly throttle during high usage

Contract Verification Hell (And How to Escape It)

Contract verification fails more often than deployments. Etherscan verification breaks if your compiler settings don't exactly match what Etherscan expects:

// hardhat.config.ts - Verification that actually works
etherscan: {
  apiKey: {
    mainnet: process.env.ETHERSCAN_API_KEY,
    polygon: process.env.POLYGONSCAN_API_KEY,
    arbitrumOne: process.env.ARBISCAN_API_KEY,
  },
  customChains: [
    {
      network: "arbitrumOne", 
      chainId: 42161,
      urls: {
        apiURL: "https://api.arbiscan.io/api",
        browserURL: "https://arbiscan.io/"
      }
    }
  ]
},
sourcify: {
  enabled: true // Sourcify as backup verification
}

Verification debugging that saves hours:

• Constructor arguments must be ABI-encoded exactly
• Library linking breaks verification if addresses don't match
• Proxy contracts need separate verification for implementation and proxy

The Deployment Scripts That Don't Break

Most deployment scripts are garbage - they work once in perfect conditions then fail mysteriously. Here's what actually works in production:

// deploy/001_deploy_protocol.ts - Battle-tested deployment
import { HardhatRuntimeEnvironment } from "hardhat/types";
import { DeployFunction } from "hardhat-deploy/types";

const deployProtocol: DeployFunction = async (hre: HardhatRuntimeEnvironment) => {
  const { deployments, getNamedAccounts, network } = hre;
  const { deploy, execute } = deployments;
  const { deployer } = await getNamedAccounts();

  // Pre-flight checks that prevent disasters
  const balance = await hre.ethers.provider.getBalance(deployer);
  if (balance.lt(hre.ethers.utils.parseEther("0.1"))) {
    throw new Error(`Deployer needs more ETH. Current: ${hre.ethers.utils.formatEther(balance)}`);
  }

  console.log(`Deploying to ${network.name} with ${deployer}`);
  console.log(`Balance: ${hre.ethers.utils.formatEther(balance)} ETH`);

  // Deploy with proper error handling
  const factory = await deploy("TokenFactory", {
    from: deployer,
    args: [deployer], // Constructor args
    log: true,
    waitConfirmations: network.name === "mainnet" ? 2 : 1,
  });

  // Verify immediately - don't wait
  if (network.name !== "hardhat" && network.name !== "localhost") {
    try {
      await hre.run("verify:verify", {
        address: factory.address,
        constructorArguments: [deployer],
      });
    } catch (error) {
      console.log("Verification failed:", error.message);
    }
  }

  console.log(`TokenFactory deployed to: ${factory.address}`);
  
  // Save deployment info for later scripts
  const deploymentInfo = {
    network: network.name,
    factory: factory.address,
    deployer,
    timestamp: new Date().toISOString(),
  };
  
  require("fs").writeFileSync(
    `deployments-${network.name}.json`, 
    JSON.stringify(deploymentInfo, null, 2)
  );
};

export default deployProtocol;

Hardware Wallet Integration (Because You're Not Storing Private Keys in .env Files, Right?)

Hardhat Ignition with Ledger is the only way to deploy serious money contracts. Anyone putting mainnet private keys in environment variables should be fired:

## Deploy with hardware wallet - the only sane approach
npx hardhat ignition deploy ignition/modules/Protocol.ts \
  --network mainnet \
  --strategy create2 \
  --deployment-id production-v1.0.0 \
  --ledger

## Verify the deployment worked correctly
npx hardhat ignition verify production-v1.0.0 --network mainnet

Hardware wallet gotchas that waste time:

• Ledger device must be unlocked and Ethereum app open
• Large deployments timeout - deploy in smaller modules
• Contract interaction requires confirming each transaction on device

Production deployments aren't just about getting code on-chain - they're about making sure your protocol doesn't become the next $50M hack headline. Most teams focus on smart contract security and completely ignore deployment operations security. That's how you end up with perfect contract code deployed through completely compromised infrastructure.

The Operational Security Checklist That Prevents Headlines

Environment separation isn't just best practice - it's survival. I've watched teams accidentally deploy mainnet contracts with testnet configurations because someone had the wrong RPC URL in their environment. Hardhat's config variables and environment management help prevent these disasters:

## .env.production - What production actually needs
NODE_ENV=production
NETWORK=mainnet
RPC_URL=https://mainnet.infura.io/v3/PRODUCTION_KEY_ONLY
DEPLOYER_WALLET=hardware  # Never a private key
ETHERSCAN_API_KEY=production_only_key
SLACK_WEBHOOK=https://hooks.slack.com/critical-alerts

## .env.staging - Staging that actually mirrors production  
NODE_ENV=staging
NETWORK=sepolia
RPC_URL=https://sepolia.infura.io/v3/STAGING_KEY
DEPLOYER_WALLET=0xSTAGING_ADDRESS
ETHERSCAN_API_KEY=staging_etherscan_key

Access control that doesn't assume everyone on your team should be able to nuke production:

• Deployer accounts should be hardware wallets only
• Critical operations require 2-of-3 multi-sig approval
• RPC keys should have IP restrictions and usage monitoring
• Code signing for deployment artifacts (yes, Web3 needs this too)

Multi-Network Deployment That Doesn't Create Attack Vectors

Cross-chain deployments are where teams create the most security holes. Each network has different security assumptions, block times, and economic incentives:

// multi-network.config.ts - Security-aware network config
const networks = {
  mainnet: {
    url: process.env.MAINNET_RPC_URL,
    accounts: [], // Hardware wallet only
    gasPrice: "auto",
    confirmations: 12,    // 12 blocks ≈ 3 minutes
    timeout: 300000,      // 5 minutes for congested periods
    initialBaseFeePerGas: "7000000000", // 7 gwei minimum
  },
  polygon: {
    url: process.env.POLYGON_RPC_URL, 
    accounts: [], // Hardware wallet only
    gasPrice: 35000000000,  // 35 gwei - Polygon standard
    confirmations: 20,      // Polygon needs more confirmations  
    timeout: 120000,        // 2 minutes
    chainId: 137,           // Explicit chain ID prevents replay attacks
  },
  arbitrum: {
    url: "https://arb1.arbitrum.io/rpc",
    accounts: [],
    gasPrice: "auto", 
    confirmations: 5,       // Arbitrum confirms faster
    timeout: 180000,        // 3 minutes
    chainId: 42161,
  }
};

// Security validation before deployment
const validateNetworkConfig = (network) => {
  if (!network.chainId) throw new Error("Chain ID required for replay protection");
  if (network.confirmations < 5) throw new Error("Minimum 5 confirmations required");
  if (network.accounts.length > 0) throw new Error("Use hardware wallet, not private keys");
};

Multi-network security gotchas:

• Bridge exploits: If your protocol spans chains, bridge security is your security
• Chain reorgs: Different chains have different reorg probabilities
• Gas price manipulation: MEV opportunities vary by chain
• Contract address consistency: Same nonce deployments across chains for identical addresses

Monitoring and Incident Response (Because Things Always Break)

Production monitoring for smart contracts is completely different from traditional applications. Your contracts might be working perfectly while losing millions to MEV attacks or economic exploits. OpenZeppelin Defender and Tenderly monitoring help, but you need custom monitoring too:

// monitoring.js - Production monitoring that catches real issues
const monitorProtocol = async () => {
  // Contract state monitoring
  const tvl = await vault.totalValueLocked();
  if (tvl.lt(previousTVL.mul(95).div(100))) {
    // Alert: TVL dropped more than 5%
    await sendSlackAlert(`🚨 TVL dropped: ${formatEther(tvl)} ETH`);
  }

  // Transaction monitoring  
  const recentTxs = await getRecentTransactions(vault.address);
  const failedTxs = recentTxs.filter(tx => tx.status === 0);
  if (failedTxs.length > 5) {
    // Alert: High failure rate indicates contract issues
    await sendSlackAlert(`⚠️ High tx failure rate: ${failedTxs.length} failed`);
  }

  // Gas price monitoring
  const gasPrice = await provider.getGasPrice();
  if (gasPrice.gt(parseUnits("100", "gwei"))) {
    // Alert: High gas prices affect user experience
    await sendSlackAlert(`💰 High gas prices: ${formatUnits(gasPrice, "gwei")} gwei`);
  }

  // Security monitoring
  const balance = await provider.getBalance(vault.address);
  if (balance.eq(0) && previousBalance.gt(0)) {
    // CRITICAL: Contract drained
    await sendSlackAlert(`🔥 CRITICAL: Vault balance is ZERO!`);
    await pauseContract(); // Emergency response
  }
};

What you actually need to monitor:

• Contract balances - sudden changes indicate exploits
• Total Value Locked (TVL) - economic health indicator
• Transaction success rates - contract functionality health
• Gas usage patterns - efficiency and attack detection
• Admin key usage - unauthorized administrative actions

Incident Response Playbook (When Shit Hits the Fan)

Phase 1: Detection and Assessment (First 5 minutes)

1. Confirm the incident - false positives waste time
2. Assess financial impact - how much money is at risk?
3. Determine attack vector - ongoing exploit or one-time event?
4. Check if contracts can be paused - do you have emergency controls?

Phase 2: Immediate Response (5-30 minutes)

1. Pause contracts if possible and safe to do so
2. Notify stakeholders - team, investors, community
3. Document everything - timestamps, transactions, decisions
4. Contact security partners - auditors, bug bounty hunters

Phase 3: Investigation and Mitigation (30 minutes - hours)

1. Root cause analysis - what exactly went wrong?
2. Assess damage - how much was lost/affected?
3. Plan mitigation - upgrade, redeploy, or accept loss?
4. Prepare communications - users deserve transparency

// incident-response.js - Automated incident response
const emergencyResponse = async (incidentType) => {
  console.log(`🚨 INCIDENT DETECTED: ${incidentType}`);
  
  // Step 1: Immediate containment
  if (await vault.paused() === false) {
    console.log("Pausing all contracts...");
    await vault.emergencyPause(); // Requires multi-sig
  }
  
  // Step 2: Notify all stakeholders
  await sendSlackAlert(`🔥 CRITICAL INCIDENT: ${incidentType}`);
  await sendDiscordAlert(`Emergency response initiated: ${incidentType}`); 
  await sendEmailAlert(`URGENT: Smart Contract Incident - ${incidentType}`);
  
  // Step 3: Snapshot current state for investigation
  const snapshot = {
    blockNumber: await provider.getBlockNumber(),
    timestamp: new Date().toISOString(),
    contractBalances: await getContractBalances(),
    recentTransactions: await getRecentTransactions(),
    tvlSnapshot: await getCurrentTVL()
  };
  
  fs.writeFileSync(`incident-${Date.now()}.json`, JSON.stringify(snapshot, null, 2));
  console.log("State snapshot saved for forensic analysis");
};

Upgrade Strategies That Don't Break Everything

Contract upgrades in production are where most protocols die. The complexity of coordinating multiple contracts, preserving state, and not bricking user funds is overwhelming. OpenZeppelin's upgrade patterns and proxy security considerations are essential reading:

// upgrade-strategy.js - Coordinated upgrade pattern  
const executeProtocolUpgrade = async () => {
  console.log("Starting coordinated protocol upgrade...");
  
  // Phase 1: Deploy new implementations (non-destructive)
  const newVault = await deployContract("VaultV2");
  const newStrategy = await deployContract("StrategyV2"); 
  
  console.log("New implementations deployed");
  
  // Phase 2: Pause system (reversible)
  await vault.pause();
  await strategy.pause();
  
  console.log("System paused for upgrade");
  
  // Phase 3: Upgrade atomically (point of no return)
  await upgrades.upgradeProxy(vault.address, VaultV2);
  await upgrades.upgradeProxy(strategy.address, StrategyV2);
  
  console.log("Proxy upgrades complete");
  
  // Phase 4: Re-initialize if needed
  await vault.initializeV2(); // V2-specific initialization
  await strategy.initializeV2();
  
  // Phase 5: Unpause system
  await vault.unpause();
  await strategy.unpause();
  
  console.log("Upgrade complete - system operational");
  
  // Phase 6: Verify upgrade success
  const version = await vault.version();
  if (version !== "2.0.0") {
    throw new Error("Upgrade verification failed");
  }
};

Upgrade testing that catches breaking changes:

• Fork mainnet state and test upgrades against real data
• Test migration scripts with actual user positions using Tenderly forks
• Verify storage slots haven't been corrupted
• Check all external integrations still work post-upgrade with integration testing

The Economics of Production Operations

Running production smart contracts costs real money in ways most teams don't anticipate:

Gas cost budgeting:

• Contract deployment: $500-5000 depending on complexity
• Verification costs: $50-200 per contract across all networks
• Failed deployment recovery: $200-2000 in wasted gas
• Upgrade operations: $300-1000 per upgrade cycle

Ongoing operational costs:

• RPC provider fees: $100-500/month for production usage
• Block explorer API limits: $50-200/month
• Monitoring infrastructure: $200-500/month
• Multi-sig transaction fees: $20-100 per governance action

The hidden costs of security incidents:

• Emergency response team: $10,000+ per incident
• Security audit post-incident: $20,000-100,000
• Legal and compliance costs: $50,000+ for major incidents
• Reputation damage: Immeasurable but career-ending

This isn't just about code anymore - production smart contract operations require real infrastructure, real security practices, and real incident response capabilities. Plan for it or become another cautionary tale.