Stop Finding Out About Production Issues From Twitter

The Error → Chat → Wake Someone Up Pipeline

Sentry catches your errors, Slack yells about them, and PagerDuty wakes up whoever's unlucky enough to be on call. Simple concept, but the devil's in the webhook details.

Core Architecture Components

Sentry: The Error Catcher
Sentry is your early warning system. It sits in your app catching crashes, performance problems, and whatever else your users manage to break. Supports pretty much every language (though their JavaScript error fingerprinting is garbage for similar errors). Version gotcha: Sentry SDK 7.x to 8.x migration changed error boundary handling - check the migration docs or your React errors might not get captured.

The important part: Sentry's webhook system fires off JSON payloads when shit hits the fan. Make sure you understand webhook signature verification or you'll be debugging security issues for weeks. These webhooks include the error message, stack trace, user context, and environment info - basically everything you need to figure out what broke and why.

Slack: The Message Hub
Slack is where your team panics together in an organized fashion. When Sentry fires a webhook, your middleware formats it into a readable message and posts it to the appropriate channel. Block Kit lets you create fancy interactive messages with buttons like "View in Sentry" or "Page the on-call engineer."

Here's where rate limiting will bite you: Slack enforces a 1 message per second limit per channel. During a major outage when errors are flying, you'll hit this limit fast. Check out Slack's rate limiting guide and plan for message batching or your integration will choke when you need it most.

PagerDuty: The Human Wakeup Service
PagerDuty is expensive but it beats missing critical alerts. When your Slack integration determines an error needs human attention, it fires off an event to PagerDuty's Events API. PagerDuty then proceeds to annoy whoever's on call until they acknowledge the incident.

The Event Intelligence feature tries to group related alerts so you don't get 47 pages for the same database outage. Their alert grouping algorithms use machine learning to correlate similar incidents. Works okay most of the time, fails spectacularly during actual emergencies when you need it most. But hey, at least someone gets woken up.

How the Data Actually Flows (And Where It Breaks)

Webhook Flow Architecture

Here's what happens when your production app decides to shit the bed:

1. Error happens: Your app crashes, Sentry captures the stack trace and user context
2. Webhook fires: Sentry sends a POST request to your serverless function (this fails constantly)
3. Middleware processes: Your function verifies the webhook signature, parses the error data
4. Decision logic: Function decides if this error deserves human attention based on your rules
5. Slack notification: Posts formatted message to team channel (rate limits kick in here)
6. Escalation check: If error meets severity criteria, creates PagerDuty incident
7. Human wakeup: PagerDuty calls/texts whoever's on call until they acknowledge

Where this breaks in practice:

• Webhook delivery failures (network timeouts, server overload)
• Signature verification issues when secrets rotate (War story: Spent 4 hours debugging "invalid signature" errors during Black Friday because someone rotated the webhook secret)
• Rate limiting during outage storms (Reality check: During our Redis outage, Slack rate limited us after maybe 400-500 messages pretty quick)
• PagerDuty API throttling when you need it most
• Cold start delays in serverless functions when milliseconds matter (Gotcha: Vercel cold starts can take 2-5 seconds, AWS Lambda 500ms-2s depending on runtime)

You'll need proper retry logic with exponential backoff and dead letter queues for the inevitable failures.

Lessons Learned From Production Hell

Use Slack as Mission Control
Put all incident communication in Slack channels. Makes it easy to see who's working on what, and you get a timeline for postmortems. Just don't try to manage complex incidents over DM - that's how important context gets lost.

Stop Alert Storms Before They Start
When your database goes down, you don't need 200 individual error alerts. Build in correlation logic to group related errors by service, deployment, or error signature. PagerDuty's grouping helps, but you'll still need custom logic for your specific fuckups.

Include Actual Context
Every alert should answer: What broke? How many users affected? What changed recently? Link directly to Sentry issues, not just generic "something went wrong" messages. Your 3am self will thank you.

Monitor Your Monitoring
Track webhook delivery rates, API response times, and end-to-end notification latency. Your integration will fail when you need it most, so build in health checks and dead letter queues for failed webhooks. Most teams see 95%+ reliability once they implement proper retry logic.

Setting This Up Will Take Longer Than You Think

Plan for a full day if you're lucky, 2-3 days if you hit the usual webhook hell and API documentation gaps. Here's what actually happens when you try to build this integration.

What You Actually Need

Prerequisites (The Boring Stuff)

• Admin access to Sentry (not just "member" - you'll need webhook permissions)
• Slack workspace admin rights (or convince someone who has them)
• PagerDuty Professional plan ($25/user/month - the free tier won't cut it)
• Somewhere to host your webhook handler (Vercel is easiest, AWS Lambda if you hate yourself)

Planning (Or You'll Regret It Later)

• Figure out which errors deserve human attention (spoiler: not all of them)
• Create dedicated Slack channels per service/team (don't spam #general)
• Set up PagerDuty escalation policies BEFORE the integration (learn from my mistakes)
• Define severity levels that make sense for your team's sanity

Phase 1: Sentry Webhook Configuration

Step 1: Create the Integration

Go to Sentry Settings → Integrations → Internal Integrations. You'll need these permissions:

• Event:write - For creating error events
• Project:read - To access project data
• Organization:read - For organization-level stuff

Sentry's UI is confusing as hell - you'll probably create 3 integrations before getting the permissions right. Version gotcha: If you're using Sentry SaaS post-2023, the integration permissions UI moved - it's now under Developer Settings, not Organization Settings like every tutorial shows.

Step 2: Webhook URL Setup

Point the webhook to your serverless function. Use webhook signature verification or get ready for security audit failures:

// This webhook signature verification will save your ass
const isValidRequest = await verifySentrySignature(req);
if (!isValidRequest) {
  // Don't just return 401, log it - helps debug webhook issues
  console.log('Invalid webhook signature, possible replay attack');
  return res.status(401).json({error: 'Invalid signature'});
}

Common webhook failures you'll see:

• Error: connect ECONNREFUSED 127.0.0.1:5432 - Your function can't reach the database (check connection strings)
• SyntaxError: Unexpected token < in JSON - Sentry sent HTML error page instead of JSON (webhook URL is wrong)
• TypeError: Cannot read property 'event' of undefined - Webhook payload structure changed (Sentry updated their API)
• Error: Request timeout - Your function is taking >30 seconds (add timeouts, optimize your code)

Step 3: Alert Rules (Prepare for Noise)

Start aggressive with filtering, then loosen over time. Your first attempt will page everyone for JavaScript undefined is not a function errors:

• Actually Critical: Database connection failures, payment processing errors
• Probably Important: New error types in production, errors affecting >50 users
• Maybe Later: Performance degradation, client-side JS errors (most are garbage)

Phase 2: Slack Application Setup

Step 4: Create Slack Application

Visit the Slack App Directory and create a new application for your workspace. Configure the following OAuth scopes for comprehensive integration functionality:

• chat:write - Posts messages to channels
• channels:read - Lists public channels for configuration
• users:read - Accesses user profiles for @mentions in alerts
• files:write - Uploads screenshots and error reports

Slack API errors that will ruin your day:

• missing_scope - You forgot to add the right OAuth scope (check your app settings)
• channel_not_found - Channel was deleted or your bot isn't invited (add bot to channel)
• rate_limited - Slack's 1 msg/sec limit hit you (implement queuing or your integration dies during outages)
• invalid_auth - Your bot token expired or was revoked (rotate tokens quarterly)

Step 5: Design Interactive Message Templates

Use Slack Block Kit to make your error messages not suck. Include the actual error details, not just "something broke":

{
  \"blocks\": [
    {
      \"type\": \"header\",
      \"text\": {
        \"type\": \"plain_text\",
        \"text\": \"🚨 Critical Error Detected\"
      }
    },
    {
      \"type\": \"section\",
      \"fields\": [
        {
          \"type\": \"mrkdwn\",
          \"text\": \"*Environment:* Production\"
        },
        {
          \"type\": \"mrkdwn\",
          \"text\": \"*Users Affected:* 247\"
        }
      ]
    },
    {
      \"type\": \"actions\",
      \"elements\": [
        {
          \"type\": \"button\",
          \"text\": {
            \"type\": \"plain_text\",
            \"text\": \"View in Sentry\"
          },
          \"url\": \"https://sentry.io/organizations/example/issues/123/\"
        },
        {
          \"type\": \"button\",
          \"text\": {
            \"type\": \"plain_text\",
            \"text\": \"Create PagerDuty Incident\"
          },
          \"value\": \"create_pd_incident\"
        }
      ]
    }
  ]
}

Phase 3: PagerDuty Service Integration

Step 6: Configure PagerDuty Services

Create separate PagerDuty services for each app or service. Otherwise you'll get woken up for someone else's fuckups. Each service should include:

• Integration Key: Unique identifier for API event submission
• Escalation Policy: Multi-level on-call notification sequence
• Event Rules: Automatic incident classification and routing logic

Step 7: Implement Event Intelligence Rules

Set up PagerDuty Event Intelligence to prevent getting 47 pages for the same database outage:

• Suppression Rules: Ignore known maintenance windows and deployment periods
• Grouping Rules: Correlate related errors by service, environment, or error type
• Priority Mapping: Automatically assign incident priority based on error severity and user impact

PagerDuty API failures that will fuck you:

• Invalid routing key - Wrong integration key (check your service configuration)
• Event not processed - Your JSON payload is malformed (validate your event structure)
• Rate limit exceeded - Sending too many events (PagerDuty limits you to 120 events/minute)
• Forbidden - Your API key lacks permissions (check team/service access rights)

Phase 4: Middleware Development (The Hard Part)

Serverless Integration Architecture

Step 8: Build Integration Middleware

This is where most people get stuck. Your serverless function needs to handle webhook hell, rate limiting, and all the ways these APIs can fail. Reality check: Our first version took down production twice - once from infinite retry loops, once from memory issues with Node.js 16.x in serverless functions:

export default async function handler(req, res) {
  // Verify Sentry webhook signature
  const isValidRequest = await verifySentrySignature(req);
  if (!isValidRequest) return res.status(401).json({error: 'Invalid signature'});
  
  const sentryEvent = req.body;
  
  // Format Slack message
  const slackMessage = formatErrorNotification(sentryEvent);
  await postToSlack(slackMessage);
  
  // Create PagerDuty incident for critical errors
  if (sentryEvent.level === 'error' && sentryEvent.contexts.trace.user_count > 50) {
    await createPagerDutyIncident(sentryEvent);
  }
  
  res.status(200).json({status: 'processed'});
}

Step 9: Implement Error Handling and Retries

Your integration will break when you need it most. Build in retry logic, dead letter queues, and health checks. Use Datadog or New Relic to know when your monitoring is down (because it will be).

Serverless function errors you'll debug at 3am:

• Lambda timeout - Function takes >15 minutes (add timeouts to your API calls)
• Memory limit exceeded - Node.js memory leak (restart function or optimize)
• Cold start timeout - First invocation too slow (pre-warm functions or accept 2-5s delay)
• ESOCKETTIMEDOUT - API call to Slack/PagerDuty failed (implement proper timeouts and retries)

Phase 5: Testing and Validation

Step 10: End-to-End Testing Protocol

Test this shit end-to-end before you go live:

1. Error Injection: Trigger test errors in staging environment using Sentry's test suite integration
2. Notification Verification: Confirm Slack messages appear with correct formatting and interactive elements
3. Escalation Testing: Validate PagerDuty incident creation and on-call notification delivery
4. Resolution Tracking: Verify that incident status updates propagate correctly across all platforms

Step 11: Performance Benchmarking

Measure key performance indicators to establish baseline metrics for ongoing optimization:

• Webhook Latency: Time from Sentry error to Slack notification (target: <30 seconds)
• API Response Times: Average response time for PagerDuty incident creation (target: <5 seconds)
• Escalation Accuracy: Percentage of appropriately escalated incidents (target: >95%)
• False Positive Rate: Incidents created for non-actionable errors (target: <5%)

In our experience, this setup cuts the time from "production is broken" to "someone's actually looking at it" by about half. Not because the tools are magic, but because you stop missing alerts buried in email.