OpenAI Realtime API - Build voice apps that don't suck

It's OpenAI's attempt to fix the clusterfuck that is building voice apps. Instead of chaining together speech-to-text → GPT → text-to-speech like some Rube Goldberg machine, the OpenAI Realtime API lets you go straight from voice to voice over a WebSocket connection.

I spent months debugging that traditional pipeline bullshit - audio cutting out, latency spikes when your internet hiccupped, and don't get me started on trying to handle interruptions. This API actually works without wanting to throw your laptop out the window.

How it actually works (the parts that matter)

You open a WebSocket to wss://api.openai.com/v1/realtime, send PCM16 audio chunks, and get audio back. That's it. No more juggling Whisper, GPT-4, and ElevenLabs like you're running a three-ring circus.

The WebSocket stays open and handles bidirectional streaming. When someone talks, you get interruption detection automatically - no more implementing your own voice activity detection that works great in your quiet home office but shits the bed in a coffee shop.

What you get out of the box:

• Audio processing without format conversion nightmares
• Interruption handling that actually works
• Function calling mid-conversation (execute code while talking)
• Support for images (describe what you're looking at)
• Multiple voice options including some new ones that don't sound like robots

The cost reality check

Here's where it gets expensive. We're talking $0.06 per minute of input audio and $0.24 per minute of output as of late 2024. That's roughly $18/hour if both people are talking constantly. Our first production bill was way higher than expected - I think it was like $800-something because I had no clue how token counting works with audio.

Compare that to Whisper + GPT-4 + ElevenLabs which runs about $0.02/minute total. Yeah, it's 15x more expensive, but it saved me three weeks of WebSocket debugging hell and my sanity.

Production gotchas nobody tells you about

WebSocket connections die. A lot. You need solid reconnection logic or your users will be talking to a dead connection. I learned this the hard way when our demo worked perfectly but production was dropping connections every 2-3 minutes under load.

Browser audio permissions are fucked. iOS Safari especially - sometimes the audio starts working 30 seconds after the user grants permission. Chrome throttles WebSocket connections in background tabs. Firefox has its own special brand of audio weirdness.

Regional latency is all over the place. Works decent in the US but Europe is a shitshow - like 3-4x slower which makes conversation feel broken. There's no regional endpoints yet, so you're stuck with whatever OpenAI's infrastructure decides to route you to.

The API is solid for demos and prototypes. Production requires serious error handling, cost monitoring (set billing alerts or prepare for sticker shock), and patience with browser audio quirks.

WebSocket connection that doesn't immediately die

The basic connection looks simple enough, but there's a bunch of gotchas that'll waste your afternoon:

const ws = new WebSocket("wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview-2024-10-01", [], {
    headers: {
        "Authorization": "Bearer " + process.env.OPENAI_API_KEY,
        "OpenAI-Beta": "realtime=v1",
    },
});

What goes wrong immediately:

• Chrome blocks the connection if you don't have HTTPS on localhost (use https://localhost not http://)
• The WebSocket will close with code 1006 if your API key is wrong - no helpful error message
• iOS Safari sometimes takes 5-10 seconds to actually open the connection even though the promise resolves

Audio format hell and browser quirks

You need PCM16 at 24kHz, base64 encoded. Getting there from browser audio is a pain:

// This won't work on iOS Safari < 14.5
const audioContext = new (window.AudioContext || window.webkitAudioContext)({
    sampleRate: 24000
});

Format conversion nightmare checklist:

• MediaRecorder API gives you WebM/MP4 - you need to convert to PCM16
• Web Audio API resampling sounds like garbage on some Android phones
• iOS device speakers cause audio feedback loops if you don't use headphones/mute carefully

The events that actually matter

Skip the official docs - here's what you actually need:

// Send this to start talking
ws.send(JSON.stringify({
    type: "conversation.item.create",
    item: {
        type: "message",
        role: "user", 
        content: [{type: "input_audio", audio: base64AudioChunk}]
    }
}));

// Send this to get a response
ws.send(JSON.stringify({type: "response.create"}));

Events that will ruin your day:

• error events don't tell you what's actually wrong
• response.audio.delta comes in random chunk sizes - your audio playback will sound robotic if you don't buffer properly
• response.done doesn't mean the audio finished playing, just that OpenAI finished sending

Production deployment reality

Memory leaks and connection management

WebSocket connections WILL die. Your app needs to handle:

• Connection drops every 5-10 minutes under load
• iOS background/foreground switching kills the connection silently
• Users refreshing the page mid-conversation (obvious but everyone forgets)

I spent forever debugging some memory leak that turned out to be audio buffers not getting garbage collected. Add this or suffer:

// Clean up audio context or you'll eat RAM
audioContext.close();
mediaRecorder.stream.getTracks().forEach(track => track.stop());

Cost monitoring because $18/hour adds up fast

Set up billing alerts before testing. Seriously. We burned through a bunch of money in an afternoon because a WebSocket reconnection loop was creating duplicate conversations.

Token counting gotchas:

• Audio tokens ≠ text tokens (1 second ≈ 50-100 tokens depending on content)
• Function calls add ~200ms latency and cost extra tokens
• Long conversations get expensive fast - truncate context aggressively

Browser compatibility nightmares

iOS Safari: Audio permissions are fucked. Sometimes works immediately, sometimes takes 30+ seconds after user grants permission. The Web Audio API pretends to work but outputs silence.

Chrome Mobile: Throttles background WebSocket connections. Your voice app will mysteriously stop working when users switch apps.

Firefox: Has its own audio resampling bugs. Some users will hear robotic voices no matter what you do.

Edge: Actually works pretty well, which is suspicious.

Integration patterns that don't suck

Web apps: Use React with useEffect for connection management. Don't try to be clever with global WebSocket state.

Phone systems: Twilio has community examples that actually work. Their WebRTC → WebSocket bridging saves weeks of development.

Mobile apps: Use WebRTC libraries like react-native-webrtc. Don't try to implement WebSocket audio streaming directly in React Native - you'll hate your life.

The API works great for demos. Production is where you'll learn why voice app developers drink heavily.