OpenAI Realtime API Browser & Mobile Integration

Building voice apps with the OpenAI Realtime API looks like a weekend project until you hit mobile Safari. What starts as "just send audio over WebSocket" becomes debugging why your app works perfectly on Chrome but sounds like Karen is talking through a fish tank on her iPhone 11. WebSocket audio debugging will eat your soul and shit out your sanity, then come back for seconds.

WebSocket vs WebRTC: Pick your nightmare

WebSocket Implementation (Looks easy, isn't):

• ✅ Simple to start - just send JSON over a connection
• ✅ Server logs actually make sense when debugging
• ❌ Audio format conversion is difficult
• ❌ Mobile browsers kill connections frequently
• ❌ iOS Safari audio permissions take forever

WebRTC Implementation (Complex as hell but works):

• ✅ Browsers actually handle audio properly for once
• ✅ Mobile doesn't kill connections as aggressively
• ✅ Echo cancellation works without hacks
• ❌ Setting up STUN/TURN servers is pain
• ❌ Debugging WebRTC feels like archaeology

They finally added WebRTC support in August 2025 after developers spent years begging for it. Thank fucking god. Realtime API version 2025-08-28 is the first one that doesn't hate mobile browsers.

The audio format conversion nightmare

The Realtime API wants PCM16 at 24kHz, but browsers shit out whatever format they feel like. Chrome Desktop usually gives you 48kHz float32, Safari does 44.1kHz because Apple hates standards, and mobile varies by device like it's playing roulette.

Took me 3 weeks to figure out why my app worked fine in development but made users sound like they were talking through a fish tank in production. Turns out it only happened on iPhone 11 running iOS 14.2.1 with AirPods connected. Specific as hell, impossible to reproduce locally.

// What OpenAI actually wants
const targetFormat = {
    sampleRate: 24000,  // Always 24kHz, no exceptions
    channels: 1,        // Mono only
    bitsPerSample: 16,  // PCM16 format
    encoding: 'pcm16'
};

// What you actually get from browsers (good luck!)
const realityCheck = {
    chrome: { sampleRate: 48000, format: 'float32' },    // At least consistent
    safari: { sampleRate: 44100, format: 'float32' },    // Different, naturally
    firefox: { sampleRate: 48000, format: 'float32' },   // Same as Chrome, miraculously
    mobile: { sampleRate: 'LOL who knows', format: 'depends on moon phase' }
};

Format conversion that actually works (after 50 Stack Overflow tabs and 2 mental breakdowns):

class AudioConverter {
    convertFloat32ToPCM16(float32Array, targetSampleRate = 24000) {
        // Resampling hell - 3 hours of my life I'll never get back
        const resampled = this.resample(float32Array, this.sourceSampleRate, targetSampleRate);
        
        // Float32 to int16 conversion - \"simple\" my ass
        const pcm16 = new Int16Array(resampled.length);
        for (let i = 0; i < resampled.length; i++) {
            // Don't clamp this and enjoy destroying your eardrums
            const clamped = Math.max(-1, Math.min(1, resampled[i]));
            pcm16[i] = Math.round(clamped * 32767);
        }
        
        return pcm16;
    }
    
    // Base64 encoding because WebSockets hate binary data
    encodeToBase64(pcm16Array) {
        const buffer = new ArrayBuffer(pcm16Array.length * 2);
        const view = new DataView(buffer);
        
        for (let i = 0; i < pcm16Array.length; i++) {
            view.setInt16(i * 2, pcm16Array[i], true); // little-endian or die
        }
        
        // Convert to base64 for WebSocket transmission
        return btoa(String.fromCharCode(...new Uint8Array(buffer)));
    }
}

Browser-specific pain points and workarounds

Chrome Desktop (Actually works, shocking!)

Chrome is the only browser that doesn't actively hate developers. It still has quirks that'll ruin your Tuesday.

// Chrome config that doesn't completely suck
const chromeConfig = {
    audio: {
        echoCancellation: true,
        noiseSuppression: true,
        autoGainControl: false, // Turn this off or sound goes to shit
        sampleRate: 48000
    }
};

// Chrome reconnection configuration
const reconnectConfig = {
    maxRetries: 10,
    baseDelay: 1000,
    maxDelay: 30000  // Usually reconnects quickly, sometimes takes longer
};

Safari Desktop/iOS audio handling

Safari audio permissions are more unpredictable than the weather in Seattle. Sometimes they work instantly, sometimes Safari sits there for 45 seconds pretending to think about it, sometimes the permission dialog never appears and you're left wondering if Safari is actually broken or just fucking with you personally.

// Safari permission dance that may or may not work
class SafariAudioHandler {
    async initializeAudio() {
        // Safari needs hand-holding for everything
        const AudioContext = window.AudioContext || window.webkitAudioContext;
        this.audioContext = new AudioContext({
            sampleRate: 44100 // Safari throws tantrums with other rates
        });
        
        // Safari: \"No audio until user clicks something\"
        if (this.audioContext.state === 'suspended') {
            await this.audioContext.resume();  // Please work this time
        }
        
        // iOS Safari requires longer permission timeout
        const isSafariMobile = /iPad|iPhone|iPod/.test(navigator.userAgent);
        const permissionTimeout = isSafariMobile ? 15000 : 5000;
        
        return this.requestMicrophoneAccess(permissionTimeout);
    }
}

Mobile Implementation challenges

React Native Integration (abandon hope):

React Native audio makes grown developers cry. iOS kills WebSocket connections the nanosecond a user switches apps, Android randomly switches audio routes mid-conversation because fuck consistency, and both platforms conspire to make your life miserable in increasingly creative ways.

// React Native nightmare fuel
import { RNSoundLevel } from 'react-native-sound-level';

class MobileRealtimeClient {
    async initializeMobile() {
        // Permissions dance - prepare for rejection
        const hasPermissions = await this.requestAudioPermissions();
        if (!hasPermissions) {
            throw new Error('User said no, app is now useless');
        }
        
        // Native audio processing - works 70% of the time
        this.audioProcessor = new NativeAudioProcessor({
            sampleRate: 24000,    // Will probably get 44100 anyway
            channels: 1,          // Pray for mono
            bitDepth: 16         // Fingers crossed
        });
        
        return this.connectWebSocket();  // Good luck
    }
    
    handleBackgroundState() {
        // Mobile OS: \"Background app? KILL THE AUDIO!\"
        AppState.addEventListener('change', (nextAppState) => {
            if (nextAppState === 'background') {
                this.pauseAudioProcessing();  // Audio dies anyway
            } else if (nextAppState === 'active') {
                this.resumeAudioProcessing(); // Maybe works
            }
        });
    }
}

Connection reliability (because everything will break)

Reconnection logic that assumes the worst:

Here's the thing about WebSocket connections - they will drop. Not if, when. Mobile browsers will kill them when users switch apps, WiFi will hiccup, and your connection will die at the worst possible moment.

class ReliableWebSocketClient {
    constructor(apiKey) {
        this.apiKey = apiKey;
        this.reconnectAttempts = 0;
        this.maxReconnects = 10;  // Usually gives up before this
        this.isIntentionallyClosed = false;
        this.connectionDeaths = 0;  // For debugging your sanity
    }
    
    connect() {
        this.ws = new WebSocket(
            `wss://api.openai.com/v1/realtime?model=gpt-realtime`,
            [],
            {
                headers: {
                    'Authorization': `Bearer ${this.apiKey}`,
                    'OpenAI-Beta': 'realtime=v1'
                }
            }
        );
        
        this.setupEventHandlers();
    }
    
    setupEventHandlers() {
        this.ws.onopen = () => {
            console.log('Connected to OpenAI (miracle!)');
            this.reconnectAttempts = 0;
            this.startHeartbeat();
        };
        
        this.ws.onclose = (event) => {
            this.connectionDeaths++;
            this.stopHeartbeat();
            console.log(`Connection died #${this.connectionDeaths}. Code: ${event.code}`);
            
            if (!this.isIntentionallyClosed && this.reconnectAttempts < this.maxReconnects) {
                const delay = Math.min(1000 * Math.pow(2, this.reconnectAttempts), 30000);
                console.log(`Reconnecting in ${delay}ms... (attempt ${this.reconnectAttempts + 1})`);
                
                setTimeout(() => {
                    this.reconnectAttempts++;
                    this.connect();  // Here we go again
                }, delay);
            } else {
                console.log('Giving up. Connection is cursed.');
            }
        };
        
        this.ws.onerror = (error) => {
            console.error('WebSocket error (surprise!):', error);
        };
    }
    
    startHeartbeat() {
        // Keep connection alive or browsers will kill it
        this.heartbeatInterval = setInterval(() => {
            if (this.ws.readyState === WebSocket.OPEN) {
                this.ws.send(JSON.stringify({ type: 'ping' }));
            }
        }, 30000); // 30 seconds - any longer and mobile kills it
    }
}

Hard truth: build your WebSocket client like you're preparing for nuclear war. Connections will drop at the worst possible moment, audio will break when users need it most, permissions will get revoked randomly, and mobile browsers will actively work against you. Plan for everything to fail spectacularly.

Essential references that saved my ass:

• WebSocket Browser Support - Check what actually works
• Web Audio API Compatibility - Spoiler: Safari sucks
• getUserMedia Browser Support - Mobile permissions nightmare guide
• Chrome Background Tab Throttling - Why your app dies in background
• iOS Safari WebSocket Issues - Known Safari WebSocket bugs
• React Native WebRTC Library - Community WebRTC for React Native
• MediaRecorder Browser Support - Audio recording compatibility matrix
• Web Audio Processing Best Practices - Mozilla's guide to not screwing up audio
• Chrome WebSocket Debugging - How to debug WebSocket connections
• Safari Audio Context Limitations - Apple's audio restrictions explained
• Mobile Browser Audio Permissions - Permission patterns that actually work

After surviving the browser compatibility gauntlet, mobile development hits you with its own special flavor of hell. Mobile apps don't just need to work - they need to survive users who switch apps mid-sentence, networks that die randomly, permissions that get revoked for mysterious reasons, and aggressive battery optimization that treats your audio app like malware.

React Native integration challenges (aka why I started drinking)

React Native audio is a nightmare wrapped in a disaster inside a catastrophe. Web APIs don't exist, community libraries are hit-or-miss maintained by people who've clearly never tried to actually use them in production, and you'll end up writing native code whether you want to or not.

Here's what works in practice:

// React Native disaster management
import { NativeModules, NativeEventEmitter } from 'react-native';

class ReactNativeRealtimeClient {
    constructor(apiKey) {
        this.apiKey = apiKey;
        this.audioModule = NativeModules.RealtimeAudioModule;  // You'll need to build this
        this.eventEmitter = new NativeEventEmitter(this.audioModule);
        this.connectionDeaths = 0;  // For your sanity tracking
    }
    
    async initialize() {
        // Permissions on mobile are like asking a toddler for candy
        const permissions = await this.audioModule.requestAudioPermissions();
        if (!permissions.granted) {
            throw new Error('User said no, pack it up');
        }
        
        // Native audio setup - this will break on iOS updates
        await this.audioModule.initializeAudio({
            sampleRate: 24000,    // What we want
            channels: 1,          // What we'll probably get
            bitDepth: 16         // Hope for the best
        });
        
        this.setupWebSocket();    // Prepare for connection death
        this.setupAudioHandlers(); // Cross fingers
    }
    
    setupWebSocket() {
        this.ws = new WebSocket(`wss://api.openai.com/v1/realtime?model=gpt-realtime`);
        
        this.ws.onmessage = (event) => {
            const data = JSON.parse(event.data);
            if (data.type === 'response.audio.delta') {
                this.audioModule.playAudioChunk(data.delta);  // Pray it works
            }
        };
        
        this.ws.onclose = () => {
            this.connectionDeaths++;  // Another one bites the dust
        };
    }
}

The harsh reality: You'll spend more time building the native audio module than writing the actual app. And then iOS 16.3.2 drops and breaks everything because Apple decided to "improve" WebRTC permissions.

PWA reality check (spoiler: it's still broken)

PWAs sound great in theory - native-like experience, offline support, all that jazz. For real-time audio? Still a nightmare, just with extra steps.

// PWA audio that maybe works 60% of the time
class PWARealtimeClient {
    constructor() {
        this.isStandalone = window.navigator.standalone;  // iOS only
        this.isActuallyUseful = false;  // Reality check
    }
    
    async initializePWA() {
        if (this.isStandalone) {
            // Standalone PWA - slightly less broken
            await this.tryStandaloneAudio();
        } else {
            // Browser PWA - same old WebSocket hell
            await this.tryBrowserAudio();
        }
    }
    
    async tryStandaloneAudio() {
        // Still needs permissions, still breaks randomly
        const stream = await navigator.mediaDevices.getUserMedia({
            audio: { autoGainControl: false }  // Turn off the stupid stuff
        });
        
        return this.setupAudioProcessing(stream);  // Good luck
    }
}

Reality check: PWAs are just websites wearing a cheap tuxedo. Audio permissions still suck, connections still die randomly, and iOS Safari still fucking hates you with the passion of a thousand suns.

When everything fails (prepare a backup plan)

WebSocket connections will fail. Not if, when. After the 5th connection death, you need a Plan B or users will delete your app.

// Fallback to basic STT/TTS when Realtime API shits the bed
class FallbackRealtimeClient {
    constructor(apiKey) {
        this.apiKey = apiKey;
        this.gaveUp = false;
    }
    
    async enableFallbackMode() {
        console.log('Realtime API failed, falling back to stone age tech');
        
        // Use browser's built-in speech recognition (Chrome only, basically)
        this.speechToText = new SpeechRecognition();
        this.textToSpeech = new SpeechSynthesis();
        
        this.speechToText.onresult = async (event) => {
            const transcript = event.results[0][0].transcript;
            
            // Old school GPT API call
            const response = await fetch('https://api.openai.com/v1/chat/completions', {
                method: 'POST',
                headers: { 'Authorization': `Bearer ${this.apiKey}` },
                body: JSON.stringify({
                    model: 'gpt-4',
                    messages: [{ role: 'user', content: transcript }]
                })
            });
            
            const data = await response.json();
            const utterance = new SpeechSynthesisUtterance(data.choices[0].message.content);
            this.textToSpeech.speak(utterance);  // Robotic voice but it works
        };
    }
}

This fallback sucks compared to Realtime API but at least it works when everything else is broken.

Battery optimization (because users hate when apps kill their phone)

Real-time audio processing eats battery like it's going out of style. Audio recording, WebSocket connections, and format conversion all gang up to murder your user's battery. Learned this the hard way when my app got flooded with 1-star reviews saying "killed my phone in 30 minutes, app is trash."

The unfortunate reality: Battery API support is more broken than Internet Explorer 6. iOS doesn't have it because Apple says fuck you, Android Chrome pretends to support it but returns garbage data half the time, and you'll mostly be guessing based on angry user reviews.

// Battery optimization that probably won't work everywhere
class BatteryOptimizedClient {
    constructor() {
        this.batteryAPI = navigator.battery;  // Spoiler: probably undefined
        this.batteryIsDying = false;
    }
    
    async tryBatteryOptimization() {
        if (this.batteryAPI) {
            const battery = await this.batteryAPI;
            
            if (battery.level < 0.2) {  // Phone dying
                this.enablePotatoMode();
            }
        } else {
            // Most browsers - just assume battery sucks
            this.enablePotatoMode();
        }
    }
    
    enablePotatoMode() {
        // Reduce everything to save battery
        this.audioConstraints = {
            sampleRate: 16000,      // Potato quality
            echoCancellation: false, // Too expensive
            noiseSuppression: false  // Bye audio quality
        };
        
        this.heartbeatInterval = 60000;  // Less frequent pings
    }
}

Platform-specific hell

Every platform has its own audio quirks that will make you question your life choices:

• iOS Safari: Permissions take forever, audio context lies about being ready, background kills everything
• Android Chrome: Auto-gain control ruins audio quality, background throttling murders connections
• Desktop Chrome: Actually works but throttles background tabs
• Firefox: Surprisingly solid, rare W
• Edge: Chrome clone, same behavior

// Platform detection and damage control
function getPlatformPain() {
    const ua = navigator.userAgent;
    
    if (/iPad|iPhone|iPod/.test(ua)) {
        return 'ios-safari-hell';
    } else if (/Android/.test(ua)) {
        return 'android-audio-conflicts';
    } else {
        return 'desktop-mostly-works';
    }
}

const platform = getPlatformPain();
// Good luck!

Bottom line: Start simple, test on actual devices (not just the simulator), build fallbacks for everything that can possibly fail, and keep a therapist on speed dial for when iOS updates break your shit again.

Essential reference links for mobile development:

• Mobile Web Audio Issues - GitHub issues for Web Audio API problems
• iOS Safari Audio Limitations - Apple's audio restrictions
• Android Chrome Audio Bug Tracker - Known Chrome audio issues
• React Native Track Player - Community audio solution for React Native
• WebRTC for Mobile Development - WebRTC mobile best practices
• PWA Audio Capabilities - Progressive Web App audio features
• Battery API Browser Support - Which browsers support battery monitoring
• Mobile Browser Audio Permissions - Permission patterns across platforms
• Cross-Platform Audio Testing - Testing strategies for audio apps
• Mobile Audio Performance Optimization - Performance tips for mobile audio