OpenAI Realtime API Function Calling: Production Intelligence

Configuration

Session Setup

const sessionConfig = {
  type: "session.update",
  session: {
    tools: [{
      type: "function",
      name: "getAccountBalance",
      description: "Get user account balance", // Keep short - long descriptions cause hallucinations
      parameters: {
        type: "object",
        properties: {
          accountId: { type: "string", description: "Account ID" }
        },
        required: ["accountId"]
      }
    }],
    truncation: {
      type: "retention_ratio",
      retention_ratio: 0.8  // Cuts 20% when hitting token limits - reduces costs by ~50%
    },
    max_response_output_tokens: 4096  // Don't set too high or it rambles
  }
};

Function Response Format

// Good response - prevents retries
{ "status": "success", "result": "Account balance: $150.25" }

// Bad response - causes 3x retry loops
{ "balance": 150.25 }  // AI doesn't know if this worked

Resource Requirements

Cost Structure

• Small screenshot: $0.02-0.04 per image
• Phone photo: $0.04-0.08 per image
• High-res image: $0.10+ per image
• Text conversation: ~$0.001-0.005 per message
• Long conversation: Can reach $5+ without limits

Performance Thresholds

• Under 2 seconds: Users don't notice function delays
• 2-5 seconds: Users get antsy, need "hold on" message
• Over 5 seconds: Users start hanging up
• Over 10 seconds: Return error and retry later

Token Usage Patterns

• Long conversations: 18k+ tokens (cost spike from $130 to $900/month)
• Image processing: Hundreds of tokens per image
• Function calls: Additional tokens for each call/response cycle

Critical Warnings

Production Failure Modes

Database Timeout Disasters

• Query timeouts cause dead silence - users hang up thinking call dropped
• Set 5-second maximum timeout or lose customers
• Connection pool exhaustion crashes entire app ("FATAL: too many clients already")

Cost Explosion Triggers

• Users upload massive photos without compression (4K screenshot = $0.15)
• Weekend conversations without limits ($200 → $3,247 bill)
• Rambling customers (one 30-min call = 18k tokens)
• Function retry loops from bad response formats

WebSocket Reliability Issues

• Safari 17.x randomly drops connections on mobile app switching
• Corporate firewalls kill connections after 60 seconds
• No conversation state preservation on disconnect
• Chrome 118+ blocks audio without user interaction first

Function Calling Gotchas

• Long function descriptions make AI hallucinate non-existent functions
• AI calls same function 3x if response format unclear
• Speech-to-parameter extraction fails with accents/background noise
• "conversation already has an active response" error from concurrent requests

Implementation Reality

Error Handling Patterns

// Aggressive timeout pattern
async function getSalesReport(period) {
  try {
    const result = await Promise.race([
      database.getSalesData(period),
      new Promise((_, reject) =>
        setTimeout(() => reject(new Error('timeout')), 5000)
      )
    ]);
    return result;
  } catch (error) {
    return {
      error: "That report is taking too long. Can I help with something else?"
    };
  }
}

Cost Protection

// Hard conversation limits
let conversationCost = 0;
const MAX_COST = 5.00; // $5 limit per conversation

function trackTokens(inputTokens, outputTokens) {
  const cost = (inputTokens * 0.000005) + (outputTokens * 0.00002);
  conversationCost += cost;

  if (conversationCost > MAX_COST) {
    ws.close(1000, "Cost limit reached");
    return false;
  }
  return true;
}

Image Compression Requirements

// Mandatory compression to survive costs
function compressImage(file) {
  const maxSize = 800; // Keep small for budget survival
  const quality = 0.7; // 70% quality usually sufficient
  // Implementation reduces costs by ~60-80%
}

Decision Criteria

When NOT to Use

• Customer service replacement (too unreliable - functions fail constantly)
• High-volume image processing (costs unsustainable)
• Accent-heavy user base (speech extraction fails)
• Budget-sensitive applications (costs spike unpredictably)

Suitable Use Cases

• Internal tools with compressed data
• Demo/prototype environments
• Low-volume customer support (with human backup)
• Education applications (if image costs controlled)

Migration Intelligence

Beta vs GA Changes

Feature	Beta Behavior	GA Improvement	Production Impact
Function Flow	Dead silence during calls	Continues talking	Eliminates hangup problem
Cost	High with no truncation	Slightly lower + truncation	Still expensive but manageable
Image Support	None	Available but costly	Cool feature, budget killer
Error Handling	Raw errors exposed	Better fallbacks	Less embarrassing failures

Breaking Changes

• WebSocket connection management unchanged (still fragile)
• Token counting methodology same (images still expensive)
• Function calling syntax identical (existing code works)

Monitoring Requirements

Essential Metrics

• Function response time (alert > 5 seconds)
• Daily costs (alert at 50% budget)
• Error rates (alert > 10%)
• WebSocket disconnection frequency
• Image upload costs per session

Failure Indicators

• Multiple function retries for same request
• Cost spikes without usage increase
• High WebSocket reconnection rates
• User session abandonment after function calls

Operational Intelligence

Production Deployment Reality

• Requires fortress of error handling around core API
• Database connection pooling mandatory (max 10 connections)
• Aggressive caching needed for repeated queries
• Hard limits on everything: cost, time, tokens, uploads

Browser Compatibility Issues

• Safari WebSocket reliability poor on mobile
• Chrome requires user interaction before audio
• Long conversations cause browser memory leaks
• WebRTC compatibility varies significantly

Security Considerations

• No built-in authentication (implement separately)
• Raw database errors expose internal architecture
• Function parameters transmitted in clear text
• No session state encryption or persistence

This API works for demos and impresses investors, but production deployment requires extensive defensive programming, cost monitoring, and user experience compromises.