Vertex AI Text Embeddings API - Production Reality Check

I've been running Vertex AI embeddings in production for about 8 months. It's Google's text-to-vector API that converts your text into numbers that machines can work with for search and AI stuff.

Three models exist as of September 2025: text-embedding-005 (the newest one, released November 2024), the older text-embedding-004 (still available with no retirement date announced), and the new Gemini Embedding that costs about 6x more but handles multilingual content better.

The Reality of Using This Thing

Authentication is a pain in the ass. Setting up service accounts and IAM permissions isn't straightforward if you're new to Google Cloud. Took me 3 hours the first time because the docs assume you already know GCP. You need `aiplatform.user` role at minimum and have to set that GOOGLE_APPLICATION_CREDENTIALS environment variable or everything breaks with cryptic auth errors.

Costs surprised me. The regular embedding models charge per character (about $0.025 per million characters), but Gemini Embedding charges per token ($0.15 per million tokens). Our chatbot bill jumped from $150 to $800 per month when I switched to Gemini without doing the math. Use their count-tokens API for Gemini or you'll get fucked.

Both text-embedding models are still supported - no retirement dates announced. But I'd still recommend text-embedding-005 over 004 for new projects. I spent 2 weeks testing both against our existing vectors and 005 is slightly better at understanding code and technical docs.

When This Actually Works Well

RAG systems: We use it for document search in our support system. Works way better than keyword search - customers find answers in 30 seconds instead of 5 minutes. The semantic understanding actually gets what they're asking about.

Code documentation: text-embedding-005 understands programming concepts better than the old model. Our internal wiki search for API docs went from "mostly useless" to "actually helpful."

Multilingual stuff: If you have content in multiple languages, Gemini Embedding handles that without translating everything first. Costs more but saves the translation step.

What Doesn't Work

Rate limits hit hard. Default is 600 requests per minute. Hit that constantly during our data migration and had to implement exponential backoff. Takes 2-3 minutes to process 10K documents if you're not batching.

2048 token limit. Long documents get truncated without warning. You have to chunk everything yourself - I use 1024 tokens with 20% overlap. Pain in the ass but necessary.

Regional availability varies. Some models aren't available everywhere. Check before you commit to a region because migrating later sucks.

With that overview of what actually works and what doesn't, let's compare the different embedding models side-by-side so you can make an informed choice.

After 8 months running this in production, here's what actually goes wrong and how to fix it.

Authentication Hell

Google's service account system is needlessly complex. You'll spend half a day just getting the credentials right.

The errors you'll see:

• PermissionDenied: 403 The caller does not have permission - missing `aiplatform.user` role
• DefaultCredentialsError: Could not automatically determine credentials - `GOOGLE_APPLICATION_CREDENTIALS` not set
• InvalidArgument: 400 Location us-east1 is not supported - wrong region, some models only work in specific locations

What actually works:

1. Enable the Vertex AI API (this takes 2-3 minutes, don't refresh the page)
2. Create a service account, download the JSON key
3. Set export GOOGLE_APPLICATION_CREDENTIALS="/path/to/key.json"
4. Give it `aiplatform.user` role - don't use custom roles, they're broken

Cost Management (Because It Gets Expensive Fast)

Our embedding costs went from $200 to $2,000 per month when usage spiked. Token counting is weird - it's not 1:1 with characters.

Things that cost way more than expected:

• Repeated API calls for the same text (cache everything - reduced my API calls by 60%)
• Processing without chunking (you pay for characters even if they get truncated at 2048 tokens)
• Using Gemini Embedding for English-only content (6x price increase for marginal quality improvements)
• Processing large PDFs without checking character count first (some research papers hit 200K+ characters)

Batch processing saves 20% but requires uploading to Cloud Storage and waiting 30-60 minutes. Good for one-time migrations, useless for real-time apps.

Use `pip install google-cloud-aiplatform` and call `count_tokens()` first. Don't guess - I learned this the hard way when our bill hit $3,500 one month.

Rate Limiting That Actually Matters

600 requests/minute sounds like a lot until you try to process 50K documents. Hit rate limits constantly during data migrations.

Exponential backoff or you're fucked:

import time
import random

def embed_with_retry(text, max_retries=5):
    for attempt in range(max_retries):
        try:
            return client.predict(text)
        except Exception as e:
            if \"quota\" in str(e).lower():
                wait = (2 ** attempt) + random.uniform(0, 1)
                time.sleep(wait)
            else:
                raise e

Vector Storage Reality

Vertex AI Vector Search costs $0.32/hour per node minimum, even if you're not using it. For small applications, that's $230/month just to keep it running. Pinecone starts at $70/month and scales better.

BigQuery ML works if you're already using BigQuery, but query performance is shit for real-time search. Good for batch analytics, terrible for user-facing apps.

I use Weaviate deployed on GKE. More setup work but better control over costs and performance.

Migration Pain Points

text-embedding-004 to text-embedding-005 migration took 3 weeks (even though it's optional now):

1. Week 1: Testing similarity scores between models (they're different enough to matter)
2. Week 2: Reprocessing 2 million embeddings - took about 40 hours of API calls
3. Week 3: Fixing search relevance that broke because of vector changes

Save the old model version in metadata or you'll have no idea what's broken when similarity scores change. I learned this the hard way when users started complaining that search results were "worse" after the migration.

Debugging Production Issues

Most common failures:

• 413 Request Entity Too Large - text over 2,048 tokens, chunk it first
• 503 Service Unavailable: The service is currently unavailable - regional quota exhausted, retry in different region or wait 5-10 minutes
• 403 The caller does not have permission - service account key expired (they expire after 10 years by default, but admins can set shorter expiry)
• 400 Invalid argument: Location us-east1 is not supported for model text-embedding-005 - some models only work in specific regions
• 429 Rate limit exceeded followed by Retry after 60 seconds - you're hitting the 600 req/min limit, implement backoff

Set up billing alerts at $500, $1000, $2000. This API can get expensive fast and Google won't stop you from burning money.

The API works well once you get past the setup bullshit, but plan for 2-3 weeks of debugging and optimization before it's production-ready.

Speaking of common problems, here are the questions I get asked most often by other engineers who are trying to implement this stuff.