VectorDBBench: AI-Optimized Technical Reference

Tool Overview

Purpose: Open-source vector database benchmarking tool by Zilliz (Milvus creators)
Bias Warning: Tool creators have financial interest in Milvus performance, but methodology is transparent and Milvus doesn't always win
Primary Value: Best available benchmarking option despite limitations - alternatives are worse

Configuration

System Requirements

Minimum Viable:

• 16GB RAM, 8 cores, SSD storage
• Python 3.11+ (hard requirement due to typing features)
• Good network connection

Production Realistic:

• 32GB RAM, 16 cores, NVMe storage
• For 10M+ vectors: 64GB+ RAM required

Critical Installation Issues:

# Standard installation often fails
pip install vectordb-bench[all] --force-reinstall --no-cache-dir
# Required due to protobuf dependency conflicts (50% failure rate)

Docker Alternative: Works better but consumes 8GB+ RAM for small tests

Supported Databases

• Coverage: 20+ vector databases including Pinecone, Qdrant, Milvus, Weaviate, OpenSearch, PostgreSQL pgvector
• Real Datasets: SIFT, GIST, Cohere Wikipedia embeddings, OpenAI embeddings

Performance Benchmarking Scenarios

Insert Performance

• Purpose: Real-time ingestion pipeline capacity testing
• Critical For: Systems requiring continuous vector updates
• Measures: Insertion throughput under varying load conditions

Search Performance

• Metrics: QPS, P99 latency under concurrent load
• Real-World Impact: Most databases behave differently under parallel query load
• Key Insight: P99 latency matters more than average QPS for user experience

Filtered Search

• Critical Capability: Metadata filtering combined with vector similarity
• Failure Point: Where most vector databases completely break down
• Production Reality: Essential for real-world applications, poorly tested by most benchmarks

Resource Requirements

Time Investment

• Full benchmark run: 2-6 hours
• Failure probability: High - random disconnections and hanging processes common
• Memory leak issues: Versions 1.0.6 had Pinecone client memory leaks, fixed in 1.0.7

Financial Costs

• Cloud service testing: $200-500 for comprehensive benchmark
• Pinecone cost surprise: $80 in credits before learning to limit test duration
• AWS resources: $340 for single full benchmark due to poor resource cleanup

Human Expertise Required

• Configuration complexity: Database-specific configs poorly documented
• Example: 3 hours to fix Milvus HNSW parameters for 1M+ vectors
• Network troubleshooting: Cloud databases frequently timeout without retry logic

Critical Warnings

What Official Documentation Doesn't Tell You

Memory Usage Reality:

• Benchmarking 5M vectors requires 32GB+ RAM or OOM failures
• Process dies without graceful degradation

Performance Variability:

• Results vary 20-30% between runs on same hardware
• Cloud database performance highly inconsistent
• Network conditions dramatically affect results

Connection Stability Issues:

• Qdrant Cloud times out on network hiccups without retry
• ElasticSearch randomly disconnects during long benchmarks
• Streaming tests frequently hang requiring manual process termination

Breaking Points and Failure Modes

CI/CD Integration:

• Don't do it - random failures and massive costs
• Better: Monthly scheduled runs on dedicated hardware

Configuration Gotchas:

• Default HNSW parameters terrible for 1M+ vectors
• Database-specific tuning requires source code reading
• Error messages are cryptic Pydantic validation failures

Cloud Service Limitations:

• Rate limiting kicks in unexpectedly
• Network egress charges not documented
• Filtering performance often 50% worse than benchmarks

Performance Expectations by Database

Database	QPS Range	P99 Latency	Cost Reality	Major Issues
ZillizCloud	6k-12k	2-5ms	Expensive	Hard rate limiting
Milvus Self-hosted	2k-5k	2-8ms	Good value	Memory config critical
Qdrant Cloud	1.5k-4k	3-12ms	Reasonable	Flaky under sustained load
Pinecone	1k-3k	4-15ms	Expensive	Poor filtering performance
Weaviate	800-2.5k	5-20ms	Complex	GraphQL query overhead
OpenSearch	500-3k	7-25ms	Variable	Force merge sometimes helps

Decision Criteria

When VectorDBBench Is Worth Using

• Need standardized comparison across multiple databases
• Evaluating production workload scenarios (insert + search + filtering)
• Have dedicated hardware and time budget
• Can tolerate 20-30% result variance

When to Use Alternatives

• Single database optimization: Use database-specific tools
• Algorithm research: Use ANN-Benchmarks
• Cost-sensitive evaluation: Custom lightweight scripts
• CI/CD integration needs: Build minimal custom tests

Production Planning Reality Check

Multiply benchmark results by 3-5x for production estimates due to:

• Network jitter (users not in same datacenter)
• Load spikes (traffic never perfectly smooth)
• Runtime garbage collection pauses
• Infrastructure quality differences

Implementation Recommendations

Benchmarking Schedule

• Monthly: If performance-critical system
• Quarterly: For stable production systems
• Trigger events: Version upgrades, query pattern changes, unexplained performance drops

Custom Dataset Testing

• Essential: Generic benchmarks don't represent your data clustering patterns
• Performance impact: 40% variance between SIFT and document embeddings observed
• Configuration: YAML-based system works but documentation poor

Cost Optimization

• Use Docker deployment for resource control
• Limit test duration for cloud services
• Monitor for resource cleanup failures
• Budget 3-5x estimated cloud costs for comprehensive testing

Quality Assessment

Trustworthiness Factors

Positive Indicators:

• Open source methodology
• Milvus doesn't always win in results
• Uses real datasets vs synthetic data
• Tests actual production scenarios (filtering, concurrency)

Bias Indicators:

• Created by Milvus vendor (Zilliz)
• Test scenario selection may favor Milvus architecture
• Highlighting choices emphasize Milvus strengths

Comparison to Vendor Benchmarks

VectorDBBench advantages:

• Standardized methodology across databases
• Real-world dataset usage
• Concurrent testing scenarios
• Filtering performance measurement

Vendor benchmark issues:

• Cherry-picked datasets favoring specific architectures
• Unrealistic hardware configurations
• Avoidance of weakness scenarios
• Marketing-driven result presentation

Essential Resources

• GitHub Repository: Source code and issue tracking
• PyPI Package: Installation and versions
• Performance Leaderboard: Live benchmark results
• Troubleshooting: Community support and known issues
• Configuration Examples: Setup templates