MLServer - Production ML Model Serving

Core Function

Python inference server that handles HTTP/gRPC serving for multiple ML frameworks without custom Flask wrappers.

Framework Support

• Supported: scikit-learn, XGBoost, LightGBM, MLflow, HuggingFace Transformers
• Protocol: V2 Inference Protocol (KServe compatible)
• Multi-framework: Single server can serve models from different frameworks simultaneously

Critical Production Requirements

Python Version Support

• Required: Python 3.9-3.12
• Breaking: Python 3.8 will not work
• Current Version: 1.7.1 (maintains backward compatibility)

Installation Dependencies

pip install mlserver                    # Base package (no runtimes)
pip install mlserver-sklearn           # Usually works
pip install mlserver-xgboost          # May conflict with existing XGBoost
pip install mlserver-huggingface      # PyTorch dependency conflicts common

Critical Warning: Base package includes NO model runtimes. Framework-specific packages required.

Configuration Files (Failure-Prone)

settings.json (Server Configuration)

{
    "debug": true,                      # REQUIRED for troubleshooting
    "host": "0.0.0.0", 
    "http_port": 8080,                 # Conflicts common
    "grpc_port": 8081,
    "metrics_port": 8082
}

model-settings.json (Per Model)

{
    "name": "my-model",
    "version": "v0.1.0",
    "implementation": "mlserver_sklearn.sklearn_model.SKLearnModel",
    "parameters": {
        "uri": "./model.joblib"        # Use absolute paths
    }
}

Common Failures:

• Wrong implementation path (copy-paste errors)
• Relative file paths fail
• JSON syntax errors (trailing commas)
• Model file permissions/existence

Memory and Performance Characteristics

Memory Usage

• Behavior: Unpredictable, model-dependent
• Multi-model: Shared memory space (memory leaks affect all models)
• Large models: RAM consumption on startup
• Worker processes: Parallel inference increases memory usage

Performance Features

• Adaptive Batching: Automatic request batching (adds latency for throughput)
• Parallel Workers: Multiple inference processes (CPU-bound models benefit)
• Default Strategy: Start with defaults, tune after load testing

Critical Failure Modes

Startup Failures

• ModuleNotFoundError: Runtime package not installed
• FileNotFoundError: Model path incorrect (use absolute paths)
• Port already in use: Change ports in settings.json
• ECONNREFUSED: Server failed to start (check debug logs)

Runtime Failures

• OOM Errors: Model + batching + workers = memory explosion
• 500 Errors: Input format mismatch, corrupted models, missing dependencies
• Memory Leaks: One model affects all others in multi-model setup

Docker Issues

• Image Size: 2GB+ (includes all runtime dependencies)
• Solution: Custom images with only required runtimes

Production Deployment Gotchas

Kubernetes/KServe

• Memory Limits: Set 2x higher than estimated need
• Health Checks: Model loading takes time (extend timeouts)
• Persistent Volumes: Required for model files (don't embed in image)
• Resource Conflicts: Network policies, image pull secrets

Monitoring (Built-in)

• Prometheus: Port 8082, genuinely useful metrics
• OpenTelemetry: Distributed tracing (requires configuration)
• Health Checks: Proper implementation included
• Logging: Set debug: true until system stabilized

Framework Comparison Matrix

Aspect	MLServer	TensorFlow Serving	TorchServe	Ray Serve	Triton
Setup Difficulty	Weekend	Week+	Java nightmare	Ray learning curve	CUDA driver hell
Memory Predictability	Unpredictable	Stable but high	Medium, leaks possible	Ray overhead dependent	High but predictable
Error Quality	Usually helpful	Cryptic C++ traces	Java exceptions	"Ray worker died"	CUDA OOM
Production Readiness	Works with gotchas	Google battle-tested	Facebook production	Anyscale backing	NVIDIA production
Documentation	Decent, examples work	Google-tier	Sparse, outdated	Hit/miss	NVIDIA quality

Time and Resource Investment

Learning Curve

• Initial Setup: Weekend to basic functionality
• Production Ready: 1-2 weeks understanding gotchas
• Expertise: Month+ for complex multi-model scenarios

Operational Costs

• Memory: Higher than single-framework solutions
• Debugging Time: Moderate (good error messages)
• Maintenance: Regular updates maintain compatibility

Decision Criteria

Use MLServer When:

• Multiple ML frameworks required
• KServe/Kubernetes deployment planned
• Team lacks serving infrastructure expertise
• Standard monitoring/observability needed

Avoid MLServer When:

• Single framework (use framework-specific servers)
• Extreme performance requirements (consider Triton)
• Resource-constrained environments (image size issues)
• Custom serving logic requirements extensive

Critical Success Factors

Must-Do Configuration

1. Enable debug logging initially
2. Use absolute paths for model files
3. Set memory limits 2x estimated requirements
4. Configure health check timeouts for model loading time
5. Monitor memory usage patterns before scaling

Performance Optimization Sequence

1. Establish baseline with default settings
2. Monitor actual traffic patterns
3. Tune batch sizes based on latency requirements
4. Adjust worker processes based on CPU utilization
5. Optimize memory allocation after usage patterns established

Troubleshooting Protocol

1. Enable MLSERVER_LOGGING_LEVEL=DEBUG
2. Verify model file accessibility and format
3. Check runtime package installation
4. Monitor resource usage during failures
5. Validate input data format against model expectations

Community and Support Quality

• GitHub Issues: Active, solutions often exist
• Seldon Slack: Responsive maintainers
• Documentation: Generally accurate with working examples
• Release Cycle: Regular updates, backward compatibility maintained