MLServer - Serve ML Models Without Writing Another Flask Wrapper

Serving ML models in production is harder than it should be. You train your model in Jupyter, it works great, then someone asks "how do we actually use this thing?" That's where MLServer comes in - it handles the HTTP/gRPC serving bullshit so you don't have to write another Flask wrapper that dies under load.

The Problem MLServer Solves

Every ML engineer has been here: you have a working model and need to expose it as an API. You could write custom Flask code, but that breaks when you add a second model. TensorFlow Serving only works with TensorFlow. TorchServe only works with PyTorch. Most of these solutions assume you only have one framework in your stack, which is adorable.

MLServer works with scikit-learn, XGBoost, LightGBM, MLflow, and HuggingFace Transformers. Plus it implements the V2 Inference Protocol, which means it'll work with KServe without making you rewrite everything when you eventually move to Kubernetes.

What Makes It Different

Multi-Model Serving: You can serve multiple models in one process instead of spinning up separate containers for each one. Works great until one model memory leaks and kills everything else.

Adaptive Batching: MLServer batches requests automatically based on timing and batch size limits. This actually improves throughput without you having to implement batching logic yourself (which you probably would have screwed up anyway).

Parallel Workers: Multiple inference processes can run on the same machine. Useful when your model is CPU-bound and you have cores to spare.

Production Reality Check

MLServer includes Prometheus metrics and OpenTelemetry support, which is more than most custom serving scripts provide. It handles graceful shutdown and health checks without you having to remember to implement them. The monitoring capabilities integrate with standard observability stacks that ops teams already use.

The current version 1.7.1 supports Python 3.9-3.12, which covers most reasonable deployment environments. They keep backward compatibility, unlike some projects that break your deployment with every minor release. Check the release notes and migration guide when upgrading.

MLServer isn't perfect - the Docker images are large, memory usage can be unpredictable, and configuration has some gotchas. But it beats writing your own serving infrastructure from scratch. The community benchmarks show it's competitive with alternatives like TorchServe and BentoML for most workloads.

MLServer tries to make serving models easy, but "easy" is relative when you're dealing with Python dependencies and Docker. Here's what actually happens when you try to get it working.

Installation Hell

You need Python 3.9+ (don't try 3.8, it won't work). Start with:

pip install mlserver

This will probably work. The fun starts when you install framework-specific runtimes:

pip install mlserver-sklearn  # Usually fine
pip install mlserver-xgboost  # Might conflict with existing XGBoost
pip install mlserver-huggingface  # Good luck with the PyTorch dependencies - they conflict with everything and CUDA versions are hell

Pro tip: Use a virtual environment or you'll fuck up your system Python. Docker is safer but the images are huge (2GB+ because they include everything). Check the installation guide and Docker deployment docs for specific setup instructions.

Configuration Files From Hell

MLServer needs two JSON files that it's very picky about. First, settings.json:

{
    "debug": true,
    "host": "0.0.0.0", 
    "http_port": 8080,
    "grpc_port": 8081,
    "metrics_port": 8082
}

Set debug: true or you'll hate yourself when things break. The ports are configurable, which matters when you inevitably have conflicts.

Then model-settings.json for each model:

{
    "name": "my-model",
    "version": "v0.1.0",
    "implementation": "mlserver_sklearn.sklearn_model.SKLearnModel", 
    "parameters": {
        "uri": "./model.joblib"
    }
}

Common fuckups:

• Wrong implementation path (copy-paste from docs, double-check it)
• Model file path is wrong (use absolute paths to be safe)
• JSON syntax errors (trailing commas will kill you)
• Model file doesn't exist or has wrong permissions

Starting the Server (If You're Lucky)

Run mlserver start . from your model directory. If everything works:

• REST API at http://localhost:8080/v2/models/my-model/infer
• gRPC at localhost:8081
• Metrics at http://localhost:8082/metrics

When it doesn't work (which it won't):

• ModuleNotFoundError: Your runtime isn't installed properly
• FileNotFoundError: Model path is wrong, use absolute paths
• Port already in use: Something else is using 8080, change the port
• ECONNREFUSED: Check if the server actually started, look at the logs

Enable debug logging: MLSERVER_LOGGING_LEVEL=DEBUG mlserver start . - check the logging documentation for more configuration options.

Memory and Performance Gotchas

MLServer loads models into memory on startup. Large models will eat RAM fast. Multi-model serving shares memory, which is great until one model memory leaks.

The parallel inference feature spawns worker processes. This helps with CPU-bound models but uses more memory. Start with default settings, tune later when you actually have load.

Adaptive batching can improve throughput but adds latency. Configure batch size and timeout based on your actual traffic patterns, not theoretical optimums.

Docker and Kubernetes Pain

The official Docker images are convenient but large. They include multiple runtime dependencies whether you need them or not.

For Kubernetes with KServe, MLServer works well because it follows the V2 protocol. But you'll spend time debugging:

• Resource limits (always set memory limits higher than you think)
• Health check timeouts (model loading takes time)
• Persistent volumes for model files (don't put models in the image)

Check the KServe deployment guide and Kubernetes examples for working configurations.

Model repositories let you load/unload models dynamically. Cool in theory, but adds complexity. Start simple with static model loading. The model management API documentation covers advanced scenarios.

What Actually Works in Production

Use Prometheus metrics - they're built-in and actually useful for monitoring. The health checks work properly, unlike some custom serving scripts. Set up Grafana dashboards to visualize performance metrics and Alertmanager for incident response.

Log everything with debug: true until you understand what's happening. MLServer's error messages are usually helpful, which is more than you can say for most Python ML infrastructure. The troubleshooting guide covers common production issues and their solutions.