Weights & Biases - Because Spreadsheet Tracking Died in 2019

What is Weights & Biases?

Sick of tracking experiments with spreadsheets? Weights & Biases (W&B) is an ML experiment tracking platform that tracks your shit without falling over. If you've ever lost track of which hyperparameters produced your best model, or spent hours trying to reproduce results from three weeks ago, W&B solves that problem.

Started by engineers who got tired of managing experiments with spreadsheets and folder names like "model_final_v2_ACTUALLY_FINAL", W&B tracks your training runs automatically. The web UI doesn't look like it was built in 2005, which is already better than most ML tools. Big companies use it because it scales from solo PhD projects to enterprise teams without completely falling apart.

What W&B Actually Does

Experiment Tracking: Logs metrics, hyperparameters, and system info as your models train. Finally you can compare hundreds of runs side-by-side instead of squinting at terminal output or trying to remember what you changed between runs.

Model Registry: Stores your trained models so you can actually find the model you deployed to production. No more "model_best.pkl" vs "model_best_final.pkl" debates.

Dataset Versioning: Tracks dataset changes. Useful when your data team updates the training set and suddenly your model performance tanks.

Hyperparameter Sweeps: Automatically tries different parameter combinations. Beats manually editing config files and forgetting what you tested.

Integration Reality Check

W&B works with the usual suspects: PyTorch, TensorFlow, Hugging Face, scikit-learn. Adding wandb.init() to your training script is easy. Getting it to work when PyTorch 2.1.0 breaks their auto-logging, your custom data loader throws NoneType errors, and your Docker container can't reach their servers because of networking bullshit? That's where you'll spend your weekend and most of Monday morning.

The integrations mostly work as advertised, until you need distributed training with a custom dataset and suddenly you're debugging TypeError: 'NoneType' object is not subscriptable at 2AM. Their docs cover the happy path well enough, but anything remotely complex means digging through 500 GitHub issues to find the one comment that actually fixes your problem.

Two Main Products

W&B Models: The original experiment tracking everyone uses. Solid for traditional ML workflows.

W&B Weave: Their newer LLM and AI application platform. Still beta but doesn't completely suck for prompt engineering and LLM evaluation work.

W&B Models: Here's Where It Breaks

Experiment Tracking Reality: W&B tracking logs your metrics, hyperparameters, and system info automatically. Works great until your training script crashes and you lose 6 hours of metrics because you forgot to call wandb.finish(). The UI is actually decent for comparing runs, but expect it to timeout on large experiments.

Either wrap your training loop with wandb.finish() or use atexit.register(wandb.finish) or you'll lose data when shit inevitably breaks. I learned this the hard way after losing 6 hours of training metrics when my server died. Don't log metrics faster than once per second or you'll hit undocumented rate limits that just return "Request failed" with no other context.

Version Gotchas: PyTorch 2.1.0 breaks their auto-logging completely, 2.0.1 works fine, 2.1.1 fixes the W&B issue but breaks something else entirely. They hook into private APIs that PyTorch changes without warning. Always check their GitHub issues before upgrading - someone else definitely hit the same problem 3 weeks ago and posted a workaround buried in comment #47.

Hyperparameter Sweeps: W&B Sweeps automatically tries different parameter combinations. Beats manually editing YAML files, but can spawn way too many processes if you're not careful with the agent configuration. Misconfigured sweeps can spawn infinite instances. Set proper resource limits or prepare for surprise AWS bills.

The early stopping works but is aggressive - you might lose promising runs that start slow. Monitor your sweeps because they occasionally get stuck and stop launching new runs without telling you. Use proper sweep configuration and parallel execution strategies for better results.

Model Registry Pain Points: The model registry tracks your models through deployment stages. Useful for avoiding "model_final_v2_ACTUALLY_FINAL.pkl" situations, but the API for programmatic model promotion is clunky. Expect to write wrapper scripts.

Artifact uploads fail silently more often than they should. Lost a 2GB model file because the upload "succeeded" but the file wasn't actually there. Could have been network issues, could have been their chunked upload system choking, could have been my shitty exception handling - the error just said "Artifact upload failed" with zero useful context. Always verify with artifact.verify() and check the web UI manually before deleting anything local.

Artifacts System Gotchas: W&B Artifacts handles dataset versioning. Works well for small datasets, but large file uploads are unreliable. Their chunked upload system helps with big files but doesn't retry failed chunks automatically.

The deduplication is clever but confusing as hell when artifacts share data - you'll spend hours figuring out why your 10GB model shows up as 200MB.

W&B Weave: Why This Sucks

LLM Tracing: Weave traces LLM calls and agent workflows. Still beta-ish but shows promise for debugging prompt engineering. The automatic tracing catches most calls but manual instrumentation is needed for custom setups.

Token counting is approximate - don't rely on it for billing accuracy. The cost tracking helps but verify against your actual OpenAI/Anthropic bills.

Evaluation Framework: Built-in evals for LLM applications work for standard cases. Custom evaluations require more setup than their marketing suggests. The semantic similarity metrics are decent but not magic - garbage prompts still produce garbage results.

Prompt Management: Version control for prompts with A/B testing. Useful for tracking what prompt changes actually improve performance vs. just feeling better. The deployment automation is basic - you'll probably want to build your own promotion pipeline.

Enterprise: Prepare Your Wallet

W&B has the usual enterprise checkboxes: SOC 2, GDPR, SSO. Their security documentation is thorough. On-premises deployment is available but requires weeks of setup - plan for weeks, not days.

Deployment options include SaaS (easiest), dedicated cloud (middle ground), and on-prem (your headache). Enterprise support is actually responsive, unlike some vendors. Just prepare for "contact us" pricing that scales with your pain.

Here's What'll Bite You

• Storage costs add up fast if you log everything
• The free tier limit hits sooner than you think - plan for scaling costs
• Distributed training setup requires reading GitHub issues, not just docs
• Team permissions are more complex than they need to be - use service accounts wisely
• Export functionality exists but isn't obvious - vendor lock-in concerns are valid