Mojo Development Workflow - What They Don't Tell You About Daily Development

Let's start with the elephant in the room. As of March 2025, Modular officially deprecated the REPL. They walked it back after community backlash, but the writing's on the wall - interactive development isn't their priority anymore.

This isn't just about losing a cool feature. The REPL was how most of us debugged our Mojo code, tested snippets, and figured out why our "simple" matrix multiplication was segfaulting. Now we're back to the stone age of print-debugging and recompiling entire binaries to test a one-line change.

What Actually Broke When the REPL Died

No more interactive debugging. You can't just fire up a REPL, import your broken function, and poke at it until you understand what's going wrong. Everything has to be a complete program now.

Docstring tests are gone. All those nice examples in your code documentation? They don't run anymore. The testing framework used to execute them via the REPL, so now they're just pretty comments.

Learning curve became a cliff. New developers could experiment with Mojo syntax interactively. Now they have to write complete programs and deal with the compiler's cryptic error messages from day one.

No more calculator mode. Python developers use the REPL as a smart calculator. Need to check [math.pow(2, 64)](https://docs.python.org/3/library/math.html#math.pow)? Fire up Python and try it. With Mojo, you're back to writing throwaway .mojo files.

The VS Code Extension: Your Only Real IDE Option

The VS Code extension is literally your only choice if you want any IDE support at all. Here's what actually works:

Syntax highlighting: Works fine. Basic but functional.

Debugging: Works when the stars align. The LLDB integration is real, but good luck when it shows you MLIR intermediate representation instead of your actual code.

Autocompletion: Exists for basic types. Breaks spectacularly on parameterized functions and anything involving generics.

Error reporting: Shows you cryptic compiler errors inline, which is better than nothing but not by much.

What doesn't work: refactoring tools, go-to-definition for complex types, intelligent code suggestions, or basically anything you'd expect from a modern IDE.

Setting Up a Development Environment That Won't Drive You Insane

Here's what I've learned after 8 months of daily Mojo development:

1. Keep Python Around for Everything

Don't go all-in on Mojo. Keep your data loading, business logic, and anything that doesn't need performance in Python. Use Mojo only for the hot paths that you've actually profiled.

## Project structure that doesn't suck
project/
├── python_src/          # All your business logic
│   ├── data_loading.py
│   ├── preprocessing.py
│   └── main.py
├── mojo_kernels/        # Only the performance-critical stuff
│   ├── matrix_ops.mojo
│   └── custom_loss.mojo
└── tests/
    ├── test_python_logic.py
    └── test_mojo_kernels.mojo

2. Build a Debugging Workflow That Actually Works

Since the REPL is dead, here's how to debug without losing your mind:

Use `breakpoint()` everywhere. The builtin breakpoint() function is your best friend. When VS Code debugging fails (and it will), programmatic breakpoints work.

Keep a scratch.mojo file. Create a throwaway file where you can test snippets quickly. Much faster than setting up a full project structure for every experiment.

Log everything. Print statements are primitive but they work. MLIR errors are useless, but seeing your actual data values helps.

from testing import assert_equal

fn debug_matrix_multiply():
    var a = Matrix[Float32](2, 3)
    var b = Matrix[Float32](3, 2)

    # Fill with test data
    for i in range(a.rows):
        for j in range(a.cols):
            a[i, j] = Float32(i * a.cols + j)

    print("Matrix A:")
    print(a)  # Actually show what your data looks like

    breakpoint()  # Stop here to inspect state

    var result = a @ b
    print("Result shape:", result.rows, "x", result.cols)

3. Testing Without Docstrings

Since docstring tests are dead, here's how to actually test your Mojo code:

Use dedicated test files. Create test_yourmodule.mojo files with explicit test functions.

Steal from the standard library. Look at Mojo's own test files for patterns that actually work.

Keep tests simple. Complex test setups break in weird ways. Write lots of small, focused tests instead of comprehensive integration tests.

4. Development Workflow for Hybrid Projects

This is what actually works when you're mixing Python and Mojo:

1. Prototype everything in Python first. Get the algorithm working with numpy/torch.
2. Profile to find bottlenecks. Use cProfile to identify what's actually slow.
3. Port only the hot path to Mojo. Don't rewrite everything, just the 10% that matters.
4. Keep Python fallbacks. When (not if) your Mojo code breaks, you need a working Python version.
5. Test both versions. Make sure Python and Mojo implementations give the same results.

Error Messages That Make You Question Reality

Let's talk about the real development experience. When your Mojo code breaks, you get errors like this:

error: 'linalg.generic' op operand #0 does not dominate this use
  %2 = linalg.generic {indexing_maps = [#map], iterator_types = ["parallel"]}
       ^

What the fuck does that mean? Here's what I've learned:

MLIR errors are compiler internals. They're not meant for human consumption. The compiler is showing you its internal representation, not your actual code.

Line numbers are lies. The error points to the MLIR IR, not your source code. The actual bug could be anywhere.

Context is everything. Start with the simplest possible code and add complexity one line at a time. When it breaks, you know the last line caused it.

Discord is your debugger. The Modular Discord has humans who can translate MLIR gibberish into actual programming concepts.

The debugging experience is getting better (supposedly), but right now it's like debugging assembly code while blindfolded. Plan for frustration.

Here's a fun one that took us 4 hours to debug: Mojo 24.4 has a bug where using @parameter with certain generic types causes silent memory corruption. The error only shows up when you run the same code path 1000+ times. Version 24.5 fixed it, but broke something else with SIMD alignment.

Now let's dig into the specific tools and workflows that might save your sanity when dealing with production deployments.

The good news? Mojo compiles to regular binaries that deploy like any other executable. The bad news? Those binaries can segfault in creative ways that make debugging in production a special kind of hell.

Docker Deployment That Actually Works

Modular provides Docker containers, but they're focused on development, not production. Here's what actually works in production:

## Start with a minimal base - Mojo binaries are self-contained
FROM ubuntu:22.04

## Install minimal runtime dependencies
RUN apt-get update && apt-get install -y \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

## Copy your compiled Mojo binary
COPY --from=build-stage /app/target/release/my_mojo_app /usr/local/bin/
COPY --from=build-stage /app/config/ /app/config/

## Create non-root user for security
RUN useradd -r -s /bin/false appuser
USER appuser

EXPOSE 8080
CMD [\"my_mojo_app\"]

The build stage is crucial. Compile your Mojo code in a container with the full Modular toolkit, then copy only the binary to a minimal runtime container. The compiled binary includes all the MLIR runtime dependencies you need.

Memory limits matter. Mojo's memory management is more predictable than Python, but when it runs out of memory, it doesn't gracefully degrade. It just dies. Set container memory limits and monitor actual usage. We learned this when our inference service would randomly die under load - turns out we were hitting the 1GB container limit.

No Python packaging hell. This is actually a win. No pip dependencies, no virtual environments, no "works on my machine" issues with different Python versions. The binary includes everything.

Kubernetes Deployment Patterns

Mojo binaries work fine in Kubernetes, but you'll hit some unique issues:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mojo-inference-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: mojo-inference
  template:
    metadata:
      labels:
        app: mojo-inference
    spec:
      containers:
      - name: inference
        image: myregistry/mojo-inference:v1.2.3
        resources:
          requests:
            memory: \"1Gi\"
            cpu: \"500m\"
          limits:
            memory: \"2Gi\"
            cpu: \"2000m\"
        # Critical: Set this for GPU workloads
        env:
        - name: NVIDIA_VISIBLE_DEVICES
          value: \"all\"
        # Health checks that actually work
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5

Resource requests are critical. Mojo binaries can use significant CPU during startup (compilation overhead), but steady-state usage is usually much lower. Set requests based on steady-state, limits based on startup needs.

GPU resource management. If your Mojo code uses GPU, you need proper device plugin setup. The has_gpu() detection can fail in containerized environments with partial GPU visibility.

Startup time is unpredictable. Cold starts can take 10-30 seconds depending on model loading and MLIR compilation. Set your health check delays accordingly.

Cloud Platform Deployment

AWS/GCP/Azure Compute

Mojo binaries run fine on standard cloud compute, but there are gotchas:

Instance types matter for GPU code. The same Mojo binary will run on different GPU architectures, but performance varies wildly. An optimized kernel for V100s might run 3x slower on T4s.

CPU feature detection. Mojo's SIMD optimization uses available CPU features (AVX, AVX-512). Performance can vary significantly between instance families even within the same cloud provider.

Memory bandwidth bottlenecks. Mojo's performance advantages show up most in memory-intensive workloads. Choose instance types with high memory bandwidth for best results.

Serverless Deployment (Lambda/Functions)

This is where things get interesting. Mojo binaries can theoretically run in Lambda, but:

Cold start times suck. 5-15 second cold starts are common when your binary includes large MLIR runtime components. That's fine for batch jobs, terrible for interactive APIs. We tried using Lambda for real-time inference and users thought the service was broken.

Memory usage is unpredictable. The binary might use 200MB at startup for MLIR compilation, then drop to 50MB during execution. Lambda's memory billing doesn't care about your average usage.

Limited GPU support. AWS Lambda doesn't support GPU yet. If your Mojo code uses GPU kernels, it'll fall back to CPU (usually with terrible performance).

Some teams are using Lambda for CPU-only Mojo workloads, but it's not ideal. Better suited for containerized services on ECS/EKS.

Monitoring and Observability

Traditional APM tools don't understand Mojo binaries. Here's what actually works:

Application Metrics

## Build basic instrumentation into your Mojo app
from time import time

fn process_request() -> Response:
    let start_time = time.now()

    let result = your_actual_work()

    let duration = time.now() - start_time
    log_metric(\"request_duration_ms\", duration * 1000)

    return result

Manual instrumentation is your friend. No automatic tracing like you get with Python frameworks. Add timing and counter metrics manually.

Memory usage tracking. Unlike Python, Mojo doesn't have built-in memory profiling. Use system-level monitoring (container memory metrics) to track actual usage.

Error Reporting

When Mojo binaries crash in production, you don't get Python-style stack traces. You get:

Segmentation fault (core dumped)

That's it. No file names, no line numbers, no context. Here's how to survive:

Structured logging everywhere. Log inputs, intermediate states, and outputs. When something crashes, logs are your only debugging info.

Defensive programming. Validate inputs aggressively. Unlike Python, bad inputs can cause silent memory corruption that manifests later as mysterious crashes.

Graceful degradation. Always have a fallback. When your optimized Mojo kernel crashes, fall back to a slower Python implementation rather than bringing down the whole service.

The Production Readiness Checklist

Before you deploy Mojo to production:

✓ Load testing with realistic data. Mojo's performance characteristics are different from Python. Memory usage and CPU patterns change significantly with data size.

✓ Error scenarios tested. What happens when your binary runs out of memory? When GPU kernels fail? When input data is malformed? Test failure modes explicitly.

✓ Rollback strategy. Keep a Python fallback version deployed and ready. When (not if) your Mojo deployment breaks, you need a working alternative immediately.

✓ Monitoring and alerting. Memory usage, crash rates, request latency, GPU utilization if applicable. The binary won't tell you what's wrong, so external monitoring is critical.

✓ Team knowledge. At least one person on your team needs to understand MLIR errors and Mojo debugging. Don't deploy without internal expertise.

The Reality Check

Mojo in production is possible - companies like Inworld and San Francisco Compute are doing it successfully. But it requires more operational maturity than deploying Python services.

The performance wins are real. The debugging pain is also real. Whether it's worth it depends on how much performance you need and how much operational complexity your team can handle.

Most teams should wait another 6-12 months for tooling to mature. If you're going to production now, budget extra time for debugging and have a solid fallback strategy.

The fundamental challenge isn't the language - it's building confidence in your deployment when the debugging tools are still primitive.