Apache Airflow - Python Workflow Orchestrator That Doesn't Completely Suck

Airflow is a Python-based workflow orchestrator that lets you define data pipelines as code. It was born out of necessity at Airbnb in 2015 when their data team got tired of managing hundreds of brittle cron jobs that would fail silently and leave everyone wondering why reports were broken.

Airflow Web Interface: The UI shows your DAGs as interactive graphs where you can see which tasks are running, failed, or completed. It's actually pretty useful once you figure out where everything is.

How This Thing Actually Works

Here's what you're dealing with when you deploy Airflow:

Scheduler: The thing that actually runs your workflows. It's supposed to be reliable, but will randomly stop working when you hit around 300 DAGs unless you tune it properly. We learned this during a Black Friday deployment. Scale early.

Web Server: The UI where you'll spend most of your time trying to figure out why tasks failed. It's actually pretty decent - you can see logs, trigger reruns, and pretend you understand why the scheduler decided to skip your task.

Executor: Defines where tasks run. LocalExecutor works for single machines, CeleryExecutor works great until Redis crashes at 2am, and KubernetesExecutor is what you use when you want to make your infrastructure team hate you.

Metadata Database: PostgreSQL is your best bet. MySQL works but has weird edge cases. SQLite is fine for testing until you have more than one user.

Workers: The things that actually run your code. They'll work fine until you run out of memory, then they'll die silently and your tasks will hang forever.

DAGs (Directed Acyclic Graphs)

Workflow Structure: DAGs represent your data pipeline as a graph of connected tasks, each with dependencies and retry logic.

This is Airflow's fancy name for "workflow." You write Python code that defines:

• What tasks to run (PythonOperator, BashOperator, etc.)
• When to run them (scheduling and dependencies)
• What to do when they fail (retry 3 times then wake up the on-call engineer)
• How long to wait before giving up (task timeouts)

Dependencies look like this: task_a >> task_b >> task_c. Simple enough that even your product manager can understand it.

DAG Visualization: Tasks show up as colored boxes (green = success, red = failed, yellow = running). You can click on any task to see logs, retry failed tasks, or mark them as successful if you're feeling dangerous.

Production Reality Check

High Availability: You can run multiple schedulers, but the first one will do most of the work anyway. The metadata database is still a single point of failure.

Security: Has RBAC, OAuth, SSL, and encrypted connections. Basically enterprise-ready if you spend the time configuring it properly.

Monitoring: Built-in Prometheus metrics and email alerts. You'll still need external monitoring because the scheduler can fail without any alerts.

Scaling: Works until it doesn't. The scheduler becomes a bottleneck around 1000+ DAGs. You'll need to tune scheduler performance or split into multiple Airflow instances.

Version Reality (September 2025)

Apache Airflow 3.0.6 is the current version as of September 2025. Airflow 3.0 was a major release in early 2025 with breaking changes. Don't use anything before 2.7 - too many security issues.

Major 3.0 changes include:

• New execution model with better task isolation
• Improved scheduler performance (finally)
• Modern web UI that doesn't look like it was designed in 2010
• Better Kubernetes integration for cloud-native deployments

Netflix runs 100k+ workflows daily on Airflow, but they also have an army of engineers keeping it running. Adobe manages thousands of pipelines, and both companies contribute heavily to making it less terrible for the rest of us.

Now that you know what you're getting into, let's see how Airflow stacks up against the competition - because there are definitely easier alternatives if your use case doesn't demand Airflow's complexity.

Airflow installation will fight you. Here's how to win without questioning your career choices.

System Requirements (The Stuff That Will Actually Break)

Airflow runs on POSIX systems (Linux, macOS). Windows works via WSL2, but you'll spend more time debugging networking issues than actually building pipelines.

Python: Supports 3.9, 3.10, 3.11, 3.12, and 3.13 (added in July 2025). Python 3.8 was dropped in Airflow 2.7.0 because it hit end-of-life. Python 3.13 is fully supported as of Airflow 3.0, but check your provider packages for compatibility if you use many third-party operators.

Memory: Start with 8GB minimum. The scheduler will eat 2-4GB by itself, and each worker needs 1-2GB depending on your DAGs. We've seen the scheduler OOM on 4GB instances when parsing 200+ DAG files.

Database: Use PostgreSQL 12+ in production. Period. MySQL 8.0 technically works but has encoding issues with certain task metadata. SQLite is fine for testing until you try to run more than one component.

MariaDB is not supported and will break in weird ways. Just don't.

Installation Methods (Pick Your Poison)

pip Installation (Works until it doesn't):

## This will conflict with your existing packages
pip install apache-airflow[postgres,celery,redis]

Pro tip: Use a virtual environment or pipx to avoid dependency hell. The [postgres,celery,redis] extras are what you actually need - don't install the base package alone.

Docker Setup: The easiest way to get Airflow running locally without dealing with dependency hell.

Docker Deployment (15 minutes if lucky, 2 hours if not):

curl -LfO 'https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml'
docker-compose up

This downloads a 2GB compose file that starts 7 containers. Port 8080 will be taken by something else on your machine guaranteed. Check the Docker setup guide when it breaks.

Kubernetes Deployment (For when you hate your weekend):
The official Helm chart is production-ready if you enjoy spending 3 days configuring YAML files. Your infra team will thank you (after they stop cursing your name).

The Setup Ritual (What Actually Breaks)

Copy these commands and pray:

## Initialize database (will fail if DB isn't running)
airflow db init

## Create admin user (will prompt for password you'll forget)
airflow users create --username admin --firstname Admin --lastname User --role Admin --email admin@example.com

## Start components
airflow webserver -p 8080 &
airflow scheduler &

Pro tip: The DAGs folder defaults to ~/airflow/dags. If you put files anywhere else, they won't show up and you'll waste an hour wondering why.

Common gotchas:

• The scheduler needs read permissions on the DAGs folder and will fail silently if it doesn't have them
• Task logs go to ~/airflow/logs by default - make sure there's disk space
• Connections for databases/APIs go through the web UI under Admin > Connections

Your First DAG (That Actually Works)

Here's a DAG that won't immediately break:

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.bash import BashOperator

## This prevents backfill hell
default_args = {
    'owner': 'your_name_here',
    'depends_on_past': False,
    'retries': 2,
    'retry_delay': timedelta(minutes=5)
}

## catchup=False prevents Airflow from running every day since start_date
with DAG(
    'hello_world_that_works',
    default_args=default_args,
    description='A simple tutorial DAG',
    schedule='@daily',
    start_date=datetime(2025, 9, 1),
    catchup=False,
    tags=['example']
) as dag:
    
    hello_task = BashOperator(
        task_id='say_hello',
        bash_command='echo "Hello World from task $(date)"',
    )

Save this as hello_world.py in your DAGs folder. It'll show up in the UI after 30-60 seconds (the scheduler parses DAG files every 30 seconds by default).

DAGs View: Your DAGs appear in the web UI with status indicators. Green = running fine, red = something's broken and you need to fix it before someone notices.

Things That Will Definitely Go Wrong

"My DAG isn't showing up": Check the scheduler logs. It's probably a Python syntax error, import error, or the file isn't in the DAGs folder.

"Tasks are stuck in queued state": The executor died. Restart everything with docker system prune -a && docker-compose up if using Docker.

"Scheduler randomly stops working": This happens around 300+ DAGs. You'll need to tune the scheduler settings or add more memory.

"Web UI shows 500 errors": Database connection is probably broken. Check your database is running and connection string is correct.

Nuclear option: Delete everything and start over: rm -rf ~/airflow && docker system prune -a

Start with the official tutorial once you get the basic setup working. It covers TaskFlow API, dependencies, and the stuff you'll actually use in production.

Even with this guide, you're going to run into weird issues. Below are the questions every Airflow developer asks - usually at 3am when something's broken and deadline pressure is mounting.