Introduction to Apache Airflow

Airflow organizes workflows as Directed Acyclic Graphs (DAGs), where each node represents a task and the edges define dependencies between them. This ensures each task runs in the correct order — for instance, a model training step can only start after data preprocessing completes.

Airflow's scheduler automatically executes these tasks according to a defined schedule, ensuring consistency and reproducibility. Engineers can easily rerun failed tasks, monitor progress through the Airflow UI, and scale workflows as projects grow.

Basic DAG Example

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator

def print_hello():
    print("Hello from Airflow DAG!")

default_args = {
    "owner": "mlops_engineer",
    "retries": 1,
    "retry_delay": timedelta(minutes=5),
}

dag = DAG(
    "hello_airflow_example",
    default_args=default_args,
    description="A simple DAG example",
    schedule_interval=timedelta(days=1),
    start_date=datetime(2024, 6, 1),
    catchup=False,
)

hello_task = PythonOperator(
    task_id="say_hello",
    python_callable=print_hello,
    dag=dag,
)