No workflow runs perfectly every time. A network request may fail, a database might be down, or a file might not arrive on time. Apache Airflow provides built-in error handling and retry mechanisms to make your pipelines resilient and reliable.

In this lesson, you’ll learn how Airflow handles task failures, how to configure retries, and how to implement best practices for robust pipelines.

Learning Objectives

By the end of this lesson, you will be able to:

  • Understand how Airflow detects and reports task failures.

  • Configure retries and delays when tasks fail.

  • Use callbacks and alerts to notify your team of failures.

  • Apply best practices for handling errors gracefully.

1. How Airflow Handles Failures

  • Each task run can either: success, fail, skip, or be up for retry.

  • If a task fails, Airflow logs the error and stops downstream tasks (unless configured otherwise).

  • Failed tasks are highlighted in red in the Web UI.

2. Configuring Retries

Retries are controlled in the task parameters.

Example:

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime, timedelta

with DAG(
    dag_id="retry_example",
    start_date=datetime(2025, 1, 1),
    schedule_interval="@daily",
    catchup=False
) as dag:

    task = BashOperator(
        task_id="unstable_task",
        bash_command="exit 1",  # fails intentionally
        retries=3,              # retry up to 3 times
        retry_delay=timedelta(minutes=5)  # wait 5 min between retries
    )

 

Here:

  • The task will retry 3 times if it fails.

  • Each retry is delayed by 5 minutes.

  • If all retries fail, the task is marked as failed.

3. Exponential Backoff

Instead of fixed retry delays, you can enable exponential backoff.

 
task = BashOperator(
    task_id="unstable_task",
    bash_command="exit 1",
    retries=5,
    retry_delay=timedelta(minutes=1),
    retry_exponential_backoff=True,
    max_retry_delay=timedelta(minutes=30)
)

👉 Here, retries happen with increasing delays (1 min → 2 min → 4 min … up to 30 min).

4. Error Handling with Callbacks

You can set callbacks to run functions when tasks fail or succeed.

def notify_failure(context):
    print(f"Task {context['task_instance'].task_id} failed.")

task = BashOperator(
    task_id="unstable_task",
    bash_command="exit 1",
    retries=2,
    on_failure_callback=notify_failure
)

Best practice

Send notifications via Slack, Email, or PagerDuty for production workflows.

5. Skipping and Handling Failures Gracefully

  • Use the trigger_rule parameter to control how downstream tasks behave.

  • Example: If one upstream task fails but you still want to continue:

from airflow.operators.dummy import DummyOperator

task1 = BashOperator(task_id="task1", bash_command="exit 1")
task2 = BashOperator(task_id="task2", bash_command="echo 'Task 2'")
final = DummyOperator(task_id="final", trigger_rule="all_done")

task1 >> final
task2 >> final

Here, final runs even if task1 fails because trigger_rule="all_done".

6. Best Practices

  • Always set a reasonable retry limit (avoid infinite loops).

  • Use exponential backoff for unstable systems.

  • Add notifications for failures.

  • Use trigger_rule carefully to prevent skipped tasks from blocking pipelines.

  • Avoid retrying tasks that are not idempotent (e.g., tasks that duplicate data).

PreviousNext