In this lesson, you’ll learn how Apache Airflow schedules workflows automatically, how to configure schedules with CRON and presets, and the different ways to trigger DAGs manually or programmatically.

The Role of the Scheduler

  • Airflow includes a scheduler service that checks DAGs and triggers tasks at the right time.

  • It ensures tasks run according to the schedule defined in the DAG.

Example:

  • Run every day at midnight.

  • Run every Monday at 8 AM.

  • Run only once on a specific date.

Scheduling with schedule

The schedule parameter in a DAG defines when it runs.

Common options:

  • None : Don’t schedule.

  • @once : Run only once.

  • @continuous : Run as soon as the previous run finishes.

  • @hourly : Run once per hour.

  • @daily : Run every day at midnight.

  • @weekly : Run once per week.

  • @monthly : Run once per month.

from airflow import DAG
from datetime import datetime

with DAG(
    dag_id="daily_etl_pipeline",
    start_date=datetime(2025, 1, 1),
    schedule_interval="@daily",
    catchup=False
) as dag:
    ...

Using CRON Expressions

For more precise schedules, use CRON syntax.

Examples:

  • "0 9 * * *": Every day at 9 AM.

  • "0 0 * * 0" : Every Sunday at midnight.

  • "*/15 * * * *": Every 15 minutes.

💡 To build or verify a CRON expression, you can use an online editor like Crontab Guru.

Triggering Workflows

Besides scheduling, DAGs can also be triggered manually:

  1. Airflow Web UI:  Click the “Trigger DAG” button.

  2. Command Line: airflow dags trigger dag_id

  3. API:  Trigger programmatically using REST API calls.

This is useful for ad-hoc runs or testing.

Catchup and Backfill

1. Catchup

In some cases, you may need to re-execute your DAG. One common scenario is when a scheduled DAG run fails.

  • catchup=True: Runs all missed DAG runs between start_date and today.

  • catchup=False: Runs only the latest DAG run from now onward.

Example:

If a daily DAG starts on Jan 1 and today is Jan 10:

  • With catchup=True, it runs Jan 1–9.

  • With catchup=False, it only runs Jan 10.

2. Backfill

You may need to run a DAG for a specific historical period. For example, a DAG may be created with a start_date of 2025-11-01, but another user might require output data starting from 2025-10-01. This process is known as backfilling. It can be performed using either the UI or the CLI.

Lesson Summary

  • Airflow schedules tasks using schedule.

  • You can use presets (@daily, @weekly) or CRON expressions for flexibility.

  • DAGs can also be triggered manually via UI, CLI, or API.

  • catchup controls whether past runs are executed.

PreviousNext