Airflow workflows often need to interact with external systems such as databases, APIs, or cloud storage. Instead of writing all the connection logic yourself, Airflow provides hooks and sensors.

In this lesson, you’ll learn how hooks and sensors work, how they extend Airflow’s functionality, and how to use them in real-world workflows.

What are Hooks?

  • A hook is an interface that provides reusable connections to external systems.

  • Instead of writing raw Python code for every connection, you can use Airflow’s built-in hooks.

Examples of hooks:

  • MySqlHook: Run queries in MySQL.

  • S3Hook : Interact with Amazon S3 buckets.

  • HttpHook: Make API requests.

  • PostgresHook, GoogleCloudStorageHook, etc.

👉 Hooks simplify authentication, connection handling, and error management.

What are Sensors?

  • A sensor is a special type of operator that waits for a condition to be met before continuing.

  • They are used to check for files, API responses, or external triggers.

Examples of sensors:

  • S3KeySensor: Waits until a file appears in S3.

  • ExternalTaskSensor: Waits for another DAG/task to complete.

  • HttpSensor : Waits for an API endpoint to return a valid response.

  • FileSensor : Waits for a file to appear in a local directory.

Hooks in Action (Example)

from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.providers.mysql.hooks.mysql import MySqlHook
from datetime import datetime

def query_mysql():
    hook = MySqlHook(mysql_conn_id="my_mysql")
    result = hook.get_records("SELECT COUNT(*) FROM users;")
    print(result)

with DAG(
    dag_id="mysql_hook_example",
    start_date=datetime(2025, 1, 1),
    schedule_interval="@daily",
    catchup=False
) as dag:
    run_query = PythonOperator(
        task_id="query_mysql_task",
        python_callable=query_mysql
    )

Here, the hook handles the connection details and authentication with MySQL.

Sensors in Action (Example)

from airflow import DAG
from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor
from datetime import datetime

with DAG(
    dag_id="s3_sensor_example",
    start_date=datetime(2025, 1, 1),
    schedule_interval="@hourly",
    catchup=False
) as dag:
    wait_for_file = S3KeySensor(
        task_id="wait_for_s3_file",
        bucket_name="my-data-bucket",
        bucket_key="incoming/data.csv",
        aws_conn_id="my_aws"
    )

Here, the sensor pauses the DAG until data.csv appears in the S3 bucket.

Lesson Summary

  • Hooks = Reusable interfaces to connect with external systems.

  • Sensors = Special operators that wait for external conditions.

  • Together, they allow Airflow to integrate smoothly with databases, APIs, cloud storage, and other workflows.

PreviousNext