In Airflow, tasks usually run independently. But sometimes, one task needs to pass data to another. This is where XComs (Cross-Communication) come in.

In this lesson, you’ll learn what XComs are, how they work, and how to use them effectively to share data between tasks in your DAGs.

What are XComs?

  • XCom stands for Cross-Communication.

  • It’s a mechanism that lets tasks share small pieces of data.

  • XComs are stored in Airflow’s metadata database.

Example:

  • Task A extracts today’s file name → pushes it to XCom.

  • Task B reads that file name → pulls it from XCom.

How XComs Work

  • Push : A task can push a value into XComs.

  • Pull : Another task can pull that value later.

XComs are identified by:

  • key: The name of the stored value.

  • task_id: The task that pushed the value.

  • dag_id: The DAG to which the task belongs.

Using XComs – Example

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

# Task A: Push a value into XCom
def push_value(**context):
    context['ti'].xcom_push(key='file_name', value='data_2025.csv')

# Task B: Pull the value from XCom
def pull_value(**context):
    file_name = context['ti'].xcom_pull(key='file_name', task_ids='push_task')
    print(f"Processing file: {file_name}")

with DAG(
    dag_id="xcom_example",
    start_date=datetime(2025, 1, 1),
    schedule_interval="@daily",
    catchup=False
) as dag:

    push_task = PythonOperator(
        task_id="push_task",
        python_callable=push_value,
        provide_context=True
    )

    pull_task = PythonOperator(
        task_id="pull_task",
        python_callable=pull_value,
        provide_context=True
    )

    push_task >> pull_task

Here:

  • push_task saves "data_2025.csv" in XCom.

  • pull_task retrieves it and uses it in processing.

Auto XComs

  • Some operators automatically push results to XCom.
    Example: PythonOperator returns value and  stored it in XCom.

Example:

def return_value():
    return "Hello XCom!"

task = PythonOperator(
    task_id="auto_push",
    python_callable=return_value
)

Here, "Hello XCom!" is automatically available in XCom.

Best Practices

  • Use XComs for small data only (e.g., file names, IDs, messages).

  • Don’t use XComs for large datasets (instead, store in a database or file system).

  • Use meaningful keys for clarity.

  • Monitor XComs via the Airflow Web UI → Admin → XComs.

Lesson Summary

  • XComs allow tasks to share small pieces of data.

  • Tasks can push values into XComs and pull them later.

  • Some operators automatically push results.

  • Use XComs carefully: great for metadata, not for big data.

PreviousNext