In this segment, you will learn about backfilling in Airflow.
Let’s watch the upcoming video in which Ajay will introduce the same.
So, in this video, we discussed the concept of backfilling. Let’s summarise it as follows:
- Backfilling is used to run the DAGs for older schedules.
- It is useful to reprocess the data if the application code has been updated or the data in the source has been corrected/refreshed.
- Backfilling using CLI can be done using the following command:
- airflow backfill -s START_DATE -e END_DATE dag_id
- Airflow supports automatic backfilling of data according to the start time, the current time and the schedule. To disable it, set catchup=False in the DAG object.
In the next segment, you will understand another advanced concept in airflow known as airflow variables.
Additional Reading
Report an error