In this segment, you will learn about SubDAGs and try to implement the DAG that we created in this 4th session using SubDAGs.
In the upcoming video, Ajay will introduce you to the concept of SubDAGs. So, in the video, you learnt about SubDAGs in Airflow.
- They are widely used to group logically similar or parallel tasks in a DAG.
- A SubDAG is defined as a function that returns a DAG object.
- In terms of a graph, each SubDAG behaves like a vertex (a single node in a graph.
Please refer to the diagram given below to understand this concept better. The following diagram explains how you can define a simple SubDAG:
In the next video, Ajay will be showing you an implementation of the DAG that we created in the previous using SubDAGs.
The code used in the demonstration is provided below:
Note:
The steps to be followed for this demonstration are exactly the same as the one for the real-world use case.
In the next segment, you will understand another advanced concept in Airflow known as backfilling.
Note:
SubDAGs have limited parallelism and can be executed only by the Sequential Executor, regardless of which Executor is employed for all other tasks, which made for a challenging and unreliable user experience. As of Apache Airflow 2.0, features such as Task Groups, TaskFlow and CrossDAGs have been added to improve the limitations of SubDAGs. You may refer to the following links for more information:
- https://www.astronomer.io/blog/introducing-airflow-2-0/
- https://airflow.apache.org/blog/airflow-two-point-oh-is-here/
Additional Reading
Using SubDAGs in Airflow
Report an error