In this segment, you will learn about SubDAGs and try to implement the DAG that we created in this 4th session using SubDAGs.
In the upcoming video, Ajay will introduce you to the concept of SubDAGs.
So, in the video, you learnt about subDAGs in Airflow.
- They are widely used to group logically similar or parallel tasks in a DAG.
- A SubDAG is defined as a function that returns a DAG object.
- In terms of a graph, each SubDAG behaves like a vertex (a single node) in a graph.
Please refer to the diagram given below to understand this concept better.
The following diagram explains how you can define a simple subDAG:
In the next video, you will learn about the implementation of the DAG that we created in the previous session using subDAGs.
The code used in the demonstration is provided below:
Note:
The steps to be followed for this demonstration are exactly the same as the one for the real-world use case
In the next segment, you will understand another advanced concept in Airflow known as backfilling.
Additional Reading
Report an error