In this segment, you will take a look at the task dependencies between the tasks that are defined in our DAG.
For your reference, the DAG with the task dependencies has been given below. In the upcoming video, Ajay will explain these task dependencies to you.
Note:
Please note that the Airflow UI in the video below is slightly different from the UI that you will see however, the overall steps and configuration remain the same.
Note:
The below code was used on a different installation of Airflow and so some code lines will be different from our sample solution, however, the overall solution remains the same. Please refer to the code that we have shared on the platform as the final code for this use case.
Note:
There is an additional task in the code for the ETL DAG used with EMR called switch_java_version which is a basic Bash operator which will run the java EMR command to switch the java JDK version to JDK 8 as you have also manually used in the previous session. You can refer to the ETL DAG python file below for this.
In the previous video, we understood the dependencies between the tasks in our DAG.
In the next segment, we will run our DAG and take a look at our data pipeline in action.