In this segment, we will discuss the Spark application used in the DAG that we constructed.
In the upcoming video, Ajay will demonstrate the Spark applications for filtering our data.
Note:
The code shown in the videos below was used on a different installation of Airflow and so some code lines will be different from our sample solution, however, the overall solution remains the same. Please refer to the code that we have shared on the platform as the final code for this use case.
In the next video, you will gain an understanding of the Spark applications used for generating KPIs.
The file for the Spark applications are attached below.
In the next segment, you will take a look at the dependencies between the tasks that are defined in our DAG.