In this segment, we will discuss monitoring and alerting with regards to our DAGs in Airflow.
Let’s watch the upcoming video in which Ajay will introduce this concept.
In the video, you learnt about the different ways in which you can use Airflow to monitor our workflows.
These are listed below.
- Using the Airflow UI, you can monitor the task statuses (running, success, failed, skipped, etc.).
- You can monitor the time taken to execute dependency tasks.
- You can monitor any delays in DAG execution with SLA duration.
- Health checks of workers and schedulers can be configured.
- The StatsD package helps in collecting Airflow statistics.
You also learnt how alerting and notifications can be handled using Airflow, which is explained below.
- Email operators can be used to send automated emails whenever a task fails or is completed at the DAG level.
- Airflow can also send notifications to popular online services such as Slack and HipChat.
Report an error