Now that you have learnt about operators in Airflow, in this segment, you will learn about another important Airflow concept: Sensors.
In the upcoming video, Ajay will discuss sensors in detail.
So, in the video, you learnt about sensors in Airflow.
Sensors are useful for checking for a certain condition before a task is run. They wait for a certain action to be completed and succeed if the action is completed within a specific timeout duration. If the action is not completed within the specified time, then they fail.
The different arguments for sensors are listed below:
- soft_fail: If True, it marks the task as SKIPPED on failure.
- poke_interval: Interval between each try (>1 minute recommended).
- timeout: Time before a sensor fails.
Like operators, there are several different types of sensors in Airflow. The ones discussed in the video are listed below:
- WebHdfsSensor: Waits for a file or a folder to land in HDFS.
- ExternalTaskSensor: Waits for a different DAG or a task in a different DAG to complete for a specific execution_date.
- TimeDeltaSensor: Waits for a timedelta after a task’s execution_date +schedule_interval.
- TimeSensor: Waits until the specified time of the day.
Additional Reading
You can visit the following link to find out more about the different sensors available: Sensors.