In this segment, we will take a look at the components that make up the Airflow Architecture and discuss the single-node and distributed architecture of Airflow.
In the following video, Amit will discuss the components of Airflow as well as the single-node architecture in detail.
In the previous video, we learnt about the different components that make up Airflow’s architecture as seen in the following diagram:
We also saw the single-node architecture as shown in the following graphic:
In the next video, we will learn about executors in more detail and also look at the multi-node architecture of Airflow.
Sequential Executor:
- It is the default executor and will only run one task instance at a time.
- It is the only executor that can be used with SQLite which is the default database in Airflow.
- It is suitable for testing and debugging DAGs before they are implemented in an industrial environment.
Local Executor:
- It is like a sequential executor with unlimited parallelism.
- It runs tasks by spawning processes in a controlled fashion in different modes.
Celery Executor:
- It is used in scalable environments.
- It needs RabbitMQ and Redis for configuration.
- Each worker is in a different node. So, the number of workers can be easily scaled by adding more nodes.
- It is recommended in a production scenario.
These are the important Airflow executors that you should know. There are many more types used for different use cases of Airflow.
Next, we discussed the multi-node/distributed architecture used in the industry and is the reason behind Airflow’s highly scalable and reliable nature.
Additional Reading
Executors – Refer to link if you want to know more about the types of Airflow executors.
Report an error