Before we our discussion on how to automate large scale data pipelines using Airflow. we need a better understanding of what data pipelines are and what a typical data pipeline looks like.
In the next video, our SME Amit will discuss these concepts in detail.
In the previous video, you learnt about data pipelines and their similarity to Process Flow-Charts.
We discussed how databases efficient for Online Transactional Processing (OLTP) are not best suited for Online Analytical Processing (OLAP), and the first step in a scalable solution for data processing is to move the data from OLTP databases to OLAP systems.
This is where data pipelines come to our rescue.
We make use of data pipelines to move one system to the other and possibly transform and validate it in the process.
Finally, we discussed the phases involved in a typical data pipeline as you will see in the following graphic:
Additional Reading
OLTP vs OLAP: A article showing a comparison between OLTP and OLAP.
Report an error