In this segment, you will understand the concepts of Kafka Connect in further detail. Let’s watch the next video to learn about Connectors.
In the video above, you learnt what connectors are. Connectors are responsible for moving the data from one place to another. Connectors are of two types:Source Connectors and Sink Connectors. Source Connectors take data from an external system and write it to a Kafka topic. Sink Connectors are responsible for moving data from Kafka topics to external systems. There are many different open source connectors that are available for use.
Now that you have learnt about Connectors, let’s watch the next video to understand the concept of Tasks in kafka Connect.
In the video, you learnt about Tasks. Each connector instance coordinates a set of tasks that copy data. You can divide a single job into multiple tasks to increase parallelism. Tasks get rebalanced across different workers whenever a worker stops working.
In the next video, you will understand the concept of Workers.
Workers refer to the running processes that are responsible for executing connectors and tasks. A connected cluster is a group of workers. Depending on the number of workers in a cluster, the workers can be divided into two modes: Standalone and Distributed.
In the standalone mode, a single process is responsible for executing all the connectors and tasks. This mode is useful when you want to test Kafka Connect on local machines. As only one process is responsible for everything, fault tolerance is not guaranteed.
In the distributed mode, many worker processes start and automatically coordinate to execute the connectors and tasks. Here, fault tolerance is achieved because more than one worker is present. Tasks get rebalanced when a new worker joins or shuts down. This is similar to the case of consumer groups.
The following image describes what a Kafka Connect cluster looks like.
A Kafka Connect cluster is nothing but a group of workers responsible for executing connectors and tasks. There are multiple workers present in the cluster and execute the connectors and tasks.
Additional Reading
Kafka Connect – Read about the different concepts about Kafka Connect.
Report an error