Welcome to the session on ‘Getting Started with Structured Streaming’.
In the previous session, you learnt what streaming is, in addition to learning how Spark Streaming deals with streaming data in micro-batches, along with the architecture of Spark Streaming API.
In the upcoming video, our SME, Kautuk, will provide an overview of the topics that will be covered in this session.
- To begin with, you will learn what structured streaming is, its APIs and its advantages, followed by a coding walkthrough where we will build a simple Spark Streaming that will consume streaming data.
- Next, you will learn about triggers and various output modes, following which we will work with structured streams, and through a coding lab, you will learn how to read from files into a stream, how to operate with triggers and the various output modes.
- The next segment will cover transformations such as Filter, As and GroupBy, in addition to aggregations functions such as Avg and Sum.
- We will end the session by learning about joins with streams, exploring the various types of joins and analysing their significance, followed by a coding lab on the same.
Since this session will have coding segments, make sure you practise writing the codes yourselves and understand it thoroughly.
So, let’s get started!
People you will hear from in this session
Subject Matter Expert
Senior Data Engineer
Kautuk is currently working as a senior data engineer. He has over 9 years of experience in the IT industry and has worked for several companies. He has deep knowledge of the various tools and technologies that are in use today.