In this segment, let’s summarise your learnings from this module.
Now that you have come to the end of this module, let’s take a quick look at what you learnt.
You first gained a basic understanding of what streaming is and why it is important. Then, you looked at a few examples of streaming data and also learnt about the difference between micro-batching and streaming. At the end of the first session, you got an introduction to Spark Streaming and its APIs.
In the next session, you learnt about the importance of Structured Streaming API and its advantages over the DStream API. Then, you looked at the differences between Static and Streaming DataFrames with transformations along with triggers and various output modes. The session concluded with joins on streams and a few coding examples.
The next session gave you insights into event and processing times. Then, you learnt about windows and late-arriving data and understood how late-arriving data can be handled with watermarks.
In the final session, you looked at a few industry use cases, wherein you learnt how to integrate Kafka with your streaming application and how to read and process tweets in real-time.
This brings us to the end of the module. I hope you will now be confident about your skills and apply them to the best of your efforts.
Best of luck!
Additional Readings:
- Spark Docs – Structured Streaming Programming Guide – Spark 2.2.0 Documentation
- Spark Streaming API Cookbook – PySpark.SQL module
- Kafka Integration Guide – Structured Streaming + Kafka Integration Guide (Kafka broker version 0.10.0 or higher) – Spark 2.2.0 Documentation
Report an error