Now that you have learnt how a simple Spark Streaming application works, in this segment, you will learn about triggers and output modes. The trigger settings of a streaming query define the timing of streaming data processing, specifically, whether the query will be executed as a micro-batch query with a fixed batch interval or as a continuous processing query. The output modes of a streaming query define what DataFrames are to be written to the output sink.
So, let’s watch the upcoming video and listen to our expert as he talks about triggers and output modes.
So, in the video, you learnt about various output modes, which include the following:
- Append– Only new records are added to the sink.
- Update– Only modified records are added to the sink.
- Complete– All records are added to the sink.
There are, however, a few restrictions on output modes, for example, no aggregations with the update or append mode without watermarks.
Triggers are a means to decide when new data gets processed in a stream. The default setting would be to do it every time a new micro-batch comes up. You could also do it continuously at each record level or just once for a single micro-batch.
Now, let’s move on to the next segment and learn how to implement output modes and triggers.
Additional Reading
Triggers and Output Modes – The two articles will provide further insights into the topics learnt in this segment.