Structured Streaming in Spark

Structured Streaming is the stream "processing engine" in Spark, designed for scalability and fault tolerance.  

It is build on the Spark SQL engine. Pause. This has implications to the programming model.

The "weird element" in Structured Streaming (following on from our comment on SQL engine above) is that streaming computations are expressed like queries on SQL tables (check).

The DataSet/DataFrame API is used.

A crazy case study is presented below.

Suppose you want to maintain a running word count of text data from a data server listening on a TCP socket.  This can be expressed in structured streaming!

Ironically, the most concise representation for this is in R which is least best supported language in Spark.

This is what the Python looks like.

import [STUFF]  ---> from pyspark.sql import SparkSession

spark = SparkSession(  ... .appName("WordCount") ..)

Now we create a streaming DataFrame representing text data received from the server listening on localhost:9999 and transform the DataFrame to word counts.

A quick word on port 9999. Why use this?  It is easy to remember and avoids conflicts with standard ports. Java apps sometimes use it for debugging.

(To check if it is being used: netstat -ano | find "9999").