Mechanics of RDDs - Python Examples

Here are some examples showing the "mechanics" of how RDDs work in Python.

lines = sc.textfile("data.txt")

We shall first count the lines in the text file (our data file).  The line above creates a base RDD from a text file. The data is not loaded into memory - it's just a pointer to a file at this stage.

We then do a line count of the data.

lineLengths = lines.map( lambda s: len(s))

So here we create the array of lengths.  We then apply reduce to distil this to a single number (classic map-reduce pattern).

totalLength = lineLengths.reduce( lambda a, b: a+b )

Due to lazy evaluation, when we do the map operation, nothing is computed. Only once we do reduce, which is an action, is the computation performed with Spark breaking it down to run on several machines.

If we want to use lineLengths again later, we can do

lineLengths.persist()

before the reduce operation, to save the output into memory after first-computation.