Scala vs Java - The Duel

Scala and Java are highly compatible as both are designed to run beautifully on the JVM (the JVM spec should be read/re-read periodically for all serious Java enthusiasts).

Scala uses lambda expressions more aggressively than Java. Both use invokedynamic under the hood.

Scala dominates now in distributed systems (Spark, Akka) particularly for data engineering and ML pipelines, as well as high performance back-ends.

The following is a good website to keep up to speed on Java: Dev.java: The Destination for Java Developers.

Mechanics of RDDs - Python Examples

A Simple Map-Reduce Pattern Example

Here are some examples showing the "mechanics" of how RDDs work in Python.

lines = sc.textfile("data.txt")

We shall first count the lines in the text file (our data file).  The line above creates a base RDD from a text file. The data is not loaded into memory - it's just a pointer to a file at this stage.

We then do a line count of the data.

lineLengths = lines.map( lambda s: len(s))

So here we create the array of lengths.  We then apply reduce to distil this to a single number (classic map-reduce pattern).

totalLength = lineLengths.reduce( lambda a, b: a+b )

Due to lazy evaluation, when we do the map operation, nothing is computed. Only once we do reduce, which is an action, is the computation performed with Spark breaking it down to run on several machines.

If we want to use lineLengths again later, we can do

lineLengths.persist()

before the reduce operation, to save the output into memory after first-computation.

Parallelized Collection Example

Parallelized collections are easily created using a special SparkContext method.

data = [1, 2, 3, 4, 5]

distData = sc.parallelize(data)

Once built, this distributed dataset (distData) can be operated on in parallel.  For example, to add up the members of the list we can do: distdata.reduce( lambda a, b: a+b ).

RDD - Regen vs Replication

RDDs are not replicated the way databases typically are replicated. Instead, the transformations converting data between states are memorised.

To make this regeneration efficient, only partitions on lost nodes are recomputed, and assigned to working nodes.

Note: some lineage chains can be very long. In these cases, Spark can checkpoint an RDD to stable storage (HDFS/S3). This prevents "ruanway recomputation".

The Different Definitions of Ground Truth

Ground truth - informally - means information known to be true. It is based on empirical evidence rather than inference.

The origin of the term was in remote sensing literature in 1972 in usage by Nasa to reference data about the earth's surface.

The term has since been hijacked by statistics and machine learning communities with altered semantics.

Why Spark Works - Secrets of its Parallel Processing Powers

A Spark progam consists of a driver program that runs the user's main function and runs several parallel operations on a cluster.   It is a parallel processing engine for data. The data structure that enables this is the RDD abstraction.

RDDs are resilient distributed datasets (perhaps a better acronym could be RDDS?) - a collection of elements partitioned across nodes of a cluster that can be operated on in parallel.  

Clearly, this definition speaks to the distributed dataset aspect, but what about the resilient aspect?  Can we say it is implicit in the "can be operated on in parallel" dimension? 

We need to probe what that actually means, to uncover the "secret" of RDDs. Resilient means data remains correct and queryable in the event that nodes go down (which can happen in the physical world).

RDDs are born as files in the Hadoop system (or any Hadoop-supported file system) or an existing Scale collection. RDDs can be persisted in memory for efficient processing and they are resilient against node failures (this bit needs to be understood better - how is this achieved - redundancy of storage??).

(Footnote - once you probe deeper you will start to see ideas percolating from older frameworks like MPI in C++).

Learning Spark from the Latest Docs

Spark is best learned from the various programming guides. 

Access to these can be found from links on its latest documentation page.

These include a Quick Start, an RDD guide, an overview of Datasets and DataFrames and at least two guides on Streaming (1) Structured Streaming, and (2) the legacy functionality of Spark Streaming. 

MLib is also worth understanding, as is GraphX for graph processing, including graph parallel processing.

In terms of interfacing with other languages, Spark and Python is encouraged, whereas Spark and R is deprecated (through PySpark and SparkR respectively).

You should also be familiar with Spark Declarative Pipelines (SDP).

There is an associated knowledge base around deployment - including making Spark play well with Hadoop, Kubernetes and Amazon Elastic Compute (EC2).

The Oh-My-Pi Coding Agent

The oh-my-pi coding agent (aka omp) is getting more attention.  It is a fork of Mario Zechner's Pi.