What Scale is Your Big Data

terabytes (TB) - 1000 gigabytes (10^12 bytes) - used in consumer storage

petabytes (PB) - 10^15 bytes  or 1000 TB

exabytes (EB) - 10^18 bytes or 1000 PB

Zettabytes and yottabytes are even bigger measures of memory.

Marker Icon in Microsoft Word

The "marker icon" on the Font group in the Ribbon is what determines background (highlight) colour. It can be hard to identify when the background colour is set to grey and thus blends into the ribbon.

Think of Programming as a Martial Art

Something that requires continuous practice and continuous learning.

Databricks Apache Spark-Oriented Origins

It is enlightening to reflect that Databricks grew out of the AMPLab project (AMPLab was an acronym for Algorithms, Machines and People Lab) at UC Berkeley which worked on a variety of big data projects.

AMPLab invented Apache Spark, Apache Mesos (for compute cluster management, retired in August 2025, having lost mindshare to Kubernetes which was backed by Google and CNCF) and Alluxio. Of these projects, Spark is currently the most impactful product to come out of the AMPLab project.

It was founded in 2013 and offers a cloud-based platform for data analytics and AI. It operates in Azure, AWS and GCP (the "big three" providers).

Databricks the data "lakehouse" architecture, combining aspects of data lakes and data warehouses for managing structured and unstructured data. The company develops the Delta Lake open source project adding ACID transactions to data lakes. A paper describing the challenge of adding ACID transactional capability to data lakes has been published (Armbrust et al).

To recap: ACID is an acronym which stands for atomicity, consistency, isolation and durability. These properties are used to characterize reliable database transactions and was first expounded in an influential 1983 paper "Principles of Transaction-Oriented Database Recovery" (Theo Harder and Andreas Reuter).

Literacy is a Superpower in Software Engineering

Literacy is literally a superpower in software engineering. It is the reason so many good software engineers are avid readers and definitely not just of technical tomes, but enjoy all sorts of fiction (though science fiction is a particularly popular genre for obvious reasons).

Structured Streaming in Spark

Structured Streaming is the stream "processing engine" in Spark, designed for scalability and fault tolerance.  

It is build on the Spark SQL engine. Pause. This has implications to the programming model.

The "weird element" in Structured Streaming (following on from our comment on SQL engine above) is that streaming computations are expressed like queries on SQL tables (check).

The DataSet/DataFrame API is used.

A crazy case study is presented below.

Suppose you want to maintain a running word count of text data from a data server listening on a TCP socket.  This can be expressed in structured streaming!

Ironically, the most concise representation for this is in R which is least best supported language in Spark.

This is what the Python looks like.

import [STUFF]  ---> from pyspark.sql import SparkSession

spark = SparkSession(  ... .appName("WordCount") ..)

Now we create a streaming DataFrame representing text data received from the server listening on localhost:9999 and transform the DataFrame to word counts.

A quick word on port 9999. Why use this?  It is easy to remember and avoids conflicts with standard ports. Java apps sometimes use it for debugging.

(To check if it is being used: netstat -ano | find "9999").

Scala vs Java - The Duel (Both JVM languages)

Scala and Java are highly compatible as both are designed to run beautifully on the JVM (the JVM spec should be read/re-read periodically for all serious Java enthusiasts).

Scala uses lambda expressions more aggressively than Java. Both use invokedynamic under the hood.

Scala dominates now in distributed systems (Spark, Akka) particularly for data engineering and ML pipelines, as well as high performance back-ends.

The following is a good website to keep up to speed on Java: Dev.java: The Destination for Java Developers.