Monday, 7 September 2026

Learning Spark from the Latest Docs

Spark is best learned from the various programming guides. 

Access to these can be found from links on its latest documentation page.

These include a Quick Start, an RDD guide, an overview of Datasets and DataFrames and at least two guides on Streaming (1) Structured Streaming, and (2) the legacy functionality of Spark Streaming. 

MLib is also worth understanding, as is GraphX for graph processing, including graph parallel processing.

In terms of interfacing with other languages, Spark and Python is encouraged, whereas Spark and R is deprecated (through PySpark and SparkR respectively).

You should also be familiar with Spark Declarative Pipelines (SDP).

There is an associated knowledge base around deployment - including making Spark play well with Hadoop, Kubernetes and Amazon Elastic Compute (EC2).

No comments: