Tuesday, 8 September 2026

RDD - Regen vs Replication

RDDs are not replicated the way databases typically are replicated. Instead, the transformations converting data between states are memorised.

To make this regeneration efficient, only partitions on lost nodes are recomputed, and assigned to working nodes.

Note: some lineage chains can be very long. In these cases, Spark can checkpoint an RDD to stable storage (HDFS/S3). This prevents "ruanway recomputation".

The Different Definitions of Ground Truth

Ground truth - informally - means information known to be true. It is based on empirical evidence rather than inference.

The origin of the term was in remote sensing literature in 1972 in usage by Nasa to reference data about the earth's surface.

The term has since been hijacked by statistics and machine learning communities with altered semantics.

Monday, 7 September 2026

Why Spark Works - Secrets of its Parallel Processing Powers

A Spark progam consists of a driver program that runs the user's main function and runs several parallel operations on a cluster.   It is a parallel processing engine for data. The data structure that enables this is the RDD abstraction.

RDDs are resilient distributed datasets (perhaps a better acronym could be RDDS?) - a collection of elements partitioned across nodes of a cluster that can be operated on in parallel.  

Clearly, this definition speaks to the distributed dataset aspect, but what about the resilient aspect?  Can we say it is implicit in the "can be operated on in parallel" dimension? 

We need to probe what that actually means, to uncover the "secret" of RDDs. Resilient means data remains correct and queryable in the event that nodes go down (which can happen in the physical world).

RDDs are born as files in the Hadoop system (or any Hadoop-supported file system) or an existing Scale collection. RDDs can be persisted in memory for efficient processing and they are resilient against node failures (this bit needs to be understood better - how is this achieved - redundancy of storage??).

(Footnote - once you probe deeper you will start to see ideas percolating from older frameworks like MPI in C++).



Learning Spark from the Latest Docs

Spark is best learned from the various programming guides. 

Access to these can be found from links on its latest documentation page.

These include a Quick Start, an RDD guide, an overview of Datasets and DataFrames and at least two guides on Streaming (1) Structured Streaming, and (2) the legacy functionality of Spark Streaming

MLib is also worth understanding, as is GraphX for graph processing, including graph parallel processing.

In terms of interfacing with other languages, Spark and Python is encouraged, whereas Spark and R is deprecated (through PySpark and SparkR respectively).

You should also be familiar with Spark Declarative Pipelines (SDP).

There is an associated knowledge base around deployment - including making Spark play well with Hadoop, Kubernetes and Amazon Elastic Compute (EC2).

Saturday, 5 September 2026

The Oh-My-Pi Coding Agent

The oh-my-pi coding agent (aka omp) is getting more attention.  It is a fork of Mario Zechner's Pi.

Lean - Can we Trust the Trust Engine?

The de Bruin principle - it's all in the Kernel

Once upon a time, in the 1960s, a computer scientist called de Bruijn build a system called Automath. His full name as Nicolaas Govert de Bruijn. It incorporated ideas such as dependent types to make it work.

We call him here a computer scientist, but he can equally be called a mathematician, having made contributions to number theory, analysis, combinatorics and logic, and served as Professor of Mathematics at the University of Amsterdam.

One of the things he studied were sequences now known as de Bruijn sequences. Polish programmer Andrzej Trybulec's Mizar system was influenced by Automath.

De Bruijn came up with the idea of creating a kernel separate from the rest of his system.  This kernel is correctness-validation component of the system - the "rule-checker" if you will. It is like the part of a chess computer that validates legal moves - rather than makes smart moves.

Keeping the kernel separate is known as the de Bruijn principle, joining lots of other artefacts attached to his name (such as the de Bruijn-Newman constant, which is closely associated with the Riemann hypothesis).

What underlying logic to use for the Kernel?

"First order" logic, augmented with axioms of set theory, can serve as a good starting point for a theorem prover/correctness checker.   This approach results in a small kernel but lots of coding for complex mathematics. More complex underlying systems are available, including Higher Order Logic (HOL) and Calculus of Inductive Constructions (CIC). Lean uses the latter, or at least features of the latter, to increase expressivity and make programming easier. Features used include proof irrelevance, quotients and nested inductive types.

Taking one example: proof irrelevance is the principle that given a proposition and any two proofs of the same, the two proofs are (considered) equal. This simplifies type theoretic reasoning.

The result is that Lean has a kernel that is much smaller than Lean itself but verifying it is not trivial.

lean4lean and the "multiple kernel" principle

Creator of Lean, L. de Moura, has advocated the "multiple kernel" principle, strengthening the de Bruijn principle by supporting multiple independent kernels.  Lean 3 thus had three independent kernels, one in Lean, one in Haskell and one in Scala (trepplein). Lean 4 invalidated these kernels. Nanoda, an independent kernel in Rust, was thus commissioned together with procedures to write more kernels.

lean4lean was started by Mario Carneiro, an implementation of Lean in Lean which includes an independent kernel.  Efforts are ongoing to verify kernel correctness.

Friday, 4 September 2026

XSS from an OWASP Perspective

Cross site scripting (XSS) is a dangerous type of attack. OWASP analysis can be found here.