Why Spark Works - Secrets of its Parallel Processing Powers

A Spark progam consists of a driver program that runs the user's main function and runs several parallel operations on a cluster.   It is a parallel processing engine for data. The data structure that enables this is the RDD abstraction.

RDDs are resilient distributed datasets (perhaps a better acronym could be RDDS?) - a collection of elements partitioned across nodes of a cluster that can be operated on in parallel.  

Clearly, this definition speaks to the distributed dataset aspect, but what about the resilient aspect?  Can we say it is implicit in the "can be operated on in parallel" dimension? 

We need to probe what that actually means, to uncover the "secret" of RDDs. Resilient means data remains correct and queryable in the event that nodes go down (which can happen in the physical world).

RDDs are born as files in the Hadoop system (or any Hadoop-supported file system) or an existing Scale collection. RDDs can be persisted in memory for efficient processing and they are resilient against node failures (this bit needs to be understood better - how is this achieved - redundancy of storage??).

(Footnote - once you probe deeper you will start to see ideas percolating from older frameworks like MPI in C++).

Basic Subsystems of Azure Service Fabric

To understand the architecture of Azure Service Fabric (or A18F) we must first be aware of, and understand, the subsystems underneath it.

A18F is a collection of subsystems that enable value-added services to a distributed system.

First we consider the Transport Subsystem. 

Think about what kind of communication you need to support in the Transport layer.

This provides secure point-to-point communication channels within a Service Fabric cluster and between a Service Fabric cluster and its clients (internal and outbound/inbound communication).

It is used internally by Service Fabric and is not directly accessible to application developers for application programming.  Security is provided by X509 certificates or Windows Security.

Then you have the Federation Subsystem. This forms the foundation of a unified cluster - composed of various nodes - and comprises provision of failure detection, leader election and consistent routing - distributed system primitives needed by other subsystems.

The system is build on distributed hash tables with a 128-bit token space.

The Reliability Subsystem is really important. This manages state replication, failover and load balancing; necessities in a highly reliable and available subsystem.

Recall:

1. Failover is when a request is redirected to an alternate server

2. Load balancing is about distributing request processing across multiple servers

The Management Subsystem relates to managing applications. It has services to manage application binaries; deploying, updating and deprovisioning applications and monitoring application health.

The Hosting Subsystem is responsible for managing application lifecycles on a cluster node.

The Communication Subsystem is actually strangely named. It is more of a "service discovery" subsystem. With workloads and infrastructure separated, services may migrate from host to host. The naming service provided by the communication subsystem allows clients to discover and connect to service instances.

The Testability Subsystem is perhaps the most interesting.  It can simulate various failure scenarios to help developers find and address design and implementation deficiencies