Describing an Azure Service Fabric Service using ServiceManifest.xml

The ServiceManifest.xml declares information relevant to your microservice. 

Whether your service is stateless (no persistent storage, or state stored outside of the service, for example, through Azure Storage, Azure SQL Database, Azure Cosmos DB) or stateful, where Service Fabric manages your service state via Reliable Collections or Reliable Actors programming models.

Stateful applications are particularly interesting from an availability standpoint. The Reliable Collections API builds on System.Collections, and in particular, System.Collections.Concurrent, to create highly available (stateful) data structures while keeping code complexity to a minimum.

Reliable Collections ensure underlying data is replicated for high availability, operate asynchronously to ensure no blocking IO operations and transactional to ensure strong consistency (ensuring transaction commits finish only once changes are replicated on the majority quorum of replicas including the primary). They can also be persisted or volatile, the former where data is persisted to disk to protect against large scale outages.

Relevant data structures can be found in Microsoft.ServiceFabric.Data.Collections and in the assembly Microsoft.ServiceFabric.Data.Interfaces.dll.

Stateful Services in Service Fabric

Managing state reliably in a distributed system is a challenge. Service Fabric rises to the challenge. The key hazards of a distributed system:

1. Messages can get lost

2. Failure of a node

3. Communication channel stalls

So when a stateful service needs to persist state, it must use a Reliable Collection! SF provides reliable Dictionaries and reliable Queues to serve this end.

Reliable data structures are distributed data structures but can be used as if they are local data structures.

Reliable collections are persisted by a Reliable State Manager component (which is represented by a class in the Microsoft.ServiceFabric.Data namespace which lives in the Microsoft.ServiceFabric.Data.dll in the Azure SDK).

The states are replicated by a Transactional Replicator to secondary replicas for availability and reliability.

Most stateful services inherit from a StatefulService class.

Comparison Corner: Azure Service Fabric versus Kubernetes

 Kubernetes - is an orchestration solution

Basic Subsystems of Azure Service Fabric

To understand the architecture of Azure Service Fabric (or A18F) we must first be aware of, and understand, the subsystems underneath it.

A18F is a collection of subsystems that enable value-added services to a distributed system.

First we consider the Transport Subsystem. 

Think about what kind of communication you need to support in the Transport layer.

This provides secure point-to-point communication channels within a Service Fabric cluster and between a Service Fabric cluster and its clients (internal and outbound/inbound communication).

It is used internally by Service Fabric and is not directly accessible to application developers for application programming.  Security is provided by X509 certificates or Windows Security.

Then you have the Federation Subsystem. This forms the foundation of a unified cluster - composed of various nodes - and comprises provision of failure detection, leader election and consistent routing - distributed system primitives needed by other subsystems.

The system is build on distributed hash tables with a 128-bit token space.

The Reliability Subsystem is really important. This manages state replication, failover and load balancing; necessities in a highly reliable and available subsystem.

Recall:

1. Failover is when a request is redirected to an alternate server

2. Load balancing is about distributing request processing across multiple servers

The Management Subsystem relates to managing applications. It has services to manage application binaries; deploying, updating and deprovisioning applications and monitoring application health.

The Hosting Subsystem is responsible for managing application lifecycles on a cluster node.

The Communication Subsystem is actually strangely named. It is more of a "service discovery" subsystem. With workloads and infrastructure separated, services may migrate from host to host. The naming service provided by the communication subsystem allows clients to discover and connect to service instances.

The Testability Subsystem is perhaps the most interesting.  It can simulate various failure scenarios to help developers find and address design and implementation deficiencies

Azure Service Fabric Basics

Azure Service Fabric is Microsoft's Platform-as-a-Service (PaaS) offering for developers looking to host scalable and highly available distributed systems.

Microsoft has used it for years to support: 

Skype for Business | Cortana | Intune (a cloud-based MDM -> Mobile Device Management | Azure Cosmos DB amongst other things.

ASF was open-sourced in March 2018.