RDDs are not replicated the way databases typically are replicated. Instead, the transformations converting data between states are memorised.
To make this regeneration efficient, only partitions on lost nodes are recomputed, and assigned to working nodes.
Note: some lineage chains can be very long. In these cases, Spark can checkpoint an RDD to stable storage (HDFS/S3). This prevents "ruanway recomputation".
No comments:
Post a Comment