Consistency & Consistency Levels: Distributed Data Stores
Consistency is about keeping the replica nodes in sync with the leader & other replica nodes in a distributed data store.
Prerequisites —
In this article, we are going to talk about consistency in distributed systems. Consistency in a distributed context is not the same as consistency in RDBMS(Single Server). Consistency in a traditional RDBMS is about ensuring data correctness through constraints. You can find out more about Consistency in RDBMS through this blog.
Consistency is about keeping the replica nodes in sync with the leader & other replica nodes in a distributed data store.
Why is Consistency Important?
If you’re using a distributed data store, & you’re not having consistency guarantees, a write to the data store might be written to the leader, but since the replica nodes are not in sync with the leader, the subsequent reads might not show the latest write.
Similarly, if some replicas are in sync, but others are not, some users might see the latest write, while others would not. This too leads to Inconsistent behaviour from the system.
Consistency Levels(CL)
Consistency Level is the number of replica nodes that must acknowledge a read/write request to be successful for the entire request to be considered successful.
Notice that I mentioned both read & write requests in the definition. That’s because we can define consistency level for both read & write requests.
Read CL — Read Consistency Level can be defined as the number of replica nodes that must acknowledge the latest copy of the data in its partition to the user.
Write CL — Write Consistency Level can be defined as the number of replica nodes that must successfully acknowledge a write of the latest data to its partition.
CL Values — You can set different values for Consistency Level in your data store for read/write requests -
ONE — Only one node must acknowledge a read/write request.
TWO/THREE… — N(2/3…) nodes must acknowledge a read/write request.
ALL — All nodes must acknowledge a read/write request.
QUORUM — The quorum of nodes must acknowledge a read/write request.
LOCAL_QUORUM — The majority of nodes in the local data centre must acknowledge read/write requests.
Consistency
I’ll talk about different scenarios of Consistency here -
If any writes to your cluster are always synchronously replicated to all replica nodes, all your replica nodes will always be in sync & your reads irrespective of the replica from which they are coming from will always be Consistent. This is called Immediate Consistency.
If any writes to your cluster are always synchronously replicated to a Quorum of replica nodes, any reads coming from a replica node that’s part of the Quorum will have the latest write, while reads coming from replica nodes that are not part of the Quorum will not have the latest write, leading to Inconsistent state as the latest write would not be available during reads.
If any writes to your cluster are always asynchronously replicated, then the replica nodes can get data at different times and hence some read requests will show you the latest writes while some won’t. Again, an Inconsistent state as the latest write would not be available during reads. This is called Eventual Consistency.
Depending on the requirements of your system, you can play around with the Read & Write Consistency Level to alter the Consistency, Availability & Throughput of your distributed systems. In the next article, we will go through the impact of choosing different consistency levels in detail.