The ClustrixDB Rebalancer: Keeping a Distributed Database Balanced

The ClustrixDB Rebalancer

ClustrixDB is a single logical DBMS made up of many individual servers. These servers have matching hardware, and are equal peers within the cluster. Each server holds part of the relational dataset. Any of them can participate in (or initiate) a SQL query. To fully utilize the total hardware in a cluster, workload and data storage must be somewhat evenly distributed across every participating machine. This is the only way to ensure that no one machine becomes a bottleneck. And we must maintain this equilibrium even when servers join or leave the cluster. We have addressed this with a set of background tasks that run on one of the servers (chosen each time a stable cluster forms). Collectively, these tasks make up the software agent we call the Rebalancer. The Rebalancer is responsible for data placement in ClustrixDB. Each of its tasks manages a different aspect of balance.

The Rebalancer is responsible for:

  • Maintaining even disk usage across every disk in the cluster
  • Keeping table data spread across the entire cluster, instead of clumped together on a few servers
  • Fixing lumpy table distributions by adjusting table slices or hash distribution functions
  • Duplicating replicate data that was partially lost during node or disk failures
  • Cleaning up extra, unneeded replicate data

In practice we have found that balancing hash-distributed table data across the cluster results in acceptably balanced workloads for each server. But to ensure that workloads are as even as possible, the Rebalancer also periodically adjusts the ranking of each piece of replicate data (used to decide which replica is preferred for reads).

All of this is done automatically, without intervention by an administrator. The Rebalancer’s work can be tracked through several system relations that record attempted and completed actions. And the Rebalancer can be tuned to act more or less aggressive by way of configuration settings.