Database Sharding

How it works

Sharding partitions data horizontally across multiple database nodes by a shard key, so each holds only a slice. It scales writes and storage past a single machine's limits — at the cost of cross-shard queries and rebalancing complexity.

Mental models

• The shard key determines which node owns each row — choose it carefully.
• Range, hash, and directory-based sharding trade locality for balance.
• Cross-shard joins and transactions are the expensive parts.
• Federation (splitting by function — users, products, forums) is the lighter-weight sibling: fewer, coarser databases instead of many uniform shards.
• Smaller shards mean more of the working set fits in memory, which is often the real performance win.

Common pitfalls

• A skewed shard key creates hot shards — a handful of power users can overload one node while others idle.
• Rebalancing after adding a shard means moving data live; hashing the key (ideally with consistent hashing) limits how much has to move.
• Application code now has to know which shard to query — 'just add an index' no longer applies uniformly.