Redis uses partitioning and sharding to distribute data across multiple nodes, allowing you to scale your database horizontally as your data grows. Instead of storing all data on a single server, Redis splits the dataset among several Redis instances. This approach increases both storage capacity and throughput, as each node handles only a portion of the data and request load.

Sharding is the process of dividing your data into distinct subsets, or "shards," each managed by a separate Redis node. When you use a Redis Cluster, the system automatically determines which node is responsible for a given key. This is achieved through a concept called hash slots.

Redis Cluster divides the keyspace into 16,384 hash slots. When you add a key, Redis calculates a hash value for the key and assigns it to one of these slots. Each node in the cluster is responsible for a subset of the hash slots. When you perform a read or write operation, Redis uses the key's hash slot to route the request directly to the correct node.

This distribution method improves performance by ensuring that no single node becomes a bottleneck. As your data or traffic increases, you can add more nodes to the cluster, and Redis will rebalance the hash slots among all nodes. This results in better load distribution, faster response times, and higher availability, making Redis suitable for applications that demand high scalability and reliability.

Redis Cluster Setup: Partitioning and Sharding Example

Suppose you are building a social media platform that expects millions of users. To handle high traffic and large datasets, you need to split your Redis data across multiple servers using partitioning (also called sharding).

Step 1: Prepare Redis Nodes

Start by launching several Redis instances on different servers or ports. Each instance will become a node in your cluster.

• Start a Redis server on port 7000:

  redis-server --port 7000 --cluster-enabled yes --cluster-config-file nodes-7000.conf --cluster-node-timeout 5000
  

  • --port 7000 sets the port number.
  • --cluster-enabled yes turns on cluster mode.
  • --cluster-config-file nodes-7000.conf specifies the configuration file for cluster state.
  • --cluster-node-timeout 5000 sets the timeout for node communication (in milliseconds).

Repeat this step for ports 7001, 7002, and so on, for as many nodes as you need.

Step 2: Create the Cluster

After all nodes are running, use the Redis CLI tool to create the cluster:

redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 --cluster-replicas 1

• The command lists each node's address and port.
• --cluster-replicas 1 means each master node will have one replica for high availability.

This command distributes data across the nodes by assigning each a range of hash slots. Redis uses 16,384 hash slots to divide data, so each key is mapped to a slot, and each slot is assigned to a node. This is how partitioning and sharding work in Redis.

Step 3: Interact with the Cluster

Connect to any node and set a key:

redis-cli -c -p 7000
> SET user:1000 "Alice"

• The -c flag enables cluster mode in the CLI.
• Redis automatically places the key in the correct shard based on its hash slot.

When you retrieve or set more keys, Redis transparently routes the request to the right node. This setup allows you to scale horizontally and handle large, distributed datasets efficiently.