Redis Deep Dive: Data Structures, Persistence, Pub/Sub and Production Patterns

Redis Is Not a Cache

Redis is often introduced as a cache in front of a database. That is one use case. Redis is actually an in-memory data structure server — it stores data structures (not just key-value pairs) entirely in RAM and operates on them with O(1) to O(log n) operations.

The distinction matters. When you understand Redis as a set of data structures with specific complexity guarantees, you stop thinking "where can I cache things?" and start thinking "which Redis data structure eliminates this application-level complexity?"

This post covers each data structure, what it is built for, and the production patterns built on top of them.

Core Redis Commands Before the Data Structures

# Key management
SET key value           # store a string value
GET key                 # retrieve a value
DEL key [key ...]       # delete one or more keys
EXISTS key              # returns 1 if exists, 0 if not
TYPE key                # returns the data type of the key
TTL key                 # returns remaining time-to-live (-1 = no expiry, -2 = not found)
EXPIRE key seconds      # set expiry on an existing key
PERSIST key             # remove expiry

# Atomic operations
INCR key                # atomically increment integer by 1
INCRBY key amount       # atomically increment by amount
DECR key                # atomically decrement

# Scan (never use KEYS * in production — O(n) blocks server)
SCAN cursor [MATCH pattern] [COUNT count]  # iterative scan, non-blocking

Data Structure 1: Strings

Despite the name, Redis Strings can store strings, integers, or binary data up to 512 MB.

# Simple caching
SET product:7:price "99.99" EX 3600    # cache for 1 hour
GET product:7:price

# Atomic counter (thread-safe — no race condition)
INCR page:views:article:42             # returns new value
INCRBY user:42:points 100              # add 100 points atomically

# Distributed lock (NX = only set if Not eXists, EX = expiry)
SET lock:order:12345 "worker-1" NX EX 30
# Returns OK if lock acquired, nil if already locked

# Rate limiting counter
INCR ratelimit:ip:192.168.1.1
EXPIRE ratelimit:ip:192.168.1.1 60     # reset window every 60 seconds

# Conditional set (compare-and-swap via Lua script)
# SET if value matches expected (optimistic concurrency)

Production use cases: Caching, distributed locks, counters, session tokens, feature flags, idempotency keys.

Data Structure 2: Lists

Ordered sequences of strings, implemented as a doubly linked list. O(1) push/pop from both ends. O(n) access by index.

# Stack (LIFO): push and pop from same end
LPUSH mystack "item3"              # push to left (head)
LPUSH mystack "item2" "item1"
LPOP  mystack                      # pop from left → "item1"

# Queue (FIFO): push to one end, pop from other
RPUSH myqueue "job1"               # push to right (tail)
RPUSH myqueue "job2" "job3"
LPOP  myqueue                      # pop from left → "job1"

# Blocking pop (waits for items if queue is empty)
BLPOP myqueue 30                   # block up to 30 seconds for an item
# Perfect for job queues — worker waits for work without polling

# Capped list (keep last N items)
LPUSH recent:orders "order:999"
LTRIM recent:orders 0 99           # keep only last 100 items
# Atomic when combined: always exactly 100 most recent items

Production pattern — Job Queue:

# Producer
redis.rpush("jobs:email", json.dumps({
    "to": "alice@example.com",
    "subject": "Order confirmed",
    "order_id": 12345
}))

# Worker (blocking — no busy polling)
while True:
    _, job_data = redis.blpop("jobs:email", timeout=30)
    if job_data:
        job = json.loads(job_data)
        send_email(job["to"], job["subject"])

Production use cases: Job queues, task processing, activity feeds (latest N items), message buffers.

Data Structure 3: Sets

Unordered collections of unique strings. O(1) add, remove, and membership check. Set operations (union, intersection, difference) in O(n).

# Add members
SADD tags:post:42 "database" "nosql" "redis"
SADD tags:post:42 "database"          # duplicate — no effect, returns 0

# Membership check (O(1))
SISMEMBER tags:post:42 "redis"        # returns 1 (is a member)
SISMEMBER tags:post:42 "python"       # returns 0 (not a member)

# All members
SMEMBERS tags:post:42                 # returns all tags (unordered)

# Set operations
SUNION  tags:post:42 tags:post:43     # all tags across both posts
SINTER  tags:post:42 tags:post:43     # tags that appear in both posts
SDIFF   tags:post:42 tags:post:43     # tags in post 42 but not post 43

# Random member (useful for sampling)
SRANDMEMBER tags:post:42 3            # 3 random tags

# Track unique visitors (HyperLogLog is better for massive scale)
SADD visitors:page:homepage "user:42" "user:17" "user:99"
SCARD visitors:page:homepage          # count of unique visitors

Production pattern — Online users / presence:

# User connects
redis.sadd("online_users", user_id)

# User disconnects
redis.srem("online_users", user_id)

# Count online users
redis.scard("online_users")

# Check if specific user is online
redis.sismember("online_users", user_id)

Production use cases: Unique visitor tracking, tagging systems, friend relationships, deduplication, presence indicators.

Data Structure 4: Sorted Sets (ZSets)

Unique members, each with a floating-point score. Sorted by score. O(log n) add, remove, and range queries. The most powerful Redis data structure.

# Add members with scores
ZADD leaderboard 9850 "alice"
ZADD leaderboard 9200 "bob"
ZADD leaderboard 9600 "carol"

# Rankings
ZRANK  leaderboard "alice"            # 0-indexed rank (0 = highest if reversed)
ZREVRANK leaderboard "alice"          # rank from highest score (0 = #1)

# Range by rank
ZREVRANGE leaderboard 0 9 WITHSCORES  # top 10 with scores

# Range by score
ZRANGEBYSCORE leaderboard 9000 9999   # all players with score 9000-9999
ZCOUNT        leaderboard 9000 9999   # count players in score range

# Update score
ZINCRBY leaderboard 150 "bob"         # atomically add 150 to bob's score

# Remove member
ZREM leaderboard "alice"

Production pattern — Real-time leaderboard:

# Record a score
def record_score(user_id: str, score: float):
    redis.zadd("game:leaderboard", {user_id: score})

# Get top 10 with ranks
def get_top10():
    return redis.zrevrange("game:leaderboard", 0, 9, withscores=True)

# Get a user's rank (1-indexed)
def get_rank(user_id: str) -> int:
    rank = redis.zrevrank("game:leaderboard", user_id)
    return rank + 1 if rank is not None else None

# Get users near a given user (±5 ranks)
def get_nearby(user_id: str):
    rank = redis.zrevrank("game:leaderboard", user_id)
    start = max(0, rank - 5)
    end = rank + 5
    return redis.zrevrange("game:leaderboard", start, end, withscores=True)

Production pattern — Delayed job queue (process jobs at scheduled time):

# Schedule a job to run at a specific Unix timestamp
def schedule_job(job_id: str, run_at: float, payload: dict):
    redis.zadd("scheduled_jobs", {json.dumps(payload): run_at})

# Worker: pull jobs whose score (run_at) <= now
def process_due_jobs():
    now = time.time()
    jobs = redis.zrangebyscore("scheduled_jobs", 0, now, start=0, num=10)
    for job_data in jobs:
        job = json.loads(job_data)
        process(job)
        redis.zrem("scheduled_jobs", job_data)

Production use cases: Leaderboards, rate limiting with sliding windows, delayed job queues, priority queues, autocomplete (prefix scoring), trending items (score = recency × engagement).

Data Structure 5: Hashes

Maps of field-value pairs stored under one key. Like a mini-document. O(1) field access.

# Store user data
HSET user:42 name "Alice" email "alice@example.com" age 30
HGET user:42 name                   # "Alice"
HMGET user:42 name email            # ["Alice", "alice@example.com"]
HGETALL user:42                     # all fields and values
HKEYS user:42                       # ["name", "email", "age"]

# Update single field without loading entire document
HSET user:42 email "newalice@example.com"

# Atomic increment on a hash field
HINCRBY user:42 login_count 1

# Check field existence
HEXISTS user:42 phone               # returns 0 (field not set)

When to use Hash vs String (JSON):

# String (JSON): entire object as one value — must serialize/deserialize
redis.set("user:42", json.dumps(user))
user = json.loads(redis.get("user:42"))
# To update one field: GET → deserialize → update → serialize → SET
# Problem: race condition if two processes update different fields simultaneously

# Hash: individual fields are independent values
redis.hset("user:42", "email", "newalice@example.com")
# No race condition — only the email field is touched
# Downside: cannot set TTL on individual hash fields (only the whole hash)

Production use cases: User profiles, session data with multiple fields, shopping carts, configuration storage, counters grouped by entity.

Data Structure 6: Streams

An append-only log of messages, each with a unique auto-generated ID. Supports consumer groups — multiple consumers sharing a stream, each processing different messages.

# Produce messages
XADD events:orders * customer_id 42 amount 99.99 product_id 7
# * = auto-generate ID (timestamp-sequence, e.g. 1710000000000-0)

# Read new messages
XREAD COUNT 10 STREAMS events:orders 0-0     # read from the beginning
XREAD COUNT 10 STREAMS events:orders $        # read only new messages

# Consumer groups (multiple consumers share the stream)
XGROUP CREATE events:orders order-processor $ MKSTREAM

# Consumer reads messages (XREADGROUP)
XREADGROUP GROUP order-processor worker-1 COUNT 5 STREAMS events:orders >

# Acknowledge processed message (prevents re-delivery)
XACK events:orders order-processor 1710000000000-0

Streams vs Pub/Sub:

Persistence: RDB vs AOF

Redis is in-memory, but it persists data to disk for recovery.

RDB (Redis Database) — Snapshots:

# redis.conf
save 900 1        # snapshot if ≥1 key changed in 900 seconds
save 300 10       # snapshot if ≥10 keys changed in 300 seconds
save 60 10000     # snapshot if ≥10000 keys changed in 60 seconds

Periodically writes a compact binary snapshot of the entire dataset. Fast to restore (load the file). Risk: data since the last snapshot is lost on crash.

AOF (Append-Only File) — Write log:

# redis.conf
appendonly yes
appendfsync everysec   # fsync every second (recommended balance)
# appendfsync always   # fsync every write (safest, slowest)
# appendfsync no       # let OS decide (fastest, least safe)

Logs every write operation. Replay the log on restart to reconstruct the dataset. Data loss window: at most 1 second (with appendfsync everysec).

In production: use both RDB + AOF:

# redis.conf — production recommended
save 3600 1           # RDB snapshot hourly (fast recovery for major outages)
appendonly yes
appendfsync everysec  # AOF for recent data (at most 1s loss)
aof-use-rdb-preamble yes  # hybrid: RDB snapshot + AOF tail (fast load + minimal loss)

Eviction Policies

When Redis reaches maxmemory, it evicts keys based on the configured policy:

# redis.conf
maxmemory 4gb
maxmemory-policy allkeys-lru   # evict least recently used from all keys

Production Patterns

Cache-Aside (Lazy Loading)

def get_product(product_id: int) -> dict:
    cache_key = f"product:{product_id}"

    # 1. Check cache
    cached = redis.get(cache_key)
    if cached:
        return json.loads(cached)

    # 2. Cache miss: fetch from database
    product = db.query("SELECT * FROM products WHERE id = %s", product_id)

    # 3. Write to cache with TTL
    redis.set(cache_key, json.dumps(product), ex=3600)

    return product

Write-Through (Always Write to Cache)

def update_product_price(product_id: int, new_price: float):
    # 1. Write to database
    db.execute("UPDATE products SET price = %s WHERE id = %s", new_price, product_id)

    # 2. Immediately update cache
    cache_key = f"product:{product_id}"
    cached = redis.get(cache_key)
    if cached:
        product = json.loads(cached)
        product["price"] = new_price
        redis.set(cache_key, json.dumps(product), ex=3600)

Rate Limiting with Sliding Window

def is_rate_limited(user_id: str, limit: int = 100, window_seconds: int = 60) -> bool:
    key = f"ratelimit:{user_id}"
    now = time.time()
    window_start = now - window_seconds

    pipe = redis.pipeline()
    pipe.zremrangebyscore(key, 0, window_start)      # remove old entries
    pipe.zadd(key, {str(now): now})                   # add current request
    pipe.zcard(key)                                   # count requests in window
    pipe.expire(key, window_seconds)                  # auto-expire the key
    _, _, count, _ = pipe.execute()

    return count > limit

Distributed Lock (Redlock Pattern)

import uuid

def acquire_lock(resource: str, ttl_seconds: int = 30) -> str | None:
    lock_key = f"lock:{resource}"
    lock_value = str(uuid.uuid4())  # unique value per lock holder
    acquired = redis.set(lock_key, lock_value, nx=True, ex=ttl_seconds)
    return lock_value if acquired else None

def release_lock(resource: str, lock_value: str) -> bool:
    # Lua script ensures atomic check-and-delete (avoid releasing another holder's lock)
    lua_script = """
    if redis.call("get", KEYS[1]) == ARGV[1] then
        return redis.call("del", KEYS[1])
    else
        return 0
    end
    """
    lock_key = f"lock:{resource}"
    return bool(redis.eval(lua_script, 1, lock_key, lock_value))

# Usage
lock_value = acquire_lock("order:12345")
if lock_value:
    try:
        process_order(12345)
    finally:
        release_lock("order:12345", lock_value)

🧭 What's Next

• Post 22: Cassandra Deep Dive — built for write-heavy workloads at massive scale, but only if you model data for your queries; the partition key, clustering key, and tunable consistency explain everything