Redis-lite: Concurrent In-Memory Key-Value Store

Project Overview

Redis-lite is a high-throughput, low-latency, in-memory key-value store built entirely from scratch in C. It demonstrates low-level systems programming concepts by implementing a custom TCP networking stack, a robust Producer-Consumer thread pool, and a dynamically resizing Hash Table synchronized via fine-grained Reader-Writer locks.

This project was built to explore and solve the concurrency, memory management, and I/O challenges inherent in modern database engines.

Features

• Custom TCP Stack: Raw POSIX socket manipulation bypassing heavy HTTP overhead.
• Concurrent Clients: Handles massive parallel connections via an asynchronous accept loop.
• Thread Pool Architecture: Fixed-size POSIX thread pool entirely bounding OS context switching.
• Dynamic Hash Table: djb2 hashed, separate-chaining table achieving strict $\mathcal{O}(1)$ lookups.
• Load-Aware Resizing: Automatically rehashes memory when the load factor exceeds 0.75.
• Reader-Writer Locks: pthread_rwlock_t guarantees unbounded parallel reads while safely isolating writes.
• Graceful Shutdown: Intercepts SIGINT to safely clean up all threads, sockets, and memory allocations.
• Strictly Modular: Absolute separation of concerns between Networking, Parsing, and Storage.

Architecture

[ TCP Network ] --> [ accept() Loop ] --> [ Thread-Safe Job Queue ]
                                                   | (Condition Var)
                                          +-----------------+
                                          | Thread Pool (N) |
                                          +-----------------+
                                                   | (RW Lock)
                                          +-----------------+
                                          |   Hash Table    |
                                          +-----------------+

Project Structure

redis-lite/
├── Makefile
├── README.md
├── benchmarks/
│   └── benchmark.c
├── docs/
│   ├── ARCHITECTURE.md
│   ├── BENCHMARK.md
│   ├── CONCURRENCY.md
│   ├── DESIGN_DECISIONS.md
│   ├── HASH_TABLE.md
│   ├── INTERVIEW.md
│   ├── NETWORKING.md
│   ├── PROJECT_JOURNEY.md
│   └── THREADPOOL.md
├── include/
│   ├── executor.h
│   ├── parser.h
│   ├── server.h
│   ├── store.h
│   └── threadpool.h
├── src/
│   ├── executor.c
│   ├── main.c
│   ├── parser.c
│   ├── server.c
│   ├── store.c
│   └── threadpool.c
└── tests/
    └── store_test.c

Build & Run Instructions

Prerequisites

• GCC or Clang (supporting C17)
• Make
• POSIX-compliant OS (Linux, macOS, BSD)

Build & Run

make all
./bin/redis-lite

Test & Benchmark

make test
./bin/store_test

make benchmark
./bin/benchmark 100000

Supported Commands

Command	Syntax	Description
PING	PING	Returns PONG. Used for connection health checks.
SET	SET <key> <value>	Creates or updates a key-value pair in memory. Returns OK.
GET	GET <key>	Retrieves the value for a key. Returns <value> or (nil).
DEL	DEL <key>	Deletes a key from memory. Returns 1 (deleted) or 0 (not found).
EXISTS	EXISTS <key>	Checks if a key exists. Returns 1 (exists) or 0 (does not exist).
EXIT	EXIT	Gracefully terminates the client TCP connection.

Example Usage

$ nc 127.0.0.1 6379
PING
PONG
SET user:1000 alice
OK
GET user:1000
alice
EXISTS user:1000
1
DEL user:1000
1
GET user:1000
(nil)
EXIT

Performance Benchmark

Tested on localhost loopback with $N$ iterations using the integrated benchmark suite.

Operation	Throughput (ops/sec)	Average Latency ($\mu$s)
SET	[ Placeholder ]	[ Placeholder ]
GET	[ Placeholder ]	[ Placeholder ]

Documentation

For an in-depth dive into the internal systems, refer to the documentation:

• Architecture
• Design Decisions
• Hash Table Internals
• Concurrency Model
• Networking Details
• Thread Pool Model
• Performance Benchmarking
• Project Evolution Journey
• Interview Prep Guide

Future Improvements

• RESP Protocol: Upgrading from the plaintext parser to the binary-safe Redis Serialization Protocol.
• Persistence: Implementing an Append-Only File (AOF) background thread.
• Replication: Master-Replica log shipping for high availability.
• Epoll/Kqueue: Replacing blocking thread sockets with an asynchronous event loop to handle millions of idle websockets.
• Per-bucket Locking: Striping the RW-Lock across hash table segments to allow parallel SET operations.

Skills Demonstrated

Domain	Applied Technologies
Language	C (C17 standard, -Werror -pedantic)
System Programming	POSIX, Linux APIs, Signal Handling (sigaction)
Networking	TCP/IP, Sockets, Byte Stream Processing
Multithreading	pthreads, Producer-Consumer Model
Synchronization	pthread_mutex_t, pthread_cond_t, pthread_rwlock_t
Memory Management	Pointers, Heap Allocation, Memory Alignment
Data Structures	Hash Tables, Linked Lists, FIFO Queues, djb2 Hashing

License

This project is licensed under the MIT License - see the LICENSE file for details.