wedb_embed: An embedded Redis-compatible LSM disk database engine in Rust

[x-at-01]

wedb_embed: An embedded Redis-compatible LSM disk database engine in Rust

Hi Kvrocks community,

Since Kvrocks is a major project in the disk-backed Redis ecosystem, I wanted to share a related project we recently open-sourced: wedb_embed.

While Kvrocks provides a client-server Redis replacement over RocksDB in C++, wedb_embed focuses on an embedded, in-process engine written in pure Rust on top of the Fjall LSM storage engine.

_{Benchmark Environment: CPU: AMD EPYC 7763 (4 cores) | Memory: 15.6 GB | Disk: Cloud Standard SSD | OS: Ubuntu 24.04.4 LTS | Rust: 1.98.0 | Redis: v8.10.1}

Storage architecture and encoding

Similar to Kvrocks's approach of mapping Redis structures onto LSM key-value storage, wedb_embed organizes data across two LSM keyspaces:

• data keyspace: stores raw string values, subkeys for composite types, and blob references. Uses 8KB block size, LZ4 compression, and large-value KV separation (4KB threshold) to mitigate write amplification during LSM compaction.
• meta keyspace: stores composite key metadata (KeyMeta), version counters, and tenant catalog entries. Uses 4KB block size, 100% in-memory hash indexing, and pinned index/filter blocks for low-latency point lookups.
• Atomic updates: cross-keyspace updates are committed atomically via write batches to preserve crash consistency.
• Key encoding: uses Order-Preserving Prefix Varint (OPPV) with 64-byte stack buffers (SmallKey) to avoid heap allocation during key construction.
• Supported structures: 16 Redis-compatible data models, including String, Hash (with field-level TTL), List, Set, ZSet, Bitmap, JSON (JSONPath), Bloom/Cuckoo filters, TimeSeries, Geo, HyperLogLog, TDigest, Streams, Full-Text Search, and HNSW vector search, alongside multi-tenant namespace isolation.

Benchmarks against standard Redis

On a 5GB dataset (5,000,000 structured items across data formats):

• Physical disk usage: 1,179 MB vs 7,531 MB for Redis AOF (84% reduction through LZ4 block compression and KV separation).
• Resident memory (RSS): 281 MB vs 4,800 MB for Redis (94% reduction). Memory is bounded by the LSM block cache and memtable limits rather than total data volume.
• Latency: Running in-process eliminates IPC, network syscalls, and RESP serialization, bringing P95 latency down to the microsecond and sub-microsecond range.

We appreciate the work the Kvrocks community has done in exploring disk-based Redis architectures and would be happy to exchange ideas around LSM encoding and embedded design.

Links:

• Repository: https://github.com/webc-site/wedb_embed
• Examples: https://github.com/webc-site/wedb_embed/tree/main/examples
• Crates.io: https://crates.io/crates/wedb_embed
• Docs.rs: https://docs.rs/wedb_embed

[git-hulk]

Please stop sending those promotional information here.