A distributed blob store in Java 21 + Spring Boot. Files are spread across storage nodes with a consistent hash ring using virtual nodes, replicated to three distinct nodes, health-monitored, and redistributed automatically when nodes fail, recover, join or leave.
Runs end to end with one command:
cp .env.example .env
docker compose up --build# upload
curl -X POST -F "file=@sample.pdf" http://localhost:8080/api/files
# → {"fileId":"b0c2...","primaryNode":"node-2","replicas":["node-3","node-1"],"status":"ACTIVE", ...}
# kill the node that holds the primary copy - the file is still there
docker stop dfs-node-2
curl -s -D - http://localhost:8080/api/files/b0c2... -o out.pdf | grep X-Served-By
# → X-Served-By-Node: node-3- Consistent hash ring with virtual nodes (100 per node) for placement — adding a 4th node moves ~25 % of the data instead of the ~75 % that
hash % Nwould move. - Replication factor 3 across distinct physical nodes, with parallel replica writes and automatic primary failover during an upload.
- Fault-tolerant reads — a download transparently falls over to a healthy replica; the response names the node that served it.
- Scheduled failure detection with failure/success thresholds, so a GC pause does not trigger a cluster-wide reshuffle.
- Real rebalancing — a node join or departure moves only the affected files, driven by one idempotent reconcile routine.
- Self-healing — a background repair job restores under-replicated files, finishes deletes that a down node blocked, and reconciles uploads abandoned by a crashed coordinator.
- SHA-256 end to end — computed at the edge, verified on write, on copy, on download, and on demand (
POST /api/files/{id}/verify); a corrupt replica is detected, quarantined and rebuilt. - Honest status reporting — a file whose third replica failed comes back
DEGRADEDwith the reason, never as a silent success. - Metadata in PostgreSQL (Flyway migrations, indexes, optimistic locking), Redis for metadata caching, shared node-health state and rebalancing leases — and the system keeps working when Redis is down.
- Observability — Actuator, Prometheus metrics for every operation, structured logs.
- Tested — 136 tests: consistent-hashing properties, service behaviour, and integration tests on real PostgreSQL + Redis (Testcontainers) covering upload → failure → failover → recovery → rebalancing.
┌──────────┐
│ Client │
└────┬─────┘
│ REST (multipart upload / streaming download)
▼
┌────────────────────────────────────────────────────────────────────────────────┐
│ COORDINATOR (Spring Boot) │
│ │
│ FileController ─▶ FileService ─▶ FilePlacementService ─▶ ConsistentHashRing │
│ │ NavigableMap<Long, │
│ │ nodeId>, 100 vnodes │
│ ├─▶ ReplicationService (streams blobs, parallel) │
│ ├─▶ FileMetadataStore (short transactions only) │
│ └─▶ MetadataCache (Redis, optional) │
│ │
│ background: NodeHealthService ──▶ ring membership │
│ RebalancingService ──▶ FileReconciler ◀── ReplicationRepairJob │
└──────┬──────────────────────────┬──────────────────────────┬───────────────────┘
│ JDBC │ RESP │ HTTP, streaming
▼ ▼ ▼
┌──────────────┐ ┌──────────────────┐ ┌───────────┬───────────┬───────────┐
│ PostgreSQL │ │ Redis │ │ node-1 │ node-2 │ node-3 │
│ │ │ │ │ │ │ │
│ files │ │ metadata cache │ │ /data/ │ /data/ │ /data/ │
│ file_replicas│ │ node health │ │ node-1/ │ node-2/ │ node-3/ │
│ storage_nodes│ │ rebalance lease │ │ ab/cd/ │ ab/cd/ │ ab/cd/ │
└──────────────┘ └──────────────────┘ └───────────┴───────────┴───────────┘
metadata optimisation Spring Boot + local disk each
(authoritative) (never required) (no cluster knowledge at all)
Two applications, because they have opposite failure and scaling profiles:
- the coordinator owns placement, metadata, failure detection and data movement, and is stateless (its ring is rebuilt from the database on startup, so it can run behind a load balancer);
- a storage node is a keyed blob store with a health endpoint — no ring, no peer list, no replication logic. Adding capacity is "start another container", and there is no gossip protocol to get wrong.
A shared common module holds the wire contract and the checksum code so the two cannot drift apart.
Full detail, including sequence diagrams and failure analysis: ARCHITECTURE.md.
| Layer | Choice | Why |
|---|---|---|
| Language / runtime | Java 21 | Virtual threads for health probes, records for DTOs |
| Framework | Spring Boot 3.5 (Web, Data JPA, Data Redis, Validation, Actuator) | |
| Metadata store | PostgreSQL 16 | Transactions, indexes, real queries over replica placement |
| Migrations | Flyway | Schema is versioned SQL; Hibernate runs ddl-auto: validate |
| Cache / coordination | Redis 7 (Lettuce) | Metadata cache, shared health view, TTL'd distributed locks |
| Blob transport | JDK HttpClient |
True streaming both ways (BodyPublishers.ofFile, BodyHandlers.ofInputStream) |
| Metrics | Micrometer + Prometheus | |
| API docs | springdoc-openapi (Swagger UI) | |
| Build | Maven multi-module (common, coordinator, storage-node) + wrapper |
|
| Tests | JUnit 5, Mockito, AssertJ, Testcontainers | |
| Deployment | Docker + Docker Compose | Whole cluster, one command |
Nodes and files are hashed into the same address space, [0, 2^63), treated as a circle. A file belongs to the first node position clockwise from the file's own position.
0 ─────────────────────────────────────────────────────► 2^63
┌────────────────────────────────────────────────────────┐
...──┤ n1#7 n3#22 n2#4 n1#88 n3#5 n2#61 ... ├──...
└────────────────────────────────────────────────────────┘
▲ ▲
file Y ──┘ └── file X
owner: n2#4 → node-2 owner: n1#88 → node-1
replicas: next 2 distinct nodes clockwise → node-1, node-3
Each physical node is hashed onto 100 positions (node-1#0 … node-1#99), held in a NavigableMap<Long, String> (ring position → node id):
// coordinator/src/main/java/com/example/distributedstorage/hashing/ConsistentHashRing.java
public boolean addNode(StorageNode node) { ... } // inserts V positions
public boolean removeNode(String nodeId) { ... } // removes V positions
public Optional<StorageNode> getNode(String key) { ... } // ceilingEntry, wrapping to firstEntry
public List<StorageNode> getReplicaNodes(String key, int replicationFactor) { ... }- Lookup is
ring.ceilingEntry(hash(key)), wrapping tofirstEntry()past the end — O(log V). - Replica selection keeps walking clockwise and collects distinct physical nodes, so "3 replicas" can never mean three copies on one machine.
- Adding a node inserts its 100 positions and steals only the keys in the arcs immediately counter-clockwise of each one — no other node's ownership changes.
- Removing a node deletes its positions; each arc is absorbed by the next node clockwise. Nothing else moves.
- The hash is MD5 (first 8 bytes, sign bit cleared): fast, uniform, and byte-identical on every JVM and platform — a file must map to the same position on every coordinator, forever. It is not a security decision; data integrity uses SHA-256.
Why virtual nodes are not optional. With one position per node, three random positions on a circle give arcs like 60 / 25 / 15, and when a node dies its entire arc lands on one neighbour — doubling that neighbour's load exactly when the cluster is already degraded. With V positions each node's share is the sum of V samples, so deviation shrinks with 1/√V, and a dead node's arcs spread over all survivors.
Measured, not asserted (GET /api/nodes/ring reports each node's summed arc length):
{ "physicalNodes": 3, "virtualNodesPerNode": 100, "totalRingPositions": 300,
"nodes": [ {"nodeId":"node-3","keyspaceSharePercent":35.21},
{"nodeId":"node-1","keyspaceSharePercent":33.05},
{"nodeId":"node-2","keyspaceSharePercent":31.74} ] }hash(fileId) % N ties every key's owner to the number of nodes, so changing N changes almost every key's owner:
| Change | hash % N moves |
Consistent hashing moves |
|---|---|---|
| 3 → 4 nodes | ~75 % of all files | ~25 % (1/(n+1)) |
| 4 → 5 nodes | ~80 % | ~20 % |
| 1 of 4 nodes dies | ~75 % | ~25 % (only the dead node's share) |
A 75 % reshuffle means the cluster saturates its own network for the whole migration window, and until it finishes every read has to guess whether a key has already moved. The test suite measures both schemes on 10 000 keys and asserts the gap, plus the property that matters most: every key that moved, moved to the new node — no key is disturbed for no reason.
// ConsistentHashRingTest
assertThat(movedFraction).isBetween(0.15, 0.40); // ~1/(n+1)
assertThat(moved).allSatisfy(key -> assertThat(after.get(key)).isEqualTo("node-4"));
assertThat(moduloMoves).isGreaterThan(keys.size() * 7 / 10);
assertThat(ringMoves).isLessThan(moduloMoves / 2);fileA (checksum 4d7f9c...)
├── node-2 PRIMARY ← first node clockwise from hash(fileA)
├── node-3 REPLICA ← next distinct node clockwise
└── node-1 REPLICA ← next distinct node clockwise
- The primary write is synchronous; if that node refuses or is unreachable the next node on the preference list takes the write and is recorded as primary — an upload does not fail because one machine died.
- Replica writes fan out in parallel on a bounded pool, so
RFcosts bandwidth, not latency. - A failed replica write does not fail the upload — the bytes are already durable. The response is
201withstatus: DEGRADEDand the reason, and the repair job rebuilds the copy. - Every copy is verified: the receiving node re-hashes the body and rejects a mismatch; node-to-node copies are re-hashed at the source, in flight and on arrival.
getReplicaNodesreturns fewer nodes rather than duplicates when the cluster is smaller thanRF.
A file is ACTIVE only with RF copies. Fewer — because a write failed or because the cluster only has two nodes — is DEGRADED, since the durability risk is identical.
| Failure | What the client sees |
|---|---|
| Primary node down | Download served by a replica (X-Served-By-Node proves it). Uploads route around it. |
| Replica node down | Nothing. File reported DEGRADED; replication restored automatically. |
| Node returns | Its blobs are re-adopted, not re-copied — recovery costs no network traffic. |
| A copy silently corrupts | Detected by checksum on download or verify; read served from a good copy; bad copy rebuilt. |
| Coordinator dies mid-upload | PENDING rows let the repair job adopt whatever landed, or discard the metadata. No orphans. |
| Redis down | Cache misses and process-local locks. Uploads, downloads, deletes, health and rebalancing keep working. |
| Only one node left | Still serving reads and writes; every file DEGRADED, cluster health DEGRADED. |
| All nodes down | 503 NO_HEALTHY_NODES / 503 STORAGE_UNAVAILABLE. Nothing is lost or rewritten. |
| PostgreSQL down | Requests fail (metadata is authoritative); blobs untouched; service resumes on reconnect. The one deliberate SPOF. |
node fails
│
▼ GET /actuator/health times out (scheduled sweep, 10s, parallel probes)
consecutive failures reach failureThreshold (2)
│
▼
mark UNHEALTHY ─▶ ring.removeNode(node) ─▶ storage_node_failures_total++
│
▼ SELECT file_id FROM file_replicas WHERE node_id = ? (indexed - only its files)
reconcile each affected file
│
├─ copies on the dead node → FAILED (no longer counted as available)
├─ missing copies → copied from a survivor to the new owner
├─ dead primary → a healthy holder promoted
└─ status → ACTIVE if RF copies exist, else DEGRADED
Thresholds exist because evicting a node causes data movement: one missed probe is a GC pause, not a dead machine. Recovery needs successThreshold consecutive successes so a flapping node cannot oscillate the ring. Reads do not wait for any of this — a download tries the next holder the instant a connection fails.
Real output from a verification run (node-2 stopped, then restarted):
ERROR NodeRegistryService - Node node-2 failed 2 consecutive health checks - evicted from the ring, 2 node(s) remain
INFO RebalancingService - Starting rebalancing (trigger=FAILED, node=node-2)
INFO RebalancingService - Found 12 affected file(s) previously held by node-2
INFO FileService - Serving f3a8347e... from replica node-3 after the preferred holder was unavailable
...
INFO NodeRegistryService - Node node-2 recovered after 2 consecutive successful probes - back on the ring (3 node(s))
INFO FileReconciler - Re-adopted existing copy of f3a8347e... on recovered node node-2
INFO RebalancingService - Rebalancing completed (trigger=RECOVERED): 12 scanned, 12 affected,
0 copies created, 0 copies removed, 3 primaries promoted, 0 failure(s) in 263 ms
One routine — FileReconciler.reconcile(file) — defines correct placement, and every trigger uses it: node joined, node failed, node deregistered, manual POST /api/admin/rebalance, and the periodic repair pass. There is exactly one definition of "where should this file be", so the paths cannot disagree and shuffle data forever.
desired = ring.getReplicaNodes(fileId, RF)
1. ADOPT copies on nodes that left the ring → mark FAILED
copies on healthy nodes that HEAD confirms → mark ACTIVE (free recovery)
2. CREATE copy each missing desired copy from a healthy holder, verifying SHA-256
3. DELETE surplus copies - only once every desired copy is confirmed present
4. PROMOTE primary := first desired node that holds the file
5. STATUS ACTIVE / DEGRADED / FAILED
- Create before delete, always. Reclaiming space is never worth risking the last good copy.
- Idempotent by construction — every step is derived from observed state (the ring, plus
HEAD /internal/files/{id}), never from a plan computed earlier. Running it twice does nothing the second time; a run killed halfway leaves nothing to clean up. - Only affected files move. A departure is driven by an indexed query on
file_replicas(node_id, status), so the work is proportional to the dead node's share. A join scans metadata (an in-memory hash per file, microseconds) but moves bytes only for the~1/(n+1)of files whose owners actually changed.
Verified run — a 4th node joining a 3-node cluster holding 12 files:
INFO NodeRegistryService - Node node-4 joined the cluster at http://storage-node-4:8080 - ring now has 4 node(s)
INFO RebalancingService - Starting rebalancing (trigger=JOINED, node=node-4)
INFO FileReconciler - Migrating file 2e43a8ad... -> node-4 (source node-1)
INFO FileReconciler - Migration successful: 2e43a8ad... now stored on node-4
INFO FileReconciler - Removed surplus copy of 2e43a8ad... from node-3 (no longer assigned by the ring)
...
INFO RebalancingService - Rebalancing completed (trigger=JOINED): 12 scanned, 5 affected,
5 copies created, 5 copies removed, 2 primaries promoted, 0 failure(s) in 225 ms
7 of 12 files were never touched; afterwards every file still had exactly 3 active copies, and node-4 held 5 blobs.
storage_nodes files file_replicas
───────────── ───── ─────────────
node_id (PK) ┌──────▶ id (PK, uuid) ┌────────▶ id (PK, bigserial)
host, port │ file_name │ file_id → files.id (CASCADE)
status │ content_type │ node_id → storage_nodes.node_id
registered_at │ size_bytes │ status
last_check_at │ checksum (sha256) │ created_at, updated_at
last_healthy_at │ status │ version
consecutive_failures └──────── primary_node_id │ UNIQUE (file_id, node_id)
consecutive_successes replication_factor │
version (optimistic lock) created_at, updated_at │
version ───────────────┘
files.status:PENDING(upload in flight) →ACTIVE(full replica set) /DEGRADED(below the replication factor) /REBALANCING(copies moving now) /FAILED(no reachable copy) /DELETED(cleanup pending on a down node)file_replicas.status:PENDING(write unconfirmed) →ACTIVE/FAILED(write failed or checksum mismatch) /DELETING(removal pending on an unreachable node)storage_nodes.status:HEALTHY/UNHEALTHY/DEREGISTERED- Indexes:
file_replicas(node_id, status)(the failure-repair query),file_replicas(file_id),files(status),files(created_at DESC),files(primary_node_id),storage_nodes(status) - Blobs are never in PostgreSQL. On a node:
/data/node-1/ab/cd/<fileId>plus a.sha256sidecar, written to.tmpand published with an atomic rename.
Schema lives in V1__initial_schema.sql; Hibernate validates the mapping against it at startup and never generates it.
Full reference with every status code, failure case and curl example: API.md. Swagger UI: http://localhost:8080/swagger-ui.html.
| Method | Path | Purpose |
|---|---|---|
POST |
/api/files |
Upload (multipart file; optional Idempotency-Key) |
GET |
/api/files/{fileId} |
Download, streamed, with replica failover |
GET |
/api/files/{fileId}/metadata |
Metadata + per-replica placement |
GET |
/api/files |
List, newest first (page, size) |
DELETE |
/api/files/{fileId} |
Delete everywhere (204, or 202 if a holder is down) |
POST |
/api/files/{fileId}/verify |
Re-hash every copy; quarantine and rebuild bad ones |
POST |
/api/nodes/register |
Node self-registration (idempotent upsert) |
GET |
/api/nodes |
All known nodes |
GET |
/api/nodes/ring |
Ring: virtual nodes, positions, measured keyspace share |
GET |
/api/nodes/{nodeId} |
One node |
DELETE |
/api/nodes/{nodeId} |
Deregister and drain (terminal) |
POST |
/api/nodes/{nodeId}/rejoin |
Re-admit a deregistered node |
GET |
/api/health |
Cluster health (200 for UP/DEGRADED, 503 for DOWN) |
POST |
/api/admin/rebalance |
Run a rebalance now |
POST |
/api/admin/health-check |
Run a health sweep now |
POST/GET/HEAD/DELETE |
/internal/files/{fileId} |
Node blob API (coordinator only) |
GET |
/internal/files/{fileId}/checksum |
Node re-hashes its copy |
GET |
/internal/stats |
Node blob count and bytes used |
GET |
/actuator/health, /actuator/prometheus |
Both services |
Errors are always the same shape:
{ "timestamp": "2026-09-01T10:15:31.412Z", "status": 404, "error": "FILE_NOT_FOUND",
"message": "File 3f2b... not found", "path": "/api/files/3f2b.../metadata" }git clone <your-repo-url> && cd distributed-file-storage
cp .env.example .env # adjust ports if 8080/5432/6379 are taken
docker compose up --build # postgres, redis, coordinator, storage-node-1..3Wait for the nodes to self-register (a few seconds), then:
curl -s http://localhost:8080/api/nodes | jq '[.[] | {nodeId, status, onRing}]'
curl -s http://localhost:8080/api/health | jqAdd a fourth node to a live cluster and watch it rebalance:
docker compose --profile scale up -d storage-node-4
docker compose logs -f coordinator | grep -E "joined|Migrating|Rebalancing completed"Other useful commands:
docker compose ps # cluster state
docker stop dfs-node-2 && docker start dfs-node-2 # simulate a failure and a recovery
docker compose down # stop
docker compose down -v # stop and delete all dataPorts (all overridable in .env): coordinator 8080, nodes 9001-9004, PostgreSQL 5432, Redis 6379. Inside the network everything is addressed by compose service name (jdbc:postgresql://postgres:5432/..., http://storage-node-1:8080) — never localhost.
Needs Java 21 and a PostgreSQL + Redis you can reach.
# 1. infrastructure
docker run -d --name dfs-postgres -p 5432:5432 \
-e POSTGRES_DB=file_storage -e POSTGRES_USER=storage_user -e POSTGRES_PASSWORD=storage_password \
postgres:16-alpine
docker run -d --name dfs-redis -p 6379:6379 redis:7-alpine
# 2. build (Flyway creates the schema on first start)
./mvnw clean install
# 3. coordinator
java -jar coordinator/target/coordinator-1.0.0.jar
# 4. three storage nodes, each in its own shell
SERVER_PORT=9001 STORAGE_NODE_ID=node-1 STORAGE_DATA_DIRECTORY=./data/node-1 \
STORAGE_ADVERTISED_HOST=localhost STORAGE_ADVERTISED_PORT=9001 \
COORDINATOR_URL=http://localhost:8080 java -jar storage-node/target/storage-node-1.0.0.jar
SERVER_PORT=9002 STORAGE_NODE_ID=node-2 STORAGE_DATA_DIRECTORY=./data/node-2 \
STORAGE_ADVERTISED_HOST=localhost STORAGE_ADVERTISED_PORT=9002 \
COORDINATOR_URL=http://localhost:8080 java -jar storage-node/target/storage-node-1.0.0.jar
SERVER_PORT=9003 STORAGE_NODE_ID=node-3 STORAGE_DATA_DIRECTORY=./data/node-3 \
STORAGE_ADVERTISED_HOST=localhost STORAGE_ADVERTISED_PORT=9003 \
COORDINATOR_URL=http://localhost:8080 java -jar storage-node/target/storage-node-1.0.0.jarOr in one step: ./scripts/run-local.sh (and ./scripts/run-local.sh stop).
./scripts/demo.sh # against a running cluster
./scripts/demo.sh --start # boots the cluster firstIt walks through, pausing so the output is readable:
- cluster starts → 2. nodes self-register, ring shown with each node's keyspace share → 3. six files uploaded → 4. which nodes hold each file (and each node's blob count) → 5. download, bytes compared to the original → 6. the primary node is stopped → 7-8. same file downloaded again, served by a replica → 9. node restarted, its data re-adopted → 10. a fourth node joins, only the affected files move → 11. every copy checksum-verified → 12. cleanup.
Quick manual version:
BASE=http://localhost:8080
echo "hello distributed world" > sample.txt
FILE_ID=$(curl -s -X POST -F "file=@sample.txt" $BASE/api/files | jq -r .fileId)
curl -s $BASE/api/files/$FILE_ID/metadata | jq '{primaryNode, copies:[.replicas[].nodeId], status}'
curl -s $BASE/api/files/$FILE_ID -o downloaded.txt && diff sample.txt downloaded.txt && echo OK
docker stop dfs-node-1
curl -s -X POST $BASE/api/admin/health-check | jq # don't wait for the timer
curl -s -D - $BASE/api/files/$FILE_ID -o out.txt | grep -i x-served-by
curl -X DELETE $BASE/api/files/$FILE_ID./mvnw test # 115 unit tests (4 common + 94 coordinator + 17 storage-node), no Docker needed
./mvnw verify # + 21 integration tests (Testcontainers: real PostgreSQL + Redis)| Suite | What it actually checks |
|---|---|
ConsistentHashRingTest |
Determinism across instances, wrap-around, distinct replicas, prefix-chain preference lists, measured data movement on join/leave vs modulo hashing, virtual-node balance (and that 1 vnode is measurably worse than 200) |
Md5HashFunctionTest |
Stability, non-negative positions, spread of near-identical keys, no collisions over a 20 × 200 ring |
FilePlacementServiceTest / ReplicaSelectionTest |
No duplicate physical nodes, even load, RF > node count, only the failed node's files re-map, a recovered node gets exactly its old files back |
ChecksumServiceTest |
Known SHA-256 vectors, spool-and-hash in one pass, case-insensitive comparison |
FileServiceTest |
Upload ACTIVE/DEGRADED reporting, primary failover, rollback when no node accepts, download failover, corrupt-replica quarantine, full vs partial delete |
FileReconcilerTest |
Rebuilds missing copies, never deletes a surplus copy before the replacements exist, re-adopts a recovered node's data without copying, promotes primaries, reports unrecoverable files |
RebalancingServiceTest |
Aggregated reports, departure path uses the index (no full scan), lock contention skips instead of racing |
ReplicationRepairJobTest |
Retries DEGRADED, FAILED and files stuck mid-move; does nothing while the cluster is down; completes a deferred delete only when no holder is left; adopts or discards uploads abandoned by a crashed coordinator |
NodeHealthServiceTest |
One bad probe does not evict; thresholds do; flapping resets counters; recovery needs consecutive successes |
NodeRegistrationTest |
Idempotent re-announce moves no data, duplicate id rejected, dead node's id reclaimable, deregistration is terminal and re-admission explicit |
LocalBlobStoreTest |
Sharded paths, atomic publish, checksum sidecar, path-traversal rejection, idempotent delete, corruption detection, temp cleanup |
InternalFileApiTest |
The node's HTTP API end to end over a real server |
FileLifecycleIT |
Upload → rows in PostgreSQL → blobs on exactly the ring-selected nodes → streamed download → delete; validation errors; Redis cache hit; idempotency key |
NodeFailureIT |
Reads survive a dead primary, DEGRADED accounting, replication restored onto a spare node, returning node re-adopted with zero copies, disk-loss rebuild, deferred delete completion, single-node survival |
NodeJoinRebalancingIT |
New node gets its share, only affected files move, every file keeps 3 verified copies, rebalancing is idempotent, deregistration redistributes |
Integration tests run the real coordinator, real SQL and real Redis; only the storage nodes are substituted by an in-memory implementation of StorageNodeClient so a node can be killed, corrupted or revived at an exact point instead of with sleeps.
.github/workflows/ci.yml runs on every push and pull request:
| Job | What it proves |
|---|---|
| Unit and integration tests | ./mvnw verify — 136 tests, including the Testcontainers suites against real PostgreSQL and Redis |
| Docker images | Both multi-stage images build from a clean checkout |
| End-to-end smoke test | Boots the full compose cluster, then runs scripts/demo.sh: upload → placement on three nodes → download → kill the primary → download again from a replica → restart → add a fourth node and rebalance, failing the build on any byte mismatch |
load-test/ contains an availability harness (kills a node mid-load and counts verified downloads) and a k6 script. See load-test/README.md. No availability percentage is claimed anywhere in this repository — the harness is there so you can measure one and state the conditions it was measured under.
Everything is environment-driven; see .env.example for the full list.
| Variable | Default | Meaning |
|---|---|---|
REPLICATION_FACTOR |
3 | Copies per file, including the primary |
VIRTUAL_NODES |
100 | Ring positions per physical node |
MAX_FILE_SIZE |
512MB | Upload limit |
HEALTH_CHECK_INTERVAL |
10s | Failure-detector period |
HEALTH_FAILURE_THRESHOLD / _SUCCESS_THRESHOLD |
2 / 2 | Probes needed to evict / re-admit |
REBALANCE_ON_MEMBERSHIP_CHANGE |
true | Rebalance automatically on join/leave |
REPAIR_INTERVAL |
60s | Self-healing pass period |
VERIFY_CHECKSUM_ON_DOWNLOAD |
true | Compare the node's checksum before streaming |
CACHE_ENABLED, CACHE_METADATA_TTL |
true, 2m | Redis metadata cache |
DATASOURCE_URL, POSTGRES_USER, POSTGRES_PASSWORD |
— | Metadata store |
REDIS_HOST, REDIS_PORT |
redis, 6379 | Cache/locks |
Node: STORAGE_NODE_ID, STORAGE_DATA_DIRECTORY, STORAGE_ADVERTISED_HOST/PORT, COORDINATOR_URL, SERVER_PORT |
— | Node identity, disk and how the coordinator reaches it |
No credentials are committed; .env is gitignored and .env.example holds placeholders.
- Coordinator + dumb nodes, not peer-to-peer. One component makes placement decisions, so placement is always consistent and there is no gossip or membership consensus to get wrong. The cost — the coordinator is a bandwidth funnel — is stated in the trade-offs.
- The ring is derived state, PostgreSQL is truth. The ring is rebuilt from
storage_nodeson startup, so a coordinator restart moves no data and two coordinators agree. - No database transaction ever spans network I/O.
FileMetadataStoredoes every write in its own short transaction and orchestrators run untransacted, because holding a connection and row locks across a multi-second blob copy would make one slow node a database-wide stall. (It is a separate bean for a second reason: a@Transactionalmethod called from inside the same class silently runs with no transaction.) - Optimistic locking over pessimistic. Uploads, the health monitor, the repair job and rebalancing touch the same rows; conflicts are rare, so the loser re-reads and re-applies rather than everyone serialising behind a lock.
- Metadata before bytes.
PENDINGrows record intent, so a coordinator crash leaves a reconcilable trace instead of orphaned blobs. - One reconcile routine for all data movement. Join, failure, deregistration, manual trigger and repair all call
FileReconciler. Idempotent, interruptible, and impossible for two paths to disagree. - Create before delete. The invariant that makes rebalancing safe.
- Partial failure is reported, not smoothed over.
DEGRADEDwith a reason,202for a partial delete,unrecoverableFilesin rebalance reports. - Redis is an optimisation, never a dependency. Every call degrades to a miss; the command timeout is 500 ms so a dead Redis fails fast; the Docker healthcheck deliberately excludes Redis. Verified by pulling Redis out from under a running cluster.
- Deregistration is terminal. A drained node's own heartbeat must not silently put it back on the ring; re-admission is an explicit
POST /api/nodes/{id}/rejoin. (This was found and fixed during end-to-end verification.) - Streaming everywhere.
BodyPublishers.ofFileup,BodyHandlers.ofInputStreamdown,StreamingResponseBodyout — memory use is independent of file size. - Two checksums for two jobs. MD5 for ring placement (fast, uniform, portable, not a security decision); SHA-256 for integrity, verified on write, copy, download and demand.
| Choice | Gained | Given up |
|---|---|---|
| Central coordinator | Consistent placement, disposable nodes | It is a bandwidth funnel and a control-plane dependency |
| Consistent hashing vs a placement table | No per-file placement state, minimal movement, O(log V) lookups | Cannot pin a file to a node; rack/zone awareness needs extra work |
| PostgreSQL for metadata | Transactions, indexes, real queries, easy ops | Single-primary scaling limit; the deliberate SPOF |
| Async replication past the primary | Fast uploads that survive a dead replica | A window below RF, surfaced as DEGRADED |
DEGRADED when the cluster is smaller than RF |
Durability risk is never disguised | A 2-node cluster reports everything DEGRADED |
| Whole-file blobs | Simple, streaming, easy to reason about | No resumable or parallel transfer for huge files |
| Redis as pure optimisation | Losing it costs latency, not availability | Weaker cache/lock semantics than if authoritative |
| Nodes with no cluster knowledge | Nothing to gossip, trivially replaceable | Every decision needs a coordinator round trip |
- Chunked large files — split into fixed-size chunks placed independently on the same ring: resumable uploads, parallel reads, per-chunk repair.
- Redirect the data plane — hand clients a short-lived signed URL to the node so bytes bypass the coordinator entirely.
- Rack / availability-zone awareness — make replica selection skip nodes sharing a failure domain.
- Ring-position index on
filesso a node join can query the affected arcs instead of scanning metadata. - Read-your-writes across coordinators — currently guaranteed per instance; add cache invalidation via Redis pub/sub for a multi-instance deployment.
- Anti-entropy scrubber — background checksum sweep instead of verification only on read/demand.
- Weighted nodes — scale virtual node counts by disk capacity so heterogeneous hardware fills evenly.
- Authentication — API keys or OAuth2 on the public API, mTLS on the internal one.
- Bandwidth-throttled rebalancing so a large migration cannot starve live traffic.
- Erasure coding as an alternative to 3× replication for cold data (1.5× overhead instead of 3×).
Answers grounded in this implementation, with the file to open if someone wants to see it.
Because placement has to survive changes in cluster size without moving the whole dataset. With consistent hashing, adding the 4th node to a 3-node cluster re-maps about 1/(n+1) — a quarter — of the keys, and only to the new node; nothing else is disturbed. It also needs no placement table: given a file id and the current membership, any coordinator computes the same answer in O(log V), so placement is stateless and identical everywhere. ConsistentHashRing implements it with a NavigableMap<Long, String> of 100 virtual positions per node, and ConsistentHashRingTest measures the movement fraction rather than assuming it.
Because modulo binds ownership to the count of nodes. A key keeps its owner only when hash % 3 == hash % 4, which is roughly 1 key in 4 — so adding one machine moves ~75 % of the data. Practically: the cluster saturates its own network for the whole migration, and until it finishes every read has to guess whether the key has already moved. My test suite runs both schemes over 10 000 keys and asserts modulo moves >70 % while the ring moves less than half of that, with every moved key going to the new node.
Two independent things, and that separation is the point.
Immediately, on the read path: a download tries the primary, gets a connection failure, and moves to the next active copy — the response header X-Served-By-Node shows which node actually answered. No waiting for any detector.
Within a few seconds, in the background: NodeHealthService polls /actuator/health every 10 s in parallel; after 2 consecutive failures NodeRegistryService flips the node to UNHEALTHY, removes its 100 positions from the ring, and publishes an event. RebalancingService then queries file_replicas by node_id (indexed) — so only that node's files are visited — and for each one marks the dead copy FAILED, copies the missing replica from a survivor to the new owner, promotes a healthy primary, and sets the file ACTIVE or DEGRADED. Uploads simply stop selecting the node, because the ring no longer contains it.
The threshold matters: eviction causes data movement, so one missed probe (a GC pause) must not trigger it.
I don't guarantee it in the absolute sense — I can state the conditions. A file is readable while at least one holder is reachable, and there are RF = 3 holders on distinct machines. The read path tries them in preference order and fails over on connection failure, HTTP 404 or a checksum mismatch. Metadata availability depends on PostgreSQL, which is my single point of failure; that is a deliberate scope decision, and the fix is a replicated primary or a consensus-backed metadata store. There is no "99.9 %" claim in the repo: load-test/availability-test.sh lets you kill a node under load and measure a number with a stated window and request mix, which is the only form such a claim is worth making.
getReplicaNodes(fileId, 3) walks the ring clockwise from the file's position and collects three distinct physical nodes — the primary plus two replicas. The primary write is synchronous; the replica writes fan out in parallel on a bounded pool. If a replica write fails, the upload still succeeds (the data is durable) and returns DEGRADED naming the failed node, and the repair job rebuilds the copy within REPAIR_INTERVAL. Every write is checksum-verified by the receiving node, so a corrupted transfer is rejected rather than stored.
It registers itself (POST /api/nodes/register, an idempotent upsert it repeats every 15 s), gets health-probed, and its 100 virtual positions are added to the ring. A background rebalance then walks the metadata and, for each file, compares the ring's answer with the copies that exist. Only files whose preference list now contains the newcomer are touched: the blob is copied from an existing holder, verified, and only then is the surplus copy on the node that lost ownership deleted. In my verification run with 12 files, adding node-4 moved 5 files, created 5 copies, removed 5, and left 7 files untouched — every file still had exactly 3 verified copies.
Four things, and they compose.
- Create before delete. A surplus copy is removed only after every desired copy is confirmed present.
FileReconcilerTest.keepsTheSurplusCopyWhenARequiredCopyCouldNotBeCreatedfails if that order is ever reversed. - Verify before and after the move. The copy is re-hashed at the source, and the destination re-hashes the body and rejects a mismatch — corruption cannot be propagated and then have the good copy deleted.
- Idempotence instead of checkpoints. Every decision comes from observed state (the ring plus
HEADon the node), so a run interrupted anywhere leaves no half-finished state; the next run just sees less work. - A lease lock. Redis
SET NX PXwith a Lua compare-and-delete release means two coordinators cannot fight over the same file, and a coordinator dying mid-run cannot wedge the cluster — the lease expires.
And when there is genuinely nothing to copy from, the file is marked FAILED and named in unrecoverableFiles in the report rather than being quietly dropped.
ReplicationService.writePrimary iterates the whole preference list, not just its head. If the first node refuses the write or is unreachable, the next node clockwise takes it and that node is recorded as primary_node_id — the client gets a normal 201. Only if every candidate fails do I delete the PENDING metadata row and return 503 STORAGE_UNAVAILABLE, so a failed upload leaves nothing behind. If the coordinator dies mid-upload instead, the PENDING rows survive and the repair job asks the planned nodes what they actually hold, then either adopts the blobs (as ACTIVE/DEGRADED) or discards the metadata.
Metadata is strongly consistent: one row per file in PostgreSQL, ACID, with optimistic locking so concurrent status transitions cannot be lost. Replica placement is eventually consistent: after a failure or a join, copies converge on the ring's answer within a repair interval, and the current state is always visible (ACTIVE vs DEGRADED, per-replica statuses in the metadata response).
Concretely: a 201 means at least one durable checksum-verified copy exists and the body says how many. Reads are read-your-writes — the upload commits its final state before returning and invalidates the cache. There is no cross-file atomicity, no versioning (a blob is immutable; re-uploading creates a new id), and RF is not reached synchronously.
In the order they would actually bite:
- The coordinator is on the data path — every byte flows through it, so its NIC is the ceiling long before its CPU. Fix: signed redirects so clients talk to nodes directly.
- PostgreSQL as a single metadata primary — fine for tens of millions of rows; then read replicas for lookups, then sharding
filesby id. It is also the component whose loss stops the service. - Full metadata scan on node join — O(files) reads even though only O(affected) bytes move. Fix: store each file's ring position and query the affected arcs.
- Health-check fan-out — one probe per node per interval is fine for tens of nodes, awkward at thousands. Fix: nodes push heartbeats.
- Whole-file transfers — one long unresumable request for a very large file.
Millions of files is mostly a metadata problem, and the blob path already scales with node count. In order: put the metadata reads behind the Redis cache that is already there (it is a read-through cache today) and add read replicas; replace the join-time full scan with a ring-position column and range queries; batch the reconcile work and throttle its bandwidth so migrations do not starve live traffic; run several stateless coordinators behind a load balancer with the Redis lease keeping background work single-writer. The storage layer itself needs nothing new — the directory fan-out already keeps per-directory entry counts sane, and each node's share stays at 1/n because of the virtual nodes.
If it grew beyond a single metadata primary, I would shard files by file_id — the same hash the ring uses — so a shard maps cleanly onto a ring range.
Chunk them client-side or at the coordinator: split into fixed-size chunks (say 8 MB), treat each chunk as its own key on the same ring, and store an ordered chunk list in the file's metadata. That buys resumable uploads (retry one chunk, not 100 GB), parallel reads across nodes, parallel and finer-grained repair, and it caps the memory and time cost of any single request. It costs a chunk table and a reassembly step on read. Today a file is one blob and one request, which is honest for the size range this project targets — and the streaming implementation means even a large single file never loads into memory; it is the unresumability that is the real limit.
The compose topology maps over almost directly, and the design choices were made with that in mind:
- Storage nodes → StatefulSet with a
volumeClaimTemplate. Stable pod identity gives stable node ids, which is exactly what the ring needs:STORAGE_NODE_IDfrom the pod's ordinal name,STORAGE_ADVERTISED_HOSTfrom the headless-service DNS name (storage-node-1.storage.svc.cluster.local). Because a node re-registers itself on a timer, a rescheduled pod rejoins with no operator action and its PVC data is re-adopted rather than re-copied. - Coordinator → Deployment with several replicas behind a Service; it is stateless, and the Redis lease already prevents two instances from rebalancing at once.
- Probes:
/actuator/health/livenessand/actuator/health/readinessare already enabled on the nodes; the coordinator's/actuator/health/essentialgroup (process + database, excluding Redis) is exactly the right readiness probe. - PostgreSQL and Redis: managed services, or operators (CloudNativePG, Redis Operator) rather than hand-rolled StatefulSets.
- Config/secrets: everything is already environment-driven, so a ConfigMap plus a Secret covers it with no code change.
- Scaling storage:
kubectl scale statefulset storage-node --replicas=6— the new pods register, the ring absorbs them, and rebalancing moves~1/(n+1)of the data. - Draining a node:
DELETE /api/nodes/{id}in apreStophook, so the pod leaves the ring and its files are copied away before termination — which is precisely why deregistration had to be terminal rather than something the node's own heartbeat could undo.
What I would not do is put the coordinator's placement state in Kubernetes itself; the ring belongs in the application, rebuilt from the database, so the system behaves the same whether it runs in compose or a cluster.