Repository navigation
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A tree's writes now run as an ordered list in one short transaction at the end, so no transaction stays open across a network call, and handlers must not read their own tree's writes. CompleteMultipartUpload runs the accept as its own task before Complete reads the parts. A failed final commit is resumed instead of failing the task, and handlers fail before their first effect outside Postgres. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
24 tasks
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 2, 2026
… commit until their deadline Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'s record Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
nice, the locks table earns the RFC on its own one gap from the exit side:
in your idiom I think it's a reschedule rather than a refusal, hilt side is fil-forge/hilt#82 if it's useful to read against |
…d step departures DeleteBucket closes the bucket, aborts its multipart uploads, then deletes it, as AWS deletes a bucket with its uploads; CreateMultipartUpload admits its upload through the bucket's queue, so no lock is needed. A completed upload is reaped by its own action. Complete accepts its parts as a child. A within-space copy is a Put, a ship signs with the bucket's captured authority, and the delete's signed request is never stored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…bucket's queue The upload's accept now only queues a queue-space-shard.accept per parked shard, which concludes it under the subject its releases share, so a shard's conclude and its release never overlap; Complete waits for those accepts. Tagging, CORS and default encryption change through queue-bucket actions that act only on the bucket still in that space. The delete removes the bucket's name-keyed log before freeing the name. Handlers can submit many tasks in one statement per queue. Batched concludes are deferred. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
References are counted rather than stored: a release counts a shard's claims, pending references and open parts in one statement, and whatever drops a reference queues the release, so queue-space-shard.remove goes. Releases from a dropped claim wait the reader grace; those from teardowns, replaced parts and expiries are due now. The add's three modes, the release's checks and its held-subject retries, and the authority a task carries and the order it is read in are described as built. Reads do not fill the cache, and a write's copy enters it through a sync queue-shard.cache. DeleteBucket's steps, the multipart tasks that take their content from the waiting request, failed tasks, retention, running delayed work now, lease renewal, the metrics, and the cleanup authority Hilt now delegates are brought up to date. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The upload's accept makes no network calls since each shard is concluded under its own subject, so it no longer needs a queue apart from the upload's other changes: it becomes queue-multipart-upload.accept, and Complete queues its listed parts' shard accepts itself and waits for them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical lifecycle and concurrency flaws can delete still-referenced blobs or acknowledge unreadable uploads, risking data loss.
9 open issues
What changed in this PR
Adds an experimental RFC proposing durable Postgres task queues to replace Ingot’s assorted concurrency controls and background cleanup mechanisms. This PR documents the design; it does not implement the runner.
Changes:
- Defines task lifecycle, subject leases, transaction boundaries, retries, and queue ordering.
- Maps S3 requests onto bucket, multipart-upload, shard, and registration tasks.
- Proposes removing the local spool and documents operational requirements, cleanup authority, and future trade-offs.
| File | Description |
|---|---|
rfcs/2026-10-ingot-task-runner.md |
Introduces the task-runner design, S3 task flows, queue contracts, and operational considerations. |
Review effort: Balanced
|
|
||
| ### Retries | ||
|
|
||
| An action registers with a retry policy for when it runs async: a backoff (first delay, maximum delay, multiplier) and an optional attempt limit. A sync task is never retried: it fails its caller, and the S3 client retries the request. A handler can mark an error as permanent (an abort refused because the blob was accepted, an expired proof) to stop the retries. A task that fails permanently or runs out of attempts is recorded as failed and logged, and monitoring shows it by queue and action. By default the due tasks behind a task waiting to retry run during its backoff. A policy can hold the subject instead, so the tasks behind wait for the retry and the subject's order survives a failure. That holds up the subject's sync tasks too, so only actions whose order matters more than that use it. A task waiting to retry can still be cancelled, since its failed attempt changed nothing outside Postgres ([rule 8](#handlers-must)). |
| ### The runner guarantees | ||
|
|
||
| 1. **One consumer per subject.** At most one task per (queue, subject) runs at a time. | ||
| * A lease row per (queue, subject) in Postgres enforces it, so it holds across processes. The lease records its owner and expiry, and is renewed three times per lease period (30 seconds by default) while the task runs. A lease left unrenewed past its period is taken over by the next consumer to ask, which runs the interrupted task again; the old holder's commit then fails and writes nothing. |
|
|
||
| * the new version's sequence number and ID: a minted ID with versioning Enabled, the null version otherwise | ||
| * what it supersedes: nothing with versioning Enabled, otherwise the key's null version, whose references are removed and whose upload is retracted | ||
| * the write's preconditions (If-Match, If-None-Match), checked against the version it would supersede |
| * **part** (UploadPart): records the shard as parked, with the provider but not yet accepted, for `queue-space-shard.accept` to conclude later; | ||
| * **copy** (a copy within the bucket): sends nothing, and fails if the space no longer holds the shard. | ||
|
|
||
| A space that holds the shard already, parked or accepted, keeps it, and a streamed write's own allocation is given up. The allocation's `/http/put` invocation carries derived signer keys, so it reaches the add through its caller's context and is never stored. |
|
|
||
| Spawns: NONE | ||
|
|
||
| Retries: 5 attempts, backing off from 5 seconds, doubling. A failed conclude records nothing ([rule 2](#the-runner-guarantees)); concluding a shard again counts as a repeat. |
| type ExpireTask struct { | ||
| Task | ||
| Space did.DID | ||
| Digest multihash.Multihash |
|
|
||
| Removes the shard from the space on the network: `/blob/remove` if it was accepted, `/blob/abort` if it was parked and never accepted, and `/blob/remove` anyway if the provider answers that the space accepted it. A shard with neither a location nor a park may still be accepted, if a conclude was accepted and its location never recorded, so the release sends `/blob/remove` for it too, which costs one round trip and closes that leak. It deletes the shard's encryption params first (crypto-shred), so a shard nothing references becomes unreadable at once, and then its rows, with its commit. It evicts the cached copy unless another space's upload intent names the digest. | ||
|
|
||
| Releases are queued by whatever drops a reference: a commit that drops a version's last claim, after the reader grace period and as a replacement, so a reader of the newer version gets its full grace; and a teardown, a replaced part or an expiry, due now. It does nothing when the space still references the shard (a write reused it, so no cancellation is needed) or when the shard's upload intent is gone, which only a release's commit removes (an earlier release has run). |
|
|
||
| * (async) `queue-shard.evict` - with no `NotBefore` | ||
|
|
||
| Retries: no attempt limit, backing off from 1 second, doubling, up to 5 minutes, holding the subject: a failed attempt may already have removed the blob, so an add of the same shard must wait for the retry and then find it gone rather than reuse it. Dropping it would leave a blob nothing references on the network. |
|
|
||
| Makes a written shard durable for the space and takes the write's pending reference to it. It runs in one of three modes: | ||
|
|
||
| * **streamed** (PutObject): concludes the shard's allocation, sending `/ucan/conclude` with the `/http/put` receipt, which triggers `/blob/accept`, and records the location; |
alanshaw
added a commit
to fil-forge/ingot
that referenced
this pull request
Nov 10, 2026
The first step of #328: the task runner from the [RFC](fil-one/RFC#40), as a standalone package. Nothing in the server uses it yet; the following pull requests move Ingot's areas onto it one at a time. Tasks live in Postgres, one table per queue, and at most one task per queue and subject runs at a time, held by a lease row so it holds across processes. A subject's due tasks run oldest first, and a delayed task never holds up the ones behind it. A task can run sync children inline in later queues and spawn async tasks anywhere. Handlers add their writes to a list, and once every handler in the tree has run, the list, the spawned tasks and the task's completion commit in one short transaction, so no transaction stays open across a network call. A task is recorded before it runs, so a crash leaves it to be resumed once its lease expires; a commit that keeps failing is left the same way. Only an error a handler returns marks a task failed, and a failed child fails its whole tree. Two trees whose sync children take the same subjects in opposite orders could wait on each other forever. The younger one gives way: it releases its leases and starts again with its original age, so it eventually goes first. The RFC now describes this. The package imports nothing else from Ingot, which a test checks, so it can be released as a library later. A test helper runs a task tree twice, once with its writes held to the end and once with them applied straight away, to catch a handler that reads its own tree's writes. Not in this pull request: replacing and cancelling pending tasks, batching and retries (next), then placement and metrics. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
alanshaw
added a commit
to fil-forge/ingot
that referenced
this pull request
Nov 10, 2026
The second step of #328, stacked on #330: the scheduling features the [task runner RFC](fil-one/RFC#40) describes for async tasks. Like #330, nothing in the server uses them yet. - **Replacing.** A delayed task can be submitted as a replacement. If one with the same subject and action is waiting and has not started, whichever runs later is kept and the other is dropped. Ingot will use this to push a cached file's eviction back each time it is read. A replacement that has already started is left alone, and the new one is added beside it. - **Cancelling.** A task that has not started can be cancelled, from a handler or in a caller's own transaction, so the cancel commits with the writes that make it necessary. Ingot will use this to cancel an upload's cleanup once the write that claims its shards commits. - **Batching.** An action can take a run of consecutive tasks on one subject in a single go, up to a limit, stopping at the first task of a different action so the order still holds. Ingot will use this to register a space's queued objects with the network in one request. - **Retries.** An action can register a backoff and an attempt limit for its async tasks, and a handler can mark an error as one no retry will fix. Sync tasks are never retried: their error goes to the caller. The package also gets a README with an example of every feature, including the earlier ones. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
alanshaw
added a commit
to fil-forge/ingot
that referenced
this pull request
Nov 10, 2026
…#334) The third and last runner step of #328, stacked on #331: placing tasks on one instance, and metrics, as the [task runner RFC](fil-one/RFC#40) describes. Nothing in the server uses the runner yet; the next pull requests start moving Ingot onto it. **Placement.** A task can be tied to one instance, for work on that instance's local state such as a cached file. Only that instance runs it, and other instances also leave the tasks queued behind it on the same subject until it has run, so the subject's order holds. A sync task tied to another instance is refused, since it can only run in its caller. Ingot runs as one instance today, so this changes nothing yet; it is there so a multi-instance design can place work without changing the runner again. **Metrics.** The runner reports through OpenTelemetry, labelled by queue and action, so the deployment's pool sizes can be tuned from real numbers: how many tasks are ready and how many are delayed, how long the oldest ready task has waited, how long callers wait for a subject, how long tasks take by outcome, how busy each queue's workers are, and how often a lease is taken over from a consumer that stopped renewing it. The queue counts are read from Postgres each time the exporter collects. With no meter provider configured, every instrument does nothing. The README gains a section on each. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📖 Preview
Proposes replacing Ingot's assorted locks and background sweepers with one mechanism: durable task queues in Postgres, where each queue runs at most one task at a time for a given thing, such as a bucket, a blob or a multipart upload. Each S3 request becomes a small tree of tasks, and anything that has to happen later, such as releasing a deleted blob or aborting an abandoned upload, is a task scheduled for that time.
Today Ingot keeps concurrent writes consistent with 10 separate mechanisms of 7 kinds. With the runner, one locking mechanism remains. The RFC also drops the local spool, so blobs are kept on disk only as a cache.
The section "How S3 requests use the queues" is the best place to start reading.
One piece depends on Hilt: delayed cleanup needs delegations that outlive the caller's access key, proposed in fil-forge/hilt#113.
🤖 Generated with Claude Code