Skip to content

Acquire: persist mirror history to skip re-synchronizing already-mirrored (and promoted) digests #157

Closed
6 of 6 issues completed
Closed
Acquire: persist mirror history to skip re-synchronizing already-mirrored (and promoted) digests#157
6 of 6 issues completed
Labels
acquireRelated to the Acquire stageagent-managedCreated/managed by My Feature EngineerfeatureNew feature or requestfeature:mirror-historyMirror-history capabilitytype:featureFeature tracking issue (epic)

Description

@toddysm
toddysm
opened on Jul 31, 2026

Problem

The mirror action (mirror-image / _mirror-image.yml) decides idempotency by comparing the source manifest digest with the destination (quarantine/<image>:<tag>) digest. It copies whenever they differ or the destination is absent.

This is stateless. Once an image is promoted out of quarantine, the promotion workflow deletes the tag from quarantine — and because GHCR deletes the whole package when the last tagged version is removed, the quarantine/<image> repo disappears. On the next scheduled mirror run the destination is absent, so the digest comparison mismatches and the already-processed, already-promoted digest is re-synchronized into quarantine. The image then loops back through the pipeline.

Goal

Give the mirror a durable memory of what it has already synchronized so a digest that was mirrored once is not mirrored again, even after it has been promoted and deleted from quarantine.

Proposed approach

Store a mirror-history OCI artifact in each synchronized repo (ghcr.io/<owner>/quarantine/<image>), under a reserved tag mirror-history. It is a small, append-only JSON log (one entry per synchronized source digest, keyed by source tag) wrapped as an OCI artifact. Because it is a separate tag, it survives image-tag deletion during promotion — and, as a bonus, keeps the quarantine package alive so promotion no longer hits the "cannot delete the last tagged version" edge case.

The mirror flow gains a step: before copying, read the history artifact; if the resolved source digest for that source tag is already recorded, skip the copy (unless force). After a successful copy (or a force run), append the digest to the history and push the updated artifact.

Decisions (agreed)

  • Location: colocated in quarantine/<image>, reserved tag mirror-history.
  • History key: source manifest digest, scoped per source tag.
  • force: bypasses the history check and still records the digest.
  • Retention: append-only, unbounded.

Scope

Design first. A design doc will be added under docs/architecture/acquire/, reviewed, then broken into child work items before implementation.

Activity

  1. toddysm
    added
    featureNew feature or request
    acquireRelated to the Acquire stage
    on Jul 31, 2026
  2. toddysm
    added 2 commits that reference this issue on Jul 31, 2026

    docs(acquire): resolve O2/O3/O4, sharpen O1 dashboard question

    ed02504

    docs(acquire): resolve O1 by surfacing synchronized history in the da…

    86699cc

  3. toddysm commented on Jul 31, 2026

    @toddysm
    toddysm
    on Jul 31, 2026
    OwnerAuthor

    Design updated: O1 resolved — instead of hiding the persisted :mirror-history package, the CSSC Dashboard's Acquisition view will surface what each repo has already synchronized.

    • packages-service gains a capability + endpoint to read and parse the :mirror-history artifact (registry blob) for a quarantine/<image> repo.
    • The reserved mirror-history tag is excluded when deciding whether an image is actually present in quarantine (so a history-only, already-promoted package is not shown as awaiting promotion).
    • Each Acquisition card shows a per-repo Synchronized list (source tag/digest, timestamp, run link).

    All four design questions (O1–O4) are now resolved. Design doc: docs/architecture/acquire/mirror-history.md (branch feature/mirror-history).

  4. toddysm commented on Jul 31, 2026

    @toddysm
    toddysm
    on Jul 31, 2026
    OwnerAuthor

    Work breakdown

    Dependency order

    #158 ──▶ #159 ──────────────▶ #163
   └───▶ #160 ──▶ #161 ─────▶ #163
#159/#160/#161 ──▶ #162 (docs, last)

    Implementation can start with #158, then #159 and #160 in parallel.

  5. toddysm
    added 7 commits that reference this issue on Jul 31, 2026

    docs(acquire): flesh out mirror-history JSON schema + lock decisions

    f3418aa

    feat(acquire): add mirror-history composite action (check + record)

    922cf3f

    feat(acquire): wire mirror-history check/record into _mirror-image.yml

    507df7d

    feat(run): packages-service reads the mirror-history artifact (+ endp…

    793c9f2

    feat(run): surface per-repo synchronized history in the Acquisition view

    c2988ea

    docs(acquire): document the mirror-history feature

    6d72ff3

    fix(mirror-history): address PR #164 review

    756f267
  6. toddysm
    added
    type:featureFeature tracking issue (epic)
    feature:mirror-historyMirror-history capability
    on Oct 27, 2026

  7. 2 remaining items

  8. toddysm
    added sub-issues on Oct 27, 2026

  9. toddysm commented on Oct 27, 2026

    @toddysm
    toddysm
    on Oct 27, 2026
    OwnerAuthor

    Feature checkpoint: mirror-history validation resumed

    The requester approved completing #163 with hosted CI, Copilot review, scoped disposable registry fixtures, and a separate capability-inventory close-out pull request.

    The existing work breakdown #158–#163 is now linked as native sub-issues. #158–#162 are closed; #163 is in progress under instance 25e59c6f-a570-4253-aa3b-430d640485e0, branch feature/mirror-history/e2e-163-25e59c6f, in a temporary session-owned worktree. There are no implementation commits or unpushed changes yet (base 8240d0b83b2cd315f20d92ce3bc1bc198d328f46).

    Testing mode: hosted, normal Actions enabled. This legacy feature uses the existing approved architecture on main; no new feature scope is introduced.

    This parent stays open until #163 has evidence-backed completion, the documentation is complete, and a separate reviewed inventory close-out PR has merged. Checkpoints will be recorded here and on #163 at each stop/transition.

  10. toddysm commented on Oct 27, 2026

    @toddysm
    toddysm
    on Oct 27, 2026
    OwnerAuthor

    Feature checkpoint: #163 in review

    Implementation PR #248 is open at f6f2c5f5f0436a7413f7093809c9e298a4cc639f, branch feature/mirror-history/e2e-163-25e59c6f, instance 25e59c6f-a570-4253-aa3b-430d640485e0. The owned temporary worktree is clean and fully pushed.

    Testing mode is hosted. Local regression/type/workflow/shell/documentation checks passed; live registry acceptance and complete hosted library/service/CodeQL checks are pending. The separate inventory close-out remains required after #163 merges; this parent is intentionally still open.

  11. toddysm commented on Oct 27, 2026

    @toddysm
    toddysm
    on Oct 27, 2026
    OwnerAuthor

    Feature checkpoint: live validation exposed and fixed a registry-reader defect

    PR #248 is still in review at pushed/tested head 746ebcce405fb06a72c3173460f44b13b61bcc6d, instance 25e59c6f-a570-4253-aa3b-430d640485e0, branch feature/mirror-history/e2e-163-25e59c6f. The owned temporary worktree is clean and fully pushed; testing mode remains hosted.

    The initial live run passed all 162 hosted unit/service/library tests and successfully mirrored the fixture, but revealed that the real GHCR blob reader did not follow CDN redirects. The producer is fixed with credential-safe redirect/error regressions, and the fixture packages from the failed run were verified removed. Four review findings are also fixed with commit-referenced replies and resolved threads.

    Corrected live run https://github.com/toddysm/cssc-framework/actions/runs/40561160460 and fresh review are pending. CI-fix attempts used: 1/3. #163 and this parent remain open; the separate inventory close-out PR is still required.

  12. toddysm commented on Oct 27, 2026

    @toddysm
    toddysm
    on Oct 27, 2026
    OwnerAuthor

    Feature checkpoint: second integration defect fixed

    #163 remains in review in #248 at 364e225446e831788bec30d34dff749469f28a6d; instance 25e59c6f-a570-4253-aa3b-430d640485e0, branch feature/mirror-history/e2e-163-25e59c6f, owned temporary worktree clean and fully pushed. Hosted testing remains selected.

    The real registry reader/dashboard now work. The next live acceptance stage exposed a promotion race: a fallback referrer-index tag was independently promoted/deleted before the image could discover its referrers. The regular promotion caller now excludes those metadata tags, matching the existing SBOM caller. Failed-run fixtures were verified removed.

    Current-head local tests/type/workflow/shell checks pass. Live run https://github.com/toddysm/cssc-framework/actions/runs/40562725339 and fresh review are pending. CI-fix attempts used: 2/3. Parent closure and the separate inventory close-out PR remain gated on full evidence-backed completion.

  13. toddysm commented on Oct 27, 2026

    @toddysm
    toddysm
    on Oct 27, 2026
    OwnerAuthor

    Feature checkpoint: #163 merged; close-out pending

    #248 merged as 28c7e13f6b660012903e8bc002804fd90cb9303e; #163 is closed. Source/tested head 364e225446e831788bec30d34dff749469f28a6d passed the complete live lifecycle and cleanup: https://github.com/toddysm/cssc-framework/actions/runs/40562725339. All 182 hosted tests and CodeQL passed, all review threads are resolved, and current-head Copilot review has no open findings.

    The preserved evidence proves one initial history entry, unchanged history after promotion and a genuine skip, exactly one forced append, actual rendered dashboard states, and verified removal of both fixture packages. Real blob-redirect and fallback-index cleanup race defects discovered during validation are fixed.

    Merged-state checkpoint before cleanup: instance 25e59c6f-a570-4253-aa3b-430d640485e0, branch feature/mirror-history/e2e-163-25e59c6f, temporary session-owned worktree clean and fully pushed. The fetched main tree matches the tested source tree.

    Testing mode: hosted. Next: safely remove the merged implementation branch/worktree, open the authorized separate capability-inventory close-out PR, and keep this parent open until that PR merges and all child/documentation completion is verified.

  14. toddysm commented on Oct 27, 2026

    @toddysm
    toddysm
    on Oct 27, 2026
    OwnerAuthor

    Close-out checkpoint: implementation cleanup complete

    The merged #163 implementation branch and its temporary worktree have been safely removed, and its ownership claim released. All six native children #158–#163, including documentation #162, are verified closed with none blocked/deferred/failed.

    Instance 25e59c6f-a570-4253-aa3b-430d640485e0 now owns the separate close-out work on branch chore/mirror-history-close-out-25e59c6f in temporary session-owned mirror-history-close-out-worktree. Base commit: 28c7e13f6b660012903e8bc002804fd90cb9303e; clean tracked tree, no close-out commits or unpushed changes yet.

    Testing mode: hosted, with normal required CodeQL and authorized Copilot review. Next: move the validation capability from In progress to Implemented with architecture/user-documentation links and the actual landing month, replace the obsolete pending-validation architecture status with the validated evidence, review/merge this dedicated docs PR, and only then close this parent. No new registry fixture run is needed for these documentation-only changes.

  15. toddysm commented on Oct 27, 2026

    @toddysm
    toddysm
    on Oct 27, 2026
    OwnerAuthor

    Close-out checkpoint: In review

    Separate inventory/documentation PR: #249.

    Instance 25e59c6f-a570-4253-aa3b-430d640485e0, branch chore/mirror-history-close-out-25e59c6f, temporary session-owned close-out worktree. Tested/pushed head ac1a6862f415ab7bbed123268d36d1b1dbbb5dd4; clean tracked state and no unpushed changes. Testing mode: hosted, with normal required CodeQL and authorized Copilot review.

    Only two Markdown files changed. Five local documentation links resolve; the validated capability has exactly one Implemented row with classified architecture/user links and its real landing month; the obsolete pending-validation status is removed and the successful acceptance evidence is recorded. Self-review found no remaining issues.

    All #158–#163 children are closed. Implementation PR #248, the successful 182-test/live workflow and verified fixture cleanup are linked in the close-out PR. This parent remains open until #249 merges and the completion gate is checked again. No additional registry writes are being performed.

  16. toddysm commented on Oct 27, 2026

    @toddysm
    toddysm
    on Oct 27, 2026
    OwnerAuthor

    Merged close-out checkpoint: completion gate satisfied

    The separate inventory close-out PR #249 merged as 3eb0644524804085f4c8cf049967374dbe932dc2. Its source head ac1a6862f415ab7bbed123268d36d1b1dbbb5dd4 passed required CodeQL and completed Copilot review with approval recommended and no open findings. Five documentation links and the inventory row shape/placement were checked.

    Fresh completion reconciliation confirms all six native children #158–#163 are closed, including documentation #162, with none blocked/deferred/failed. The fetched default branch now contains the validation capability in Implemented with architecture/user-documentation links and the actual landing month. The main tree matches the tested close-out tree exactly.

    Implementation PR #248 and the successful 182-test/live acceptance run https://github.com/toddysm/cssc-framework/actions/runs/40562725339 provide the lifecycle evidence; both disposable packages were deleted and verified absent. All implementation review threads received fix-SHA replies before resolution. No protections were bypassed.

    Merged-state checkpoint before cleanup: instance 25e59c6f-a570-4253-aa3b-430d640485e0, branch chore/mirror-history-close-out-25e59c6f, temporary session-owned close-out worktree clean and fully pushed at ac1a6862f415ab7bbed123268d36d1b1dbbb5dd4. Testing mode: hosted.

    The parent completion preconditions are now satisfied. Next: close this issue as completed, remove the verified merged close-out branch/worktree, release ownership, and update the original main checkout. Tracking remains issues-only.

  17. toddysm
    closed this as completedon Oct 27, 2026

  18. toddysm commented on Oct 27, 2026

    @toddysm
    toddysm
    on Oct 27, 2026
    OwnerAuthor

    Completed checkpoint and cleanup

    State: Done. All native children #158–#163 and this parent are closed. The capability inventory is Implemented and the architecture/user documentation is complete.

    Cleanup is complete for both local/remote task branches and both temporary worktrees, after their merged-state checkpoints. All ownership claims are released. The original checkout is clean on main at 3eb0644524804085f4c8cf049967374dbe932dc2, matching origin/main, with no unpushed or unmerged task work. Successful JSON/HTML evidence is retained in the session artifact area as well as the documented run artifacts; temporary tool downloads will be removed.

    Testing mode: hosted. Tracking: issues-only. Instance: 25e59c6f-a570-4253-aa3b-430d640485e0. Next action for this feature: none.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    acquireRelated to the Acquire stageagent-managedCreated/managed by My Feature EngineerfeatureNew feature or requestfeature:mirror-historyMirror-history capabilitytype:featureFeature tracking issue (epic)

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions