Pilot weekly org review cadence and prompt tuning #412

Activity

1. ss-o

   added

   type:maintenanceNon-feature maintenance, cleanup, or org work.Non-feature maintenance, cleanup, or org work.

   area:metaOrganization-wide policy, templates, or meta-repo work.Organization-wide policy, templates, or meta-repo work.

   meta:org-trackedIndicates this issue has cross-repository tracking implications (synced to Linear).Indicates this issue has cross-repository tracking implications (synced to Linear).

   on May 20, 2026
2. ss-o

   added this to Z-shell Deliveryon May 20, 2026
3. ss-o

   moved this to Triage in Z-shell Deliveryon May 20, 2026
4. ss-o commented on Jun 1, 2026

   

   ss-o

   on Jun 1, 2026 – with ChatGPT Codex Connector

   Last edited by ss-o

   MemberAuthor

   More actions

   Agent handoff

   Status: Blocked
   Repository: z-shell/.github
   Tracker/Issue: Weekly review pilot #412

   Current state

   The review cadence remains manual and draft-only. Cycle 2 is complete; the Cycle 3 retrospective evidence and current-state review records the retained July 25 to August 1 evidence and its limits. The original live-review and Linear requirements remain in force, so the pilot remains open.

   Verification

   Public issue enumeration and independent search agree on 23 new issues. Complete retained scheduled-run queries across 15 pilot repositories agree on 43 runs: 29 successful, 13 failed and 1 cancelled. Current run and job checks distinguish superseded lock, metrics and buildx failures from remaining scheduled failures. The report identifies unassessed repositories and unavailable historical and Linear evidence.

   Blockers

   A third live weekly review and live Linear evidence and synchronization are still required. The maintainer owns availability of the mirror evidence and its synchronization. The retrospective does not replace those acceptance criteria.

   Next steps

   1. Obtain current evidence from the linked Linear mirror without sharing credentials in issue text, then select a non-overlapping weekly window for the live review.
   2. Complete the live review, reconcile its evidence with the mirror, and confirm the pilot acceptance criteria before proposing closure.
   3. Keep the zunit scheduled-test and src consumer-evidence failure diagnosis proposals separate from this pilot. Their source runs and scope limits are linked in the report.
5. ss-o commented on Jul 19, 2026

   

   ss-o

   on Jul 19, 2026 – with ChatGPT Codex Connector

   MemberAuthor

   More actions

   Cycle 2 execution kit — prepared 2026-07-19; do not execute before 2026-07-25

   This is preparation only. It does not count as Cycle 2 and authorizes no labels, comments, closures, merges, follow-up issues, or prompt changes.

   Window and inventory

   • At execution time, record an exact half-open UTC window and prove it does not overlap Cycle 1. Default: [2026-07-18T00:00:00Z, 2026-07-25T00:00:00Z); adjust the start if Cycle 1's recorded end was inclusive.
   • Refresh the current 11-repository inventory from the private workspace catalog; record additions/removals rather than silently reusing the Cycle 1 list.
   • Record execution start/end UTC and total operator time.

   Read-only query sequence

   1. Open strategic records: refresh relevant GitHub issues and Linear mirrors; record status, owner, update time, links, missing mirrors, duplicates, and contradictory states.
   2. Human PR review queue: query PRs separately from issues. Exclude bots and drafts from “Needs review.” Record human, non-draft PRs waiting more than 3 days.
   3. Disposition queue: separately list human, non-draft PRs older than 30 days whose likely next action is close/rebase/supersede rather than review.
   4. Bot queue: summarize dependency-bot volume/patterns separately; do not mix it into the human queue.
   5. Workflow failures: gather runs in the window, then group identical scheduled failures by repository + workflow + failing job/signature. Record occurrence count and latest run.
   6. Current-state refresh: before calling any historical failure urgent, inspect the current PR head or latest relevant default-branch/scheduled run and record whether the failure is active, superseded, or inconclusive.
   7. Cross-repository patterns: require at least two concrete sources and one smallest useful draft action.
   8. Security-sensitive work: publish only sanitized tracker pointers; keep alert/advisory details in the access-controlled owning record.

   Evidence fields for every reported item

   Category:
   Repository / source URL:
   Window event and UTC timestamp:
   Current state checked at UTC:
   Exact PR head or workflow run:
   Why it matters now:
   Smallest draft next action:
   Owner / owning tracker:
   GitHub ↔ Linear mirror state:
   Confidence:
   Evidence gap or access limitation:
   

   Cycle 1 hypotheses to score

   For each, record supported, not supported, or inconclusive, plus examples:

   • H1: excluding bots/drafts improves the waiting-for-review queue.
   • H2: collapsing repeated scheduled failures removes noise without hiding scope.
   • H3: current-state refresh prevents superseded failures being called urgent.
   • H4: separating “Needs review” from “Needs disposition” improves actionability.

   Do not tune the durable prompt merely because a hypothesis seems plausible. After Cycle 2, propose a before/after prompt change only where the same precision, routing, or safety failure is evidenced in both cycles; otherwise carry the hypothesis into Cycle 3 unchanged.

   Result template

   1. Urgent
   2. Needs review
   3. Needs disposition
   4. Cross-repository patterns
   5. Suggested follow-ups — draft only
   6. Mirror gaps / duplicates
   7. Evidence and access gaps
   8. Signal, noise, and hypothesis scorecard
   9. Time cost
   10. Maintainer decisions requested

   Completion evidence for Cycle 2 is the finished report synchronized to #412 and ZSH-8 on or after the date gate, with exact window and current-state timestamps. Status remains open/In Progress afterward; Cycle 3 is still required no earlier than 2026-08-01.
6. ss-o commented on Jul 27, 2026

   

   ss-o

   on Jul 27, 2026 – with ChatGPT Codex Connector

   MemberAuthor

   More actions

   Cycle 2 weekly organization review

   Draft-only review completed for the half-open UTC window
   [2026-07-18T00:00:00Z, 2026-07-25T00:00:00Z).

   Scope and coverage

   • Raw live organization enumeration: 93 repositories, including 5 forks.
   • Active non-fork baseline: 88 repositories, comprising 86 public and 2
     private repositories.
   • Pilot catalog coverage: 11 of 93 raw repositories and 11 of 88 active
     non-fork repositories.
   • The scheduled-workflow coordination issue #485
     uses a distinct 86-public-repository audit denominator. It excludes the 5
     forks and 2 private repositories.
   • This pilot does not make organization-wide absence claims from the
     11-repository subset.

   Public findings

   • 35 issues were opened in the exact window. At refresh, 22 were closed and 13
     remained open.
   • Two public-to-tracker state contradictions need maintainer reconciliation:
     the governance ADR issue #454
     remains open, while
     the completed zsh-fancy-completions CI issue #46
     still has a stale external tracker state.
   • The dependency-automation rollout remains gated by
     the organization Renovate rollout issue #452.
     Hosted processing and repository security settings were not visible in this
     review, so no safety gate should be removed.

   Scheduled workflow summary

   Nine scheduled failures collapsed to three active groups:

   1. Zi's daily lock workflow failed seven times and failed again after the
      window. It is unattended, write-capable, and still uses mutable
      dessant/lock-threads@v4. The primary decision under
      the scheduled-workflow coordination issue #485
      is remove versus retain. If retained, pin the action to a full immutable SHA
      with a readable version comment.
   2. zunit's scheduled six-version matrix failed once because its pinned buildx
      action could not be resolved. All six jobs stopped before tests, and no
      later run exists.
   3. Wiki's scheduled full audit failed once because the scheduled-only input is
      incompatible with the reusable workflow invocation. Later green push and
      pull-request runs do not exercise that input, so they do not supersede the
      scheduled failure.

   Pull-request queues

   • 10 open pull requests total.
   • 1 human non-draft pull request waiting more than 3 days:
     Zi standard-modelines PR #355.
   • 0 human non-draft pull requests older than 30 days.
   • 1 human draft and 8 bot pull requests, kept outside the human review queue.

   Cycle 2 hypothesis scores

   • H1, excluding bots and drafts improves the waiting-for-review queue:
     supported.
   • H2, grouping repeated scheduled failures reduces noise without hiding scope:
     supported.
   • H3, current-state refresh prevents superseded failures being called urgent:
     inconclusive because all three scheduled paths remain active.
   • H4, separating review from disposition improves actionability:
     inconclusive because no human non-draft pull request exceeded 30 days.

   Time and evidence gaps

   • Operator time: 19 minutes.
   • Largest coverage gap: 77 active non-fork repositories were outside the pilot
     catalog.
   • Hosted Renovate processing and repository security settings were not
     directly observable.
   • Wiki and zunit have no later scheduled rerun proving resolution.
   • Full-text tracker search cannot prove that a differently titled mirror does
     not exist.

   Cycle 3 gate

   Do not execute Cycle 3 before 2026-08-01. Its exact non-overlapping window is
   [2026-07-25T00:00:00Z, 2026-08-01T00:00:00Z). Keep the prompt and H1 through
   H4 unchanged until three real cycles can be compared.
7. ss-o

   moved this from Triage to Todo in Z-shell Deliveryon Oct 1, 2026
8. ss-o commented on Oct 1, 2026

   

   ss-o

   on Oct 1, 2026

   MemberAuthor

   More actions

   Project 28 triage pass (2026-10-02): Status Triage -> Todo.

   • Premise confirmed on main at b849468: runbooks/org-review.md exists at b849468. Issue records Cycle 1 (body) and Cycle 2 (comment 2026-07-27); no Cycle 3 report or manual/scheduled/retired decision has been posted since, so acceptance criteria 1 and 5 remain open. No merged PR references Pilot weekly org review cadence and prompt tuning #412.
   • Disposition: implement. Next action: implement against the acceptance criteria above; the work is ready to start.
9. ss-o

   marked this as blocking on Oct 1, 2026

   • ops: rationalize scheduled workflows and recurring organization maintenance #485
10. ss-o commented on Oct 3, 2026

    

    ss-o

    on Oct 3, 2026

    MemberAuthor

    More actions

    Cycle 2 is already recorded in the July 27 report. The remaining evidence is Cycle 3 for the specified window [2026-07-25T00:00:00Z, 2026-08-01T00:00:00Z), its comparison with the first two cycles, and the manual, scheduled or retired decision.

    Next action: collect the historical GitHub evidence, distinguish it from current repository observations, and identify any unavailable historical or Linear evidence before deciding whether the pilot meets its acceptance criteria. The existing weekly date gate has elapsed; evidence and the final cadence decision remain outstanding.
11. ss-o commented on Oct 3, 2026

    

    ss-o

    on Oct 3, 2026

    MemberAuthor

    More actions

    Cycle 3 retrospective evidence and current-state review

    The specified window is [2026-07-25T00:00:00Z, 2026-08-01T00:00:00Z). Retained GitHub records were inspected on October 3. This is a retrospective reconstruction, not evidence that a live review occurred during that week. Current states below are October observations, not reconstructed August 1 states.

    Window evidence

    Public issue enumeration and an independent search agree on 23 new issues: .github 7, zsh-lint 6, zsh-eza 3, wiki 3, and one each in zsh, zredis, zd and zunit. Today, 21 are closed; #478 and #485 remain open.

    The current pilot catalog contains 15 public active repositories, up from the 11 reported in Cycle 2. Complete retained-run queries for these 15 repositories returned 43 scheduled runs: 29 successful, 13 failed and 1 cancelled. The 13 failed runs form five repository/workflow groups:

    • Zi lock: seven failures; latest window run. The workflow is now deleted, so these are not current repair work.
    • zunit matrix: one failed run with six failed setup jobs; window run. The buildx pin was corrected by PR #29.
    • Organization metrics: two failures in metrics.yml and two in activity_metrics.yml. The former has a successful current scheduled run; the latter is deleted.
    • Wiki Trunk: one failed run; window run. Current scheduled failures remain tracked by wiki #857.

    The cancelled zd matrix run is separate from those failures; its latest scheduled run succeeded.

    Current attention and draft follow-ups

    The latest zunit scheduled matrix fails all six jobs at “Run tests,” a different signature from the corrected setup failure. The src consumer-evidence audit fails at “Audit published consumer evidence.” Neither has an open owning issue in the inspected repository issue inventory. Draft next action: establish dedicated diagnosis owners, inspect the failed steps and reproduce on the current target before proposing fixes. These observations do not establish root causes or authorize remediation.

    The current public PR inventory contains 27 open, non-draft PRs: 25 bot proposals and two human contributions. Both human PRs are older than 30 days: zsh-string-lib #5 and H-S-MW #14. Keep their bounded review and disposition queue separate from routine bot updates; age alone is not a closure reason. Historical PR queue state was not reconstructed.

    Coverage and review-readiness limits

    Live enumeration found 91 public repositories, including two archived repositories excluded from active coverage. Scheduled-run evidence covers 15 of the 89 public active repositories; the other 74 are unassessed here. The expanded catalog means raw cycle totals are not like-for-like measures of organization reliability.

    The current policy coverage report at daee4d0 observes 15 declared consumers. Fourteen have current routing blocks and approved review-skill pins; Zi main has older routing and pin metadata, while next already carries the approved pin and caller. Treat integration and stable branches separately rather than creating a duplicate adoption fix. Semantic suitability, hosted context configuration and runtime invocation remain unverified or unassessed; static presence is not a clean health verdict. Repositories outside the declared consumer inventory are not covered by those file checks.

    Signal, noise and acceptance

    H1, separating bots and drafts, remains supported: bot proposals dominate the current queue. H2, grouping repeated failures, remains supported: seven Zi failures describe one obsolete workflow problem. H3, refreshing current state, is supported: historical lock, metrics and buildx findings would otherwise send work toward superseded causes. H4, separating review from disposition, is supported as a queue distinction by the two old human contributions, not as evidence that either should close.

    These observations support retaining the existing safeguards. They do not isolate prompt quality from the expanded catalog and delayed observation, so no prompt tuning is justified by the totals alone.

    Linear evidence and synchronization are unavailable. Deleted run history cannot be reconstructed from retained-run counts. Historical PR state, private security evidence and organization Actions consumption were not assessed. The original live-review and Linear acceptance criteria remain unmet unless explicitly amended. Recommendation: keep the review manual and draft-only; resolve acceptance and mirror requirements before closure or scheduling.

    Time cost: evidence collection and report preparation spanned approximately 18 minutes on October 3, from 09:04 to 09:22 UTC. This is elapsed time, not a measure of human review effort or organization Actions consumption.
12. ss-o

    added

    status:blockedCannot proceed until an external dependency or decision changes.Cannot proceed until an external dependency or decision changes.

    on Oct 3, 2026