Skip to content

fix(ci): serialize app pipelines and keep plan and apply consistent - #186

Merged
Alexanderamiri merged 1 commit into
mainfrom
fix/pipeline-concurrency
Oct 31, 2026
Merged

Alexanderamiri merged 1 commit into
mainfrom
fix/pipeline-concurrency

Conversation

@Alexanderamiri

Copy link
Copy Markdown
Member

Problem

Findings 9, 20 and the ensure-tf-boilerplate gap from the DX audit. In the app pipeline:

  • 9a Only platform-ci.yml set concurrency. Two quick merges ran in parallel. The second expand push could be rejected as non-fast-forward, the applies fought over the state lock, or the second apply hit a stale saved plan.
  • 9b docker-build only skipped on tf-plan == 'failure'. A broken app.yaml fails expand, tf-plan is then skipped, and the image was still built and deployed. On a new repo this showed up as a misleading "ECR repository does not exist".
  • 9c When expand's push to a protected branch was rejected, the job failed with a raw git error. It gave no reason and no fix.
  • 9d Fork PRs failed on checkout: github.head_ref names a branch that only exists in the fork. Runs without an OIDC token failed on the AWS steps.
  • 20 .terraform.lock.hcl is gitignored and the AWS provider is pinned only to ~> 5.0, so tf-apply's init could pick a newer provider than the plan used.
  • ensure-tf-boilerplate ran only in tf-plan. It looked up the team with gh api /repos/.../teams, which GITHUB_TOKEN cannot read, so it fell back to team=unknown. The state key became apps/unknown/<repo>, which no team role can access. Repos that bring their own terraform/ without app.yaml broke.

Change

  • javabin.yml: workflow-level concurrency group javabin-pipeline-<repo>-<ref>. GitHub honours this for called workflows. Runs on the default branch queue (cancel-in-progress: false), and a newer push to a PR cancels the older run. docker-build now needs tf-plan == 'success' whenever the repo has app.yaml or terraform/. tf-apply gets the team tf-plan's broker verified.
  • tf-apply.yml: has its own concurrency group for callers outside javabin.yml, and uses -lock-timeout=5m. Before init it restores .terraform.lock.hcl from the saved plan. Terraform stores the lock file inside the plan, and the existing SHA256 check covers it. It also runs ensure-tf-boilerplate.sh with the same team as the plan.
  • platform-ci.yml apply: restores the lock file the same way.
  • expand-terraform.yml / tf-plan.yml: check out github.head_ref only for same-repo PRs, otherwise github.ref (the PR merge ref for forks). New scripts/check-aws-access.sh checks for ACTIONS_ID_TOKEN_REQUEST_URL. Without it, tf-plan posts a notice, skips the broker, plan, review and comment steps, and still runs init -backend=false, validate and fmt -check. Plan uses -lock-timeout=5m. New output team.
  • commit-generated-tf.sh: a rejected push fails with an ::error that gives the reason and the fix. The cases are a protected branch or ruleset (GH006/GH013), a branch that moved during the run, a read-only token, or anything else. After a push to a PR it posts a notice that checks don't run on the bot commit.
  • ensure-tf-boilerplate.sh: takes TEAM from the broker. If the files it needs to generate are missing and there is no team, it fails instead of writing unknown. Existing states: S3 apps/ has no unknown/ prefix, so no state moves.
  • Docs: docs/reusable-workflows.md (concurrency, fork/Dependabot PRs, push requirements and protected branches, lock file, boilerplate team), docs/ci-workflow.md, CLAUDE.md script table.

Not changed: no GitHub App was added (the repo has none for app pipelines). Image tag and deploy ordering are also untouched, since another PR owns findings 7, 16 and 17.

Verification

  • New scripts/tests/test_pipeline_scripts.py (11 tests) runs the scripts against throwaway git repos:
    • push, up-to-date and fork skip
    • protected-branch rejection, using a pre-receive hook that emits GH006
    • non-fast-forward rejection
    • boilerplate with a team, with no team, and with repo-owned files
    • OIDC present and absent
  • python3 -m unittest discover -s scripts/tests passes (144 tests). python3 -m unittest discover -s terraform/lambda-src/tests passes (219 tests).
  • actionlint on all workflows: no new findings (one SC2129 style note in detect.yml was already there). shellcheck -s sh on the changed scripts is clean.
  • Lock file: checked locally with Terraform 1.16. unzip -l tfplan lists .terraform.lock.hcl. A plan was made with hashicorp/null 3.1.0 and the config then loosened to ~> 3.0. A fresh init without the lock installed 3.3.2. After unzip -o tfplan .terraform.lock.hcl it installed 3.1.0.
  • Live checks: S3 state prefixes are cupcake/ javazoneweb/ jzwatch/ platform-test-app/ testteam/, none unknown/. Renovate PR runs in cupcake and frosting succeed: same-repo bot branches have OIDC. So the skip only triggers when the run really has no token.

🤖 Generated with Claude Code

- javabin.yml: one pipeline per repo/ref (queued on the default branch,
  cancelled on superseded PR pushes); docker-build needs a successful plan
  when the repo has app.yaml or terraform/
- expand/tf-plan: check out the PR merge ref for fork PRs; tf-plan skips
  AWS steps with a notice when the run has no OIDC token
- commit-generated-tf.sh: a rejected push fails with the reason and fix
- tf-apply (and platform-ci apply): restore .terraform.lock.hcl from the
  verified plan so init installs the planned provider versions
- ensure-tf-boilerplate.sh: take the broker-verified team instead of a
  GITHUB_TOKEN lookup that fell back to team=unknown; also run in tf-apply

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Alexanderamiri
Alexanderamiri requested a review from a team as a code owner October 31, 2026 12:09
@github-actions

Copy link
Copy Markdown

Terraform Plan

✅ No changes — infrastructure is up to date.

Plan output
Acquiring state lock. This may take a few moments...
aws_ssm_parameter.atlassian_org_id: Refreshing state... [id=/javabin/platform/atlassian-org-id]
aws_ssm_parameter.atlassian_api_key: Refreshing state... [id=/javabin/platform/atlassian-api-key]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant