Repository navigation
Cancel the Cloud Build and labelled pipeline runs when GitHub cancels an e2e job - #829
kmontemayor2-sc wants to merge 11 commits into
Conversation
… an e2e job A GitHub cancel used to stop only the runner; the Cloud Build kept running and so did every Vertex AI pipeline it launched. The Cloud Build config now tags each build gigl-ci-run-<run id>, passed through run-cloud-run-command-on-active-checkout only when the checked-out config declares the substitution (a PR branch forked before this change reaches that action from main and Cloud Build rejects an undeclared substitution). A new composite action, cancel-ci-gcp-resources, runs on `if: cancelled()` and executes .github/scripts/cancel_ci_gcp_resources.py through `uv run --frozen --only-group gcp`: it cancels the tagged builds with gcloud and the pipeline runs labelled github_run_id through VertexAIService, so each run's exit task then cancels its own child jobs. The merge queue e2e job sets E2E_FAIL_FAST=1 and passes the run id; the /e2e_test comment job passes the run id without fail fast so a PR author sees every failing pipeline. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…cancel-gcp-resources
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…cancel-gcp-resources
| make run_all_e2e_tests GITHUB_RUN_ID=${{ github.run_id }} | ||
| - name: Cancel GCP resources from this run | ||
| if: cancelled() | ||
| uses: snapchat/gigl/.github/actions/cancel-ci-gcp-resources@main |
There was a problem hiding this comment.
🟡 Medium severity issue identified in your code:
The workflow runs snapchat/gigl's cancel-ci-gcp-resources action from mutable @main. If that branch is repointed, cancelling a PR-triggered E2E run can execute attacker-controlled code with the job's GitHub and GCP access.
More details about this
cancel-ci-gcp-resources is loaded from snapchat/gigl using the mutable @main reference. The repository owner can move main to a different commit without changing this workflow, so a future cancelled integration-e2e-test run could execute attacker-controlled action code with GITHUB_TOKEN, GCP_PROJECT_ID, and the configured GCP access context. Because integration-e2e-test can be started by a pull-request comment such as /e2e_test or /all_test, an attacker could trigger a run, cancel it, and cause the action selected from the repointed main branch to run during if: cancelled().
A plausible attack would be:
- An attacker compromises the
snapchat/giglrepository or obtains permission to update itsmainbranch, then changes.github/actions/cancel-ci-gcp-resources/action.ymlso its action code sends${{ secrets.GITHUB_TOKEN }}and the GCP identifiers available to the job to an attacker-controlled endpoint, or uses the job's GCP access to alter resources. - The attacker comments
/e2e_teston a pull request, satisfying theintegration-e2e-testcondition and starting the workflow withGITHUB_TOKEN,WORKLOAD_IDENTITY_PROVIDER, andGCP_SERVICE_ACCOUNT_EMAILin its context. - The attacker cancels that run. The
cancelled()condition then invokessnapchat/gigl/.github/actions/cancel-ci-gcp-resources@main, which now resolves to the malicious commit and executes the changed action with theproject,region, andgithub_run_idinputs. - The malicious action exfiltrates accessible tokens or performs unauthorized operations in the configured GCP project instead of merely cancelling resources.
To resolve this comment:
✨ Commit fix suggestion
- Identify the reviewed 40-character commit SHA in
snapchat/giglthat contains the required action versions. - Replace the mutable
@mainreference with that SHA. For example:uses: snapchat/gigl/.github/actions/cancel-ci-gcp-resources@<40-character-commit-sha>. - Pin the other
snapchat/gigllocal actions in this workflow the same way, includingrun-command-on-pr, instead of using@main. - Keep the action path and inputs unchanged. A full commit SHA prevents the action code from changing silently when the branch is moved.
💬 Ignore this finding
Reply with Semgrep commands to ignore this finding.
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by github-actions-mutable-action-tag.
You can view more details about this finding in the Semgrep AppSec Platform.
…cancel-gcp-resources
…into kmonte/ci-cancel-gcp-resources
…cancel-gcp-resources
…cancel-gcp-resources
…fiers Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…cancel-gcp-resources
Summary
Cancelling the GitHub e2e job stops only the runner. The Cloud Build keeps running, and so does every Vertex AI pipeline it launched. This PR makes a GitHub cancel reach GCP: the build is cancelled by tag, the pipeline runs by label, and each cancelled run's exit task then cancels its own child jobs. The merge queue also turns on fail fast. Last of a four PR stack.
Approach
gigl-ci-run-<run id>.run-cloud-run-command-on-active-checkoutpasses the substitution only when the checked-out config declares it: inon-pr-comment.yml,run-command-on-pr@mainchecks out the PR branch and then calls this action from main, so a PR branch forked before this change still reaches it, and Cloud Build rejects an undeclared substitution.cancel-ci-gcp-resources, runs onif: cancelled()and executes.github/scripts/cancel_ci_gcp_resources.pythroughuv run --frozen --only-group gh-actions. The script cancels the tagged builds withgcloudand the labelled pipeline runs withVertexAIService.cancel_pipeline_runs_with_label. Both kinds are attempted even if the other fails.E2E_FAIL_FAST=1. The/e2e_testcomment job passes the run id without fail fast, so a PR author still sees every failing pipeline.on-pr-comment.ymlstay on@main, matching the seven existing references in that workflow. That event has no checkout when its first step runs, and a./reference after the PR checkout would run the PR author's copy of the action with the job's secrets.User-facing changes
on-pr-comment.ymlruns from main, so its changes apply to the next PR, not this one.Verification
make check_format_pyandmake type_checkpass; the script is under.github/scripts, which both cover.snap-umap-dev:key "_GITHUB_RUN_ID" in the substitution data is not matched in the template.snap-umap-devfour times with the same list and cancel calls the script makes. Each time the labelled runs were found and cancelled, their exit tasks ran, and the launcher reported them CANCELLED.snap-umap-devfrom an environment built withuv run --frozen --only-group gh-actionsand exited 0 after finding the finished runs.Stack:
kmonte/gcp-dep-group→kmonte/gcp-service-cancel-helpers→kmonte/kfp-fail-fast-cleanup-exit-task→ this PR.🤖 Generated with Claude Code