Skip to content

Cancel the Cloud Build and labelled pipeline runs when GitHub cancels an e2e job - #829

Draft
kmontemayor2-sc wants to merge 11 commits into
kmonte/kfp-fail-fast-cleanup-exit-taskfrom
kmonte/ci-cancel-gcp-resources
Draft

kmontemayor2-sc wants to merge 11 commits into
kmonte/kfp-fail-fast-cleanup-exit-taskfrom
kmonte/ci-cancel-gcp-resources

Conversation

@kmontemayor2-sc

Copy link
Copy Markdown
Collaborator

Summary

Cancelling the GitHub e2e job stops only the runner. The Cloud Build keeps running, and so does every Vertex AI pipeline it launched. This PR makes a GitHub cancel reach GCP: the build is cancelled by tag, the pipeline runs by label, and each cancelled run's exit task then cancels its own child jobs. The merge queue also turns on fail fast. Last of a four PR stack.

Approach

  • The Cloud Build config tags each build gigl-ci-run-<run id>. run-cloud-run-command-on-active-checkout passes the substitution only when the checked-out config declares it: in on-pr-comment.yml, run-command-on-pr@main checks out the PR branch and then calls this action from main, so a PR branch forked before this change still reaches it, and Cloud Build rejects an undeclared substitution.
  • A new composite action, cancel-ci-gcp-resources, runs on if: cancelled() and executes .github/scripts/cancel_ci_gcp_resources.py through uv run --frozen --only-group gh-actions. The script cancels the tagged builds with gcloud and the labelled pipeline runs with VertexAIService.cancel_pipeline_runs_with_label. Both kinds are attempted even if the other fails.
  • The merge queue e2e job passes the run id and sets E2E_FAIL_FAST=1. The /e2e_test comment job passes the run id without fail fast, so a PR author still sees every failing pipeline.
  • Repo-local actions in on-pr-comment.yml stay on @main, matching the seven existing references in that workflow. That event has no checkout when its first step runs, and a ./ reference after the PR checkout would run the PR author's copy of the action with the job's secrets.

User-facing changes

  • Merge queue e2e runs stop at the first failed pipeline and cancel the rest.
  • A cancelled e2e workflow run now cancels its Cloud Build and Vertex AI runs. The cancel step and the Cloud Build tag only take effect in CI after this merges, and on-pr-comment.yml runs from main, so its changes apply to the next PR, not this one.

Verification

  • YAML parses for all six changed files. make check_format_py and make type_check pass; the script is under .github/scripts, which both cover.
  • Cloud Build's rejection of an undeclared substitution was confirmed with a live submit in snap-umap-dev: key "_GITHUB_RUN_ID" in the substitution data is not matched in the template.
  • The pipeline cancel path was run live in snap-umap-dev four times with the same list and cancel calls the script makes. Each time the labelled runs were found and cancelled, their exit tasks ran, and the launcher reported them CANCELLED.
  • The script itself was run against snap-umap-dev from an environment built with uv run --frozen --only-group gh-actions and exited 0 after finding the finished runs.
  • Not verified: the GitHub Actions wiring, which only runs in CI after merge.

Stack: kmonte/gcp-dep-group → kmonte/gcp-service-cancel-helpers → kmonte/kfp-fail-fast-cleanup-exit-task → this PR.

🤖 Generated with Claude Code

kmonte and others added 5 commits October 29, 2026 19:49
… an e2e job

A GitHub cancel used to stop only the runner; the Cloud Build kept running and so
did every Vertex AI pipeline it launched. The Cloud Build config now tags each build
gigl-ci-run-<run id>, passed through run-cloud-run-command-on-active-checkout only
when the checked-out config declares the substitution (a PR branch forked before
this change reaches that action from main and Cloud Build rejects an undeclared
substitution). A new composite action, cancel-ci-gcp-resources, runs on
`if: cancelled()` and executes .github/scripts/cancel_ci_gcp_resources.py through
`uv run --frozen --only-group gcp`: it cancels the tagged builds with gcloud and the
pipeline runs labelled github_run_id through VertexAIService, so each run's exit
task then cancels its own child jobs.

The merge queue e2e job sets E2E_FAIL_FAST=1 and passes the run id; the /e2e_test
comment job passes the run id without fail fast so a PR author sees every failing
pipeline.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
make run_all_e2e_tests GITHUB_RUN_ID=${{ github.run_id }}
- name: Cancel GCP resources from this run
if: cancelled()
uses: snapchat/gigl/.github/actions/cancel-ci-gcp-resources@main

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium severity issue identified in your code:

The workflow runs snapchat/gigl's cancel-ci-gcp-resources action from mutable @main. If that branch is repointed, cancelling a PR-triggered E2E run can execute attacker-controlled code with the job's GitHub and GCP access.

More details about this

cancel-ci-gcp-resources is loaded from snapchat/gigl using the mutable @main reference. The repository owner can move main to a different commit without changing this workflow, so a future cancelled integration-e2e-test run could execute attacker-controlled action code with GITHUB_TOKEN, GCP_PROJECT_ID, and the configured GCP access context. Because integration-e2e-test can be started by a pull-request comment such as /e2e_test or /all_test, an attacker could trigger a run, cancel it, and cause the action selected from the repointed main branch to run during if: cancelled().

A plausible attack would be:

  1. An attacker compromises the snapchat/gigl repository or obtains permission to update its main branch, then changes .github/actions/cancel-ci-gcp-resources/action.yml so its action code sends ${{ secrets.GITHUB_TOKEN }} and the GCP identifiers available to the job to an attacker-controlled endpoint, or uses the job's GCP access to alter resources.
  2. The attacker comments /e2e_test on a pull request, satisfying the integration-e2e-test condition and starting the workflow with GITHUB_TOKEN, WORKLOAD_IDENTITY_PROVIDER, and GCP_SERVICE_ACCOUNT_EMAIL in its context.
  3. The attacker cancels that run. The cancelled() condition then invokes snapchat/gigl/.github/actions/cancel-ci-gcp-resources@main, which now resolves to the malicious commit and executes the changed action with the project, region, and github_run_id inputs.
  4. The malicious action exfiltrates accessible tokens or performs unauthorized operations in the configured GCP project instead of merely cancelling resources.

To resolve this comment:

✨ Commit fix suggestion
  1. Identify the reviewed 40-character commit SHA in snapchat/gigl that contains the required action versions.
  2. Replace the mutable @main reference with that SHA. For example: uses: snapchat/gigl/.github/actions/cancel-ci-gcp-resources@<40-character-commit-sha>.
  3. Pin the other snapchat/gigl local actions in this workflow the same way, including run-command-on-pr, instead of using @main.
  4. Keep the action path and inputs unchanged. A full commit SHA prevents the action code from changing silently when the branch is moved.
💬 Ignore this finding

Reply with Semgrep commands to ignore this finding.

  • /fp <comment> for false positive
  • /ar <comment> for acceptable risk
  • /other <comment> for all other reasons

Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by github-actions-mutable-action-tag.

You can view more details about this finding in the Semgrep AppSec Platform.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants