scripts

scripts/

Repo-root utility scripts: trust kit lifecycle (scaffold, register, sync, distribute, onboard), local environment/status checks, cross-service drift guards run by make targets or CI, and secret scanning. Most are invoked through make targets rather than run directly — see each section below for the wrapping target.

Trust kit scripts

These scaffold, write, and keep in sync the trust/.env.<CODE>.<env> kit files that carry a trust's identity, credentials, and Hub-shared configuration. trust_kit_lib.py is the shared in-place-upsert implementation behind the others, so a kit file's operator edits (host-local ports, bind dirs) always survive a re-run.

• new_trust.py (make new-trust TRUST_CODE=<CODE> TRUST_NAME="..." HUB=<stag|prod>; leave HUB unset for development) — scaffolds trust/.env.<CODE>.<env> from the base template (trust/.env.example), prepending the trust's identity (TRUST_NAME / TRUST_CODE / TRUST_REGION). Refuses to overwrite an existing kit (the root new-trust target also refuses beside a kit still under its pre-FLIP#1479 name). The result is ready for registration to fill in credentials (make -C deploy/providers/AWS register-trusts KIT=<CODE> HUB=<hub> on a deployed hub, make register-trust KIT=<CODE> in development).
• trust_kit_lib.py — the single kit-file writer behind distribute_trust_kits.py, sync_trust_kit.py, and the AWS registration path. Merges a kit dict into a kit file while preserving operator edits: credentials (TRUST_API_KEY / TRUST_INTERNAL_SERVICE_KEY) are written only on a new registration and never clobbered on an idempotent re-run; metadata (EXPECTED_TRUST_ID / FL_KIT_SLOT / FL_KIT_SLOT_NUMBER) and the Hub-shared block are upserted unconditionally, the latter under a sentinel header added once on first write. Writes every kit file 0600.
• distribute_trust_kits.py — writes a trust kit file from register_trust's JSON output: one kit (object or one-element array) on stdin to the required --target PATH — the dev register-trust Makefile pipe (--target trust/.env.<CODE>.<env>) and deploy/providers/AWS/scripts/register-trusts.sh for stag/prod alike.
• sync_trust_kit.py (make sync-trust-kit KIT=<CODE> HUB=<env>) — refreshes only the Hub-shared block in trust/.env.<CODE>.<env> from the caller's environment (the root Makefile includes the right .env.<env> and exports it first). Credentials and operator edits are left untouched. Portable across dev/stag/prod — no docker compose exec, no ECS round-trip.
• onboard_onprem_trust.py (make onboard-onprem-trust KIT=<CODE>) — a readiness checklist for an on-prem trust host, run as a precheck by up-onprem-trust. Diagnoses what's missing before make up-onprem-trust will succeed: public IP, docker swarm state, kit file presence, Hub-shared block, kit credentials, FL_KIT_DIR contents, OMOP/Orthanc data dirs, and GPU capacity (NUM_AVAILABLE_GPUS vs. what the host actually exposes). Each check renders ✅ / ❌ / ⏳ (pending on an earlier check) and exits non-zero if anything fails, so an operator gets concrete diagnostics instead of a cryptic compose or pydantic failure deeper in the stack.

Site upgrades

• site_upgrade.py (invoked by make -C trust upgrade-trust, and by the AWS/Helm twins — see AGENTS.md "Site release upgrades") — the plan-and-guard-and-pin phase of a site release upgrade (FLIP#1204). Resolves the target tag against what the hub reports on /api/health (defaults to that; --tag vX.Y.Z or --tag sha-<short7> overrides), says when GitHub lists a newer release (offering to stop and move to it only when the hub already runs it), runs the guards (image tag shape, downgrade → --force, release-tag checkout match → --allow-checkout-drift, every image actually built at the tag via docker manifest inspect, operator confirmation unless --yes), and — if everything passes — rewrites DOCKER_TAG / DOCKER_FL_TAG in the kit's Hub-shared block. The script stops before any container is touched; the Makefile then re-includes the updated kit and does the pull / recreate (_upgrade-trust-apply on trust; the EC2 twin and make -C trust/deploy/helm upgrade-trust-k8s do the same for the other shapes). The operator entry points wrap both phases together — make upgrade-onprem-trust KIT=<CODE> also runs onboard-onprem-trust --gate (the readiness checklist implemented by onboard_onprem_trust.py, above) before the plan phase. Exit codes (the Makefile's contract): 0 pinned, 2 no usable target, 3 refused downgrade, 4 not confirmed, 5 an image is missing at the target tag, 6 the checkout is not at the target release. Running the script directly rewrites the kit but leaves the containers at the old tag — use the Make targets. Runbook: docs/source/sys-admin/admin-upgrading-sites.rst.

Status and environment checks

• check_local_status.py (run directly: python3 scripts/check_local_status.py) — verifies the local Docker Compose stack is functioning: Docker daemon, containers, networks, application endpoints (UI, API, FL API), trust endpoints (trust-api, imaging-api, data-access-api), database connectivity, and system resources. Exits 0 when all checks pass (warnings are acceptable), 1 otherwise. Options include --skip-endpoints, --skip-docker, --project-dir, and --env-file. Resolves container and network names through compose project labels (honouring FLIP_INSTANCE / MAIN_ENV_FILE) so it reports correctly against a prefixed second stack, not just the default one.
• check_env_vars.py (pre-commit hook check-env-vars) — verifies every variable declared in .env.development.example is also present in .env.development, so the example file stays up to date and a new required variable can't be missed silently.

Drift guards

None of these is a pre-commit hook. The first two are make-time guards that run in no CI workflow; the other two are CI-only:

• check-fl-provisioned.sh (the make up / make up-no-trust guard, via the root Makefile's _check-fl-provisioned) — fails fast when a backend's per-net FL credentials are missing, instead of letting the FL containers crash-loop. Neither backend's credentials are created by make up — they come from make -C fl-services/<backend> provision....
• check-uv-version.sh (the make lock guard) — fails fast when uv is too old to understand the dependency cooldown (exclude-newer = "3 days", which needs uv >= 0.10.0). An older uv silently discards the whole [tool.uv] table and resolves with no cooldown at all, rewriting uv.lock with no warning worth noticing.
• check_fl_api_validation_sync.sh (CI: fl-api-validation-sync.yml) — verifies that safe_join and validate_bundle_url (the SSRF and path-traversal guards in front of the server-side bundle fetch) stay byte-identical between the FLARE and Flower fl-api services. They're intentionally separate copies (different Docker build contexts, uv projects, and images), so a fix applied to one and not the other would silently leave the second vulnerable.
• check_tutorial_sync.sh (CI: fl-apps-check-tutorial-sync.yml) — verifies that tutorial files kept as byte-identical copies of another file have not drifted. It holds two families of hand-listed pairs: the Ark+ FLARE pair (data_utils.py and arkplus_flat_models.py, shared between the two evaluation apps) and the EHR risk-prediction cross-backend pair (feature_engineering.py, models.py, query.sql, with the FLARE copy as the reference). The Flower tutorial-vs-fl-apps/flower/ template pairs moved to fl-tutorials/tests/test_flower_platform_parity.py, which derives them from the tree. These can't be symlinks — flwr build excludes symlinks from the FAB — so each keeps a real copy that must be resynced by hand when its reference changes.
• pr_paths_changed.py (CI: pr_paths_changed.yml, called by every gated service-test workflow) — implements the PR path gate: reads a service workflow's paths: filter, fetches the PR's changed files, and outputs run=true when the filter matches so the workflow's jobs gate on it. A PR into main always runs everything. scripts/tests/test_pr_paths_changed.py pins the gate against drift.
• utils.sh — shared shell helpers (colour-coded log_info / log_success / etc.) sourced only by the two secret-scanning scripts (scan-secrets.sh, setup-secret-scanning.sh); the drift guards above are self-contained. Not run directly.

fl_round_metrics/

Tooling for extracting and comparing per-round FL timings (not model metrics) from either a live platform run (CloudWatch logs) or a local FLARE simulator workspace. See fl_round_metrics/README.md for the tools and the make round-metrics / make reproduce-overhead wrapper in the arkplus fine-tuning tutorial.

tests/

Stdlib-only tests for the scripts in this directory (scripts/tests/test_*.py), each following the same if __name__ == "__main__": main() contract — no pytest, no external services beyond uv itself (needed because test_sync_trust_kit.py drives sync_trust_kit.py through uv run --no-config). Run the whole directory:

for test_file in scripts/tests/test_*.py; do
  echo "== $test_file"
  python3 "$test_file" || exit 1
done

CI runs the same loop in .github/workflows/test_trust_kit_scripts.yml, triggered on changes under scripts/** and on the hub/trust compose files that test_container_identity.py asserts against (so an edit to those triggers the guard even though it touches nothing under scripts/).

Secret scanning

This directory also contains scripts for detecting secrets and sensitive information in the FLIP monorepo.

Quick Start

Prerequisites

This monorepo uses uv for Python package management. If you haven't installed it yet:

curl -LsSf https://astral.sh/uv/install.sh | sh

Initial Setup

Run the setup script once to install and configure all secret scanning tools:

./scripts/setup-secret-scanning.sh

This will:

• Install pre-commit, detect-secrets, and trufflehog
• Set up pre-commit hooks
• Create a baseline for detect-secrets
• Update .gitignore to exclude security reports

Running Scans

Full Monorepo Scan

To scan the entire monorepo for secrets:

./scripts/scan-secrets.sh

Pre-commit Checks

Check staged files before committing:

pre-commit run

Check all files:

pre-commit run --all-files

Available Pre-commit Hooks (see .pre-commit-config.yaml for the authoritative list):

• check-env-vars: Verifies that all variables in .env.development.example exist in .env.development. This ensures the example file stays up to date and developers are aware of new required environment variables.
• fl-apps-required-files: Regenerates each backend's fl-apps/<backend>/required_files.json from its per-template required_files.json arrays. Rewrites-and-fails on drift (like prettier), so re-stage and commit again.
• xnat-dcm2niix-pin-sync: Checks that the dcm2niix Container Service image pin (ARG DCM2NIIX_VERSION in its Dockerfile) agrees across the three deploy configs that reference it as a literal tag string. Report-only — the sites are hand-written, not generated.
• trufflehog: Scans for high-entropy strings and verified secrets
• detect-secrets: Pattern-based secret detection with baseline support
• check-added-large-files: Prevents committing files larger than 1000 KiB (--maxkb=1000 — the hook compares against getsize // 1024, so the cap is ~1.024 MB)
• check-merge-conflict: Detects merge conflict markers
• check-yaml: Validates YAML syntax
• prettier: Formats YAML under deploy/providers/AWS/
• end-of-file-fixer: Ensures files end with a newline
• detect-private-key: Detects private SSH/SSL keys
• uv-lock (one entry per uv project — root, flip-api, docs, trust-api, imaging-api, data-access-api, omop-db, xnat-tests, aws-deploy, flip-utils, fl-api-flower, fl-api-base): verifies each project's uv.lock is in sync with its pyproject.toml (--check, so it fails on drift instead of silently regenerating). Use make lock to regenerate when a pyproject.toml legitimately changes.

Component-Specific Scans

To scan specific components (flip-api, trust, etc.):

cd flip-api
trufflehog git file://. --only-verified

Emergency: Remove Secrets from Git History

If secrets were accidentally committed, follow the GitHub guide on removing sensitive data using tools such as git filter-repo or BFG Repo-Cleaner.

⚠️ WARNING: This rewrites git history. Only use when absolutely necessary. Coordinate with all team members before force-pushing.

Tools Used

TruffleHog

High-entropy string and secret scanner with built-in detector patterns.

Direct usage:

trufflehog git file://. --only-verified --json

detect-secrets

Pattern-based scanner with baseline support.

Direct usage:

detect-secrets scan --all-files

CI/CD Integration

Secret scanning runs automatically on:

• Push to main or develop branches
• Pull requests
• Weekly schedule (Sundays at 2 AM UTC)

See .github/workflows/secret-scanning.yml for details.

Managing False Positives

detect-secrets Baseline

To update the baseline with new false positives:

detect-secrets scan --baseline .secrets.baseline --update

To audit the baseline:

detect-secrets audit .secrets.baseline

Troubleshooting

uv Not Found

This project uses uv for Python package management. Install it with:

curl -LsSf https://astral.sh/uv/install.sh | sh

After installation, ensure ~/.local/bin is in your PATH:

# Add to your ~/.bashrc, ~/.zshrc, or ~/.config/fish/config.fish
export PATH="$HOME/.local/bin:$PATH"

Commands Not Found After Installation

If pre-commit or detect-secrets aren't found after running the setup script, the uv tool directory may not be in your PATH. Add it:

export PATH="$HOME/.local/bin:$PATH"

Or reinstall the tools globally via Homebrew:

brew install pre-commit
pip install detect-secrets  # Only if using pipx or virtual environment

TruffleHog Not Found

Install via Homebrew:

brew install trufflesecurity/trufflehog/trufflehog

Or use Docker:

docker pull trufflesecurity/trufflehog:latest

Pre-commit Hook Failures

If pre-commit hooks fail due to missing tools:

1. Ensure tools are installed: which trufflehog
2. Reinstall pre-commit hooks: pre-commit install
3. Update pre-commit: pre-commit autoupdate

Best Practices

1. Run scans before committing: Use pre-commit run to catch secrets early
2. Review scan results carefully: Not all findings are real secrets
3. Never commit real secrets: Use environment variables or secret management
4. Update baselines regularly: Keep false positive baselines current
5. Scan the entire history: Use full scans periodically to find historical secrets

Security Reports

Scan reports are saved to .security-reports/ (gitignored). Review these carefully before sharing or opening issues.

Support

For issues or questions about secret scanning:

• Check tool documentation: TruffleHog, detect-secrets
• Review CONTRIBUTING.md for security guidelines