flip-api

flip-api

The Central Hub API is the coordination layer for the FLIP platform. It is the primary interface for researchers and administrators, managing projects, users, federated learning tasks, model retrieval, and cohort queries across all connected Trust environments.

Role in the FLIP Platform

The Central Hub API orchestrates the full lifecycle of a federated learning study:

1. Project management — create, approve, and track FL projects across participating Trusts
2. User management — researcher and administrator accounts in the configured OIDC identity provider (AWS Cognito in staging and production, Keycloak in local development)
3. Cohort queries — dispatch OMOP SQL queries to Trust sites and aggregate results
4. Model coordination — trigger training jobs, retrieve aggregated models, and store metrics
5. Audit logging — record all significant platform events for governance and compliance

It communicates with each Trust's trust-api and stores all state in a PostgreSQL database.

Deployment

The flip-api is deployed as a Docker container. In the full local stack, it is started via:

make central-hub   # starts flip-api and the database

or as part of the full platform:

make up

Before starting the platform, generate the internal service key:

make generate-internal-service-key            # from repo root (also invoked automatically by `make up`)

generate-internal-service-key writes INTERNAL_SERVICE_KEY and INTERNAL_SERVICE_KEY_HASH to the active env file (MAIN_ENV_FILE, default .env.development; used for fl-server-to-hub authentication).

Trusts are registered on the running hub with make register-trusts (shipped dev roster; from the repo root) or make register-trust KIT=<CODE> (one trust). Each inserts a trust row with its api_key_hash, claims an FL kit slot, and fills that trust's kit file trust/.env.<CODE>.<env> carrying TRUST_API_KEY and TRUST_INTERNAL_SERVICE_KEY — neither plaintext is stored on the hub. make up runs register-trusts automatically once the hub is up. See .env.development.example for the expected format.

The API is served on the port defined by API_PORT in .env.development.example (default: 8080). Interactive API documentation (Swagger UI) is available at:

http://localhost:<API_PORT>/api/docs

Configuration

The flip-api is configured via environment variables. In development these are set in .env.development.example. Key variables include:

Variable	Description
DB_HOST	PostgreSQL host
DB_PORT	PostgreSQL port
POSTGRES_USER	PostgreSQL username
POSTGRES_DB	PostgreSQL database name
AWS_REGION	AWS region for S3 (and Cognito in staging/production)
AUTH_BACKEND	Identity provider: keycloak (development — the keycloak service in deploy/compose.development.yml, the only value DevSettings accepts) or cognito (staging and production, the only value ProdSettings accepts)
AWS_COGNITO_USER_POOL_ID	AWS Cognito User Pool ID (staging/production only)
AWS_COGNITO_APP_CLIENT_ID	AWS Cognito App Client ID (staging/production only)
KEYCLOAK_URL	Keycloak base URL as flip-api reaches it over the Docker network (JWKS + Admin REST API); dev default http://keycloak:8080
KEYCLOAK_PUBLIC_URL	Keycloak base URL as the browser reaches it, hence the iss every token carries; dev default http://localhost:8180 (follows KEYCLOAK_PORT)
KEYCLOAK_REALM	Realm name (default flip)
KEYCLOAK_CLIENT_ID	Public browser client the UI signs in with (default flip-ui); its redirect URIs are the CORS allowlist
KEYCLOAK_AUDIENCE	aud claim the verifier requires (default flip-api)
KEYCLOAK_ADMIN_CLIENT_ID	Confidential service-account client flip-api administers users as (default flip-api-admin)
KEYCLOAK_ADMIN_CLIENT_SECRET	That client's secret; shared by the realm import and flip-api (dev placeholder in .env.development.example)
AWS_SECRET_NAME	Name of the AWS Secrets Manager secret flip_api/utils/get_secrets.py reads by default
AES_KEY_BASE64	Base64-encoded AES-256 key used to encrypt trust task payloads and project IDs. Shared between hub (encryption) and trusts (decryption)
UPLOADED_MODEL_FILES_BUCKET	S3 bucket (staging prefix) for pre-scan model-file uploads
SCANNED_MODEL_FILES_BUCKET	S3 bucket that promoted clean files land in — the platform's quarantine boundary; every consumer reads from here
FL_APP_DESTINATION_BUCKET	S3 bucket for bundled FL applications served to trusts
UPLOADED_FEDERATED_DATA_BUCKET	S3 bucket for storing models and artefacts

See .env.development.example for the full list of required variables.

Tunables

A handful of scheduler and task-queue settings are absent from .env.development.example (all but SCHEDULER_MALWARE_SCAN_RECONCILE_RATE, which appears there commented out under the malware-scan block) because their config.py defaults are meant to be left alone in normal operation; override them only when diagnosing or deliberately changing that behaviour.

Variable	Default	Meaning
HEARTBEAT_TIMEOUT_SECONDS	30	How long since a trust's last heartbeat before it's considered offline
TASK_STALE_TIMEOUT_MINUTES	30	How long a task can sit IN_PROGRESS before it's considered stale
TASK_MAX_RETRIES	3	Max times a stale task is retried before being marked FAILED
SCHEDULER_STALE_TASK_RECOVERY_RATE	10 (minutes)	How often the scheduler checks for stale tasks
MAX_TASK_RESULT_LENGTH	10_000_000 (characters)	Max size of a task result payload
PROJECT_REIMPORT_RATE	60 (minutes)	How often to reimport studies for a given project
MAX_REIMPORT_COUNT	5	Max reimport attempts before a project's imaging import stops retrying
SCHEDULE_RUN_JOBS_EXECUTION	true	Whether the scheduler registers the FL job-pickup pass (run_jobs_scheduled_task) and the FL job watchdog (watch_fl_jobs_scheduled_task). Only those two jobs are gated: the fl-api keep-alive, imaging reimport, stale-task recovery and malware-scan reconcile jobs are registered regardless
SCHEDULER_RUN_JOBS_RATE	1 (minute)	How often the scheduler's FL job-pickup pass runs
SCHEDULER_FL_JOB_WATCHDOG_RATE	1 (minute)	How often the scheduler checks each running FL job against its net's FL API
FL_JOB_WATCHDOG_STRIKES	3	Consecutive checks a job must look ended or lost on its backend before its model is set to ERROR and the net freed
FL_JOB_PREPARE_TIMEOUT_MINUTES	30	How long a picked-up job may go without reaching the FL server before its model is set to ERROR
SCHEDULER_KEEP_FL_API_SESSION_ALIVE_RATE	2 (minutes)	How often the scheduler pings fl-api to keep its session alive
SCHEDULER_REIMPORT_IMAGING_PROJECT_STUDIES_RATE	30 (minutes)	How often the scheduler checks for projects with unimported studies
SCHEDULER_MALWARE_SCAN_RECONCILE_RATE	1 (minute)	How often the scheduler re-checks uploads left SCANNING by a restart mid-scan

Database migrations

The PostgreSQL schema is owned by Alembic. At container boot the entrypoint runs alembic upgrade head (after Postgres is reachable, before seeding) — a non-zero migration exit fails fast and never starts the API. This replaces the old SQLModel.metadata.create_all(), which only ever created missing tables and never applied a schema diff to an existing database.

• Migration scripts live in src/flip_api/db/migrations/ (so they ride the dev src/ bind mount and the prod COPY src/); config is in alembic.ini.
• alembic.ini holds no database URL or credentials. env.py reuses flip_api.db.database.get_engine(), so migrations inherit the exact same connection string and the prod RDS-Proxy IAM authentication (a per-connection token minted by a do_connect hook). Tests inject their Testcontainers connection via config.attributes["connection"] instead.

Workflow

Any schema-affecting change to db/models/*.py (new column, altered type, dropped table, new model, or a new value on an existing enum) must ship an Alembic revision. The integration suite's drift guard (tests/integration/test_migrations.py) fails CI otherwise — compare_metadata covers tables/columns/types/FKs/indexes, and a companion check (test_no_enum_value_drift) covers native-enum value changes that compare_metadata is blind to.

make up                         # bring up flip-db + flip-api at the current head first
make migration MESSAGE="add foo to bar"   # autogenerate a revision from the model diff
# --> REVIEW the generated file in src/flip_api/db/migrations/versions/ (see below)
make migrate                    # apply it (alembic upgrade head)

Other targets: make migration_downgrade (down one), make migration_history, make migration_current.

Reviewing an autogenerated revision

Always hand-review the generated file — autogenerate has two known blind spots here:

1. Native PostgreSQL ENUM types. Status/action fields (JobStatus, ModelStatus, TaskStatus, …) map to ~12 native PG enum types. create_table auto-creates the type, but drop_table does not drop it — so a downgrade that removes a table with an enum column must add an explicit op.execute("DROP TYPE IF EXISTS <name>"), or a later re-upgrade fails with "type already exists" (see the baseline's downgrade for the pattern).
2. Adding a value to an existing enum. Autogenerate will not emit this (it diffs only the column's type name), but test_no_enum_value_drift fails CI to remind you to hand-write the revision. ALTER TYPE … ADD VALUE cannot run inside a transaction — wrap it in with op.get_context().autocommit_block(): op.execute(...). (The integration harness runs migrations on a plain connect(), so autocommit_block works there too.)

The queries.queried_trust_ids column is an ARRAY(UUID) (postgresql.ARRAY(sa.UUID())) — check it survives any regeneration. env.py renders SQLModel's AutoString as plain sa.String, so migrations don't need to import sqlmodel.

Testing

Tests are split into tests/unit/ (no real backing services) and tests/integration/ (real Postgres / Keycloak / Cognito / S3 / sibling APIs). See Where does my test go? in CONTRIBUTING.md for the placement rule.

Run unit tests (no Docker / DB / network needed):

make unit_test

Run integration tests against a throwaway Postgres (Docker required, but no compose stack):

make integration_test

Integration tests use testcontainers-python to start a postgres:16-alpine container per test session and tear it down at the end. The schema is built by running the Alembic migrations (alembic upgrade head, the same DDL dev/prod apply at boot) and then seeded with permissions / roles / role-permissions in tests/integration/conftest.py; per-test tables are truncated between tests. Building from migrations (not create_all) means tests/integration/test_migrations.py doubles as a drift guard — a model change shipped without a matching revision fails the suite. New integration tests just request the session (real DB) and/or client (FastAPI TestClient wired to the same DB) fixtures — no setup boilerplate.

AWS-touching tests (S3, Cognito, SES)

A session-scoped autouse fixture (aws_mock in tests/integration/conftest.py) enters moto.mock_aws() for the whole test session, intercepting boto3.client(...) calls at the botocore layer. The flip-api production code paths in src/flip_api/utils/s3_client.py, the Cognito identity provider src/flip_api/auth/identity/cognito.py, and the boto3.client("sesv2", ...) construction in src/flip_api/utils/email_sender.py (used by user_services/access_request.py and private_services/imaging_notifications.py) all hit the moto fake with no test-only branches in source.

Per-service helper fixtures bootstrap the state each test needs:

• s3_buckets — creates the buckets configured in Settings (UPLOADED_MODEL_FILES_BUCKET, SCANNED_MODEL_FILES_BUCKET, UPLOADED_FEDERATED_DATA_BUCKET, FL_APP_DESTINATION_BUCKET). The base FL application templates are not an S3 bucket — they are read from the local FL_APP_BASE_DIR tree baked into the image (FLIP#724).
• cognito_user_pool — creates a moto user pool + app client, pins AUTH_BACKEND=cognito and rebinds Settings.AWS_COGNITO_USER_POOL_ID / AWS_COGNITO_APP_CLIENT_ID to point at them, clearing the provider cache (auth/identity/factory.py) so the next request builds a Cognito provider against the fresh IDs.
• ses_send_email_recorder — captures every sesv2.send_email call. moto v5 explicitly raises NotImplementedError on send_email with Content.Template, and every flip-api SES caller uses templated content; the recorder wraps the production-code path up to the SDK boundary so the test asserts the boto3 call shape (FromEmailAddress, Destination.ToAddresses, TemplateName, TemplateData). It's the closest approximation to a real SES round-trip moto's coverage allows today. It also pins EMAIL_BACKEND="ses", since development defaults to the console backend (FLIP#919) — without that pin the SES path under test would never run. tests/integration/test_console_email_backend.py is the counterpart: it leaves the dev default alone and asserts the email paths succeed, and log rather than send, with no SES configuration at all.

Why moto and not LocalStack: cognito-idp and sesv2 are Pro-only on LocalStack — the free tier rejects CreateUserPool / CreateEmailIdentity outright. moto covers all three in OSS and runs in-process, so there's no container boot per test session.

Keycloak-touching tests

tests/integration/test_keycloak_round_trips.py is the twin of test_cognito_round_trips.py for the Keycloak provider (src/flip_api/auth/identity/keycloak.py): it boots the pinned Keycloak image (kept in step with deploy/compose.development.yml) under Testcontainers with the committed dev realm deploy/keycloak/flip-realm.json imported at boot, so register_user really creates a user, delete_user really removes one, and a token minted by Keycloak's password grant really passes verify_token. The same import proves the realm file's ${...} placeholders resolve. Separately, .github/workflows/local_auth_smoke.yml boots flip-db, keycloak, the RustFS object store and flip-api from the dev compose on a runner with no AWS credentials, then runs tests/local_auth_smoke.py (sign in as the seeded admin) and tests/local_storage_smoke.py (a model file through the store, FLIP#1291, plus repeated model-sized presigned POSTs, FLIP#1534) — the CI proof that the hub starts, authenticates and stores files without AWS.

Demo tooling (live stack)

Three developer utilities drive the running dev stack end to end (none run in CI):

• tests/demo_video.py (make demo-video from the repo root, make demo_video here) — records the scripted end-to-end demo video: plays the six segments in tests/demo_segments.py in one Playwright browser, performs the slow waits (cohort responses, imaging import, FL training) off-camera by reusing tests/e2e_smoke.py's wait functions, then assembles one mp4. Resumable via --project-id / --from-segment; --video-scale controls capture resolution (default 3 → 3840x2400).
• tests/seed_demo_projects.py (make seed_demo_projects) — seeds a curated catalogue of radiology projects in honest lifecycle states through the real API; --states phases the pull-heavy approved entries, --cleanup removes everything it recorded in tests/seed_demo_projects.json. Idempotent: a re-run skips catalogue entries whose project name already exists on the hub, so it never duplicates the real imaging imports.
• src/flip_api/scripts/create_demo_users.py (make create_demo_users, make demo-users from the repo root — runs inside the flip-api container) — provisions the demo users the recorder signs in as in the configured identity provider (the local Keycloak realm in development, the Cognito pool in staging/production; DEMO_RESEARCHER_PASSWORD / DEMO_ADMIN_PASSWORD / DEMO_TRUST_ADMIN_PASSWORD from env, never committed); restart flip-api afterwards so boot seeding grants their roles. Before any write it names the target (pool name/region/AWS account, or Keycloak realm + URL) and requires an interactive yes, so a stale AWS_COGNITO_USER_POOL_ID or wrong SSO account can't plant a known-password admin in an unintended pool.

aws_mock also pins AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN / AWS_DEFAULT_REGION to test-only stub values in the environment for the duration of the session, and clobbers any AWS_PROFILE from the developer's shell. Real-AWS credentials are never reachable while the fixture is active.

Run the full suite (lint + mypy + pytest, requires the dockerised dependencies):

make test

Further Reading

• Full FLIP Documentation
• Contributing & Development Guide