Skip to content

FINERACT-2485: Claim idempotency keys in the command store (step 5/9) - #6911

Open
adamsaghy wants to merge 5 commits into
apache:developfrom
adamsaghy:FINERACT-2485/step-5-store-claim
Open

adamsaghy wants to merge 5 commits into
apache:developfrom
adamsaghy:FINERACT-2485/step-5-store-claim

Conversation

@adamsaghy

@adamsaghy adamsaghy commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Step 5 of 9 of FINERACT-2485 — splitting #6871 into reviewable pieces. Merge in step order.

This PR targets develop, so until the earlier steps merge it also shows their commits. Review only the top commit — the changes of this step: compare with step 4. Depends on #6910.

What this step does

The command store learns to claim an idempotency key atomically and to report what was stored for it. Nothing calls this yet; step 7 does.

  • 0004_idempotency_key_unique.xml: adds custom_idempotency_distinctor (NOT NULL, default '', what a key is unique together with) and a unique constraint on (idempotency_key, custom_idempotency_distinctor). Rows from earlier builds sharing a key keep the earliest; the others get their id appended — no record is deleted.
  • 0005_command_response_cache.xml: response_status_code, for an answer that is not the command's own result.
  • CommandStore gains claim(), getReplayByKey(), matchesStoredRequest() and recordResponse(); every lookup by key also takes the distinctor (IdempotencyKeyLookupCoverageTest fails the build otherwise). A stored result that cannot be read back fails loudly instead of returning null.
  • Writes of the command row are retried on transient and recoverable data-access failures; constraint violations are not retried. Only the commandStore retry instance changes.

Interim behaviour until step 7 merges: a second request carrying a key that is already stored is rejected by the constraint instead of being executed again.

The claim/unique-key design comes from #6231 by Aleksandar Vidakovic (credited as co-author).

Verified

fineract-command-jdbc build with tests, including the migration test on PostgreSQL, MySQL and MariaDB (Testcontainers) and the store tests; Liquibase registration and DDL-safety checks. Not verified on a running Fineract instance.

Stack

  1. FINERACT-2485: Pass a CommandContext to command hooks (step 1/9) #6907 Pass a CommandContext to command hooks
  2. FINERACT-2485: Make the Jackson 3 mapper agree with the Jersey wire format (step 2/9) #6908 Make the Jackson 3 mapper agree with the Jersey wire format
  3. FINERACT-2485: Retry every command handler (step 3/9) #6909 Retry every command handler
  4. FINERACT-2485: Record what a command acted on and what it did (step 4/9) #6910 Record what a command acted on and what it did
  5. FINERACT-2485: Claim idempotency keys in the command store (step 5/9) #6911 Claim idempotency keys in the command store ← this PR
  6. FINERACT-2485: Make every command result readable from the command store (step 6/9) #6912 Make every command result readable from the command store
  7. FINERACT-2485: Add idempotency to the new command processing (step 7/9) #6913 Add idempotency to the new command processing
  8. FINERACT-2485: Refuse a repeat whose claim never finished (step 8/9) #6914 Refuse a repeat whose claim never finished
  9. FINERACT-2485: Release the idempotency key of a recoverably failed command (step 9/9) #6915 Release the idempotency key of a recoverably failed command

Umbrella and history: #6871.

@Aman-Mittal Aman-Mittal left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the top commit. The claim design is right: an insert against a unique constraint is the only way to settle a race between concurrent requests, and catching the violation rather than pre-checking is what makes it sound. Two inline questions, one about the retry around claim, one about the store surface step 1 introduced.

A few things I checked and found correct, worth recording so nobody re-derives them:

  • The text comparison in matchesStoredRequest is safe. request and response are text, not jsonb, so nothing normalises key order or numeric scale behind your back, and JsonNodeWritingConverter writes with the same injected mapper you compare against. On jsonb this would have compared re-ordered keys and refused every legitimate repeat.
  • The migration is portable. left(), concat() and ROW_NUMBER() OVER all exist on PostgreSQL, MySQL and MariaDB, and the derived table does avoid MySQL error 1093.
  • Partitioning the dedup by idempotency_key alone is correct here, because every pre-existing row gets '' for the new distinctor in the preceding changeset, so key and (key, distinctor) partition identically. That is only true at this point in history, which is worth the comment it already has.
  • Nullable idempotency_key under the unique constraint still permits many rows with no key, on all three databases.

One request on the changelog: the comment on command-00007 explains why the non-unique index from 0002 is deliberately left in place, which is good, but nothing tracks the removal. Could that get a JIRA ticket referenced in the comment, so the redundant index does not outlive everyone who remembers why it is there?

adamsaghy and others added 5 commits November 14, 2026 16:48
Command hooks now receive a CommandContext - the command, its response, its error and its
state - instead of separate arguments, so a before hook can mark a command as handled
(skipExecution) and have the dispatcher return a response it supplied, and can stop the
remaining hooks (skipHooks). The synchronous, async and disruptor dispatchers all honour it;
the disruptor dispatcher hands each command a fresh future before publishing its slot.

Each command module gets constants for its properties, and the hook switches move under the
module they belong to (fineract.command.audit.hook-*-enabled,
fineract.command.hook-*-enabled). The audit module reports at startup which of its hooks are
on. The JDBC store is renamed to its Jdbc*/Json* types and stores a CommandContext. The
webhook command hooks are adapted to the new hook interface.

Step 1 of 9 of FINERACT-2485.

Co-authored-by: Aleksandar Vidakovic <aleks@apache.org>
…ormat

The command store runs on Jackson 3 while Jersey can only run on Jackson 2
(jersey-media-json-jackson 4.0.2 has no Jackson 3 support). Jackson3ObjectMapperConfig defines
the application's Jackson 3 mapper to produce the same JSON as the web mapper: the same date,
time, ExternalId and Month formats, NON_NULL inclusion, and the two date-time defaults Jackson 3
turned off (WRITE_DATES_AS_TIMESTAMPS, WRITE_DURATIONS_AS_TIMESTAMPS). java.time.Month is
written by name, because Jackson 3 writes it as a number that a Jackson 2 client reads back as
the wrong month. Credential masking is reapplied: the Jackson 2 MaskedValueSerializer annotation
is invisible to Jackson 3.

JacksonWireFormatEquivalenceTest pins the agreement, and Jackson3ObjectMapperWiringTest checks
that this mapper, not Spring Boot's default, is the one injected.

Step 2 of 9 of FINERACT-2485.
UpdateWorkingDaysCommandHandler was the only command handler executing without a retry, so it
gave up on the first contention where every other command tries again. It gets its own
commandWorkingDaysUpdate instance, configured like the others.

The product mix and floating rate handlers named retry instances that were never configured,
so they fell back to resilience4j's defaults and retried on every exception. Their instances
are configured like the others, and the unused commandMixTaxonomyMappingUpdate instance, left
behind when the mix module was removed, is dropped.

CommandHandlerRetryCoverageTest walks every CommandHandler on the classpath and fails if one is
added without a retry, or names a retry instance that is not configured.

Step 3 of 9 of FINERACT-2485.
Every command declares its entity and action names, in the vocabulary of m_permission, and
m_command stores them in two new NOT NULL columns (rows written before are backfilled as
UNKNOWN). The audit search and the checker inbox are defined in these terms, and maker-checker
derives the approval permission from them, so a row without them is invisible to both.

Declared as abstract methods on Command rather than an optional interface: a command that
forgot them would be stored with none. The names follow m_permission, which is not always what
the class name suggests (CACHE/UPDATE, BUSINESS_DATE, COLLATERAL_PRODUCT, CHARTSLAB, MEETING for
attendance); notes and images compute the entity from their payload. HookableCommand becomes a
marker, and a command read back from the store is a StoredCommand carrying the recorded names.

Step 4 of 9 of FINERACT-2485.
The command store can now claim an idempotency key atomically and report what was stored for
it, which the idempotency module in the next step builds on.

- m_command.custom_idempotency_distinctor (NOT NULL, default '') holds what a key is unique
  together with, and a unique constraint on (idempotency_key, custom_idempotency_distinctor)
  makes a claim atomic. Rows from earlier builds that share a key keep the earliest one;
  the others get their id appended, so no record is deleted.
- m_command.response_status_code records the status of an answer that is not the command's
  own result (a failure, or an answer that is not JSON).
- CommandStore gains claim(), getReplayByKey(), matchesStoredRequest() and recordResponse();
  every lookup by key also takes the distinctor, and IdempotencyKeyLookupCoverageTest fails the
  build on one that does not. A stored result that cannot be read back fails loudly instead of
  answering null.
- Writes of the command row are retried on transient and recoverable data-access failures; a
  constraint violation is not retried, because the row is already there.

Until the idempotency module is added, a second request with a key that is already stored is
rejected by the constraint instead of being executed again.

Step 5 of 9 of FINERACT-2485.

Co-authored-by: Aleksandar Vidakovic <aleks@apache.org>
@adamsaghy
adamsaghy force-pushed the FINERACT-2485/step-5-store-claim branch from cfc6857 to 14748f6 Compare November 14, 2026 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants