Skip to content

FINERACT-2485: Add idempotency to the new command processing (step 7/9) - #6913

Open
adamsaghy wants to merge 7 commits into
apache:developfrom
adamsaghy:FINERACT-2485/step-7-idempotency-module
Open

adamsaghy wants to merge 7 commits into
apache:developfrom
adamsaghy:FINERACT-2485/step-7-idempotency-module

Conversation

@adamsaghy

@adamsaghy adamsaghy commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Step 7 of 9 of FINERACT-2485 — splitting #6871 into reviewable pieces. Merge in step order.

This PR targets develop, so until the earlier steps merge it also shows their commits. Review only the top commit — the changes of this step: compare with step 6. Depends on #6912.

What this step does

The new fineract-command-idempotency module answers a repeated request the way the legacy processing does.

  • The key is read from the Idempotency-Key header (header only) and claimed before the command runs: only one of several concurrent requests executes.
  • Repeat of a completed command → its own result, read back and rendered by the same serializer. Repeat of a failed one → the original failure. Still running → 425 Too Early. Every replay carries x-served-from-cache: true. A key reused with a different request → 409.
  • A response filter captures answers that are not the command's own result; when it cannot store one it logs key, status, content type and size — never the body.
  • Scope via fineract.command.idempotency.custom-distinctor: ENTITY_NAME_AND_ACTION_NAME (default, e.g. LOAN|CREATE, as legacy) or NONE (unique per tenant). An application can supply its own CustomIdempotencyDistinctor.
  • The audit hooks move a claimed command out of UNDER_PROCESSING, so with idempotency on the application refuses to start if auditing, or its after or error hook, is off.

Configuration change: fineract.command.idempotency-key-header-name becomes fineract.command.idempotency.key-header-name (same FINERACT_IDEMPOTENCY_KEY_HEADER_NAME variable). Adds the command-auditing doc chapter and the new-processing section of the idempotency chapter.

The module structure, hooks and exceptions come from #6231 by Aleksandar Vidakovic (credited as co-author); this step adds replay from the result object, the response capture, the distinctor and the startup guard.

Verified

fineract-command-idempotency build with unit and end-to-end tests; provider idempotency tests; command module builds. Not verified on a running Fineract instance.

Stack

  1. FINERACT-2485: Pass a CommandContext to command hooks (step 1/9) #6907 Pass a CommandContext to command hooks
  2. FINERACT-2485: Make the Jackson 3 mapper agree with the Jersey wire format (step 2/9) #6908 Make the Jackson 3 mapper agree with the Jersey wire format
  3. FINERACT-2485: Retry every command handler (step 3/9) #6909 Retry every command handler
  4. FINERACT-2485: Record what a command acted on and what it did (step 4/9) #6910 Record what a command acted on and what it did
  5. FINERACT-2485: Claim idempotency keys in the command store (step 5/9) #6911 Claim idempotency keys in the command store
  6. FINERACT-2485: Make every command result readable from the command store (step 6/9) #6912 Make every command result readable from the command store
  7. FINERACT-2485: Add idempotency to the new command processing (step 7/9) #6913 Add idempotency to the new command processing ← this PR
  8. FINERACT-2485: Refuse a repeat whose claim never finished (step 8/9) #6914 Refuse a repeat whose claim never finished
  9. FINERACT-2485: Release the idempotency key of a recoverably failed command (step 9/9) #6915 Release the idempotency key of a recoverably failed command

Umbrella and history: #6871.

@Aman-Mittal Aman-Mittal left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the top commit. This is the step everything else was building towards and it holds together. Three inline questions: the skipHooks leak into the error phase, what the startup guard actually verifies, and malformed JSON on the capture path.

Things I traced and found right, recorded so they do not get re-litigated:

  • Hook ordering is correct and load-bearing. 10/11/12 core, 20 key header, 21 claim, 30 audit. The claim therefore writes the row before AuditCommandHookBefore would, and because claim sets command.setCommandId(...), the audit hook updates that row instead of inserting a second one. Reorder any of these and the unique constraint starts rejecting legitimate first requests. Worth saying so next to the constants, since the numbers live in three different modules.
  • Replaying through ctx.setResponse(...) rather than through captured text is the right call, and the reason given (the same serializer renders both, so they match because they are the same object, not because two mappers were made to agree) is the only version of this that survives contact with BigDecimal.
  • Reading the key from the header only is correct. A key in the query string would end up in access logs and proxy logs.
  • 425 Too Early for a live claim, 409 for a key reused with a different request matches the legacy behaviour, and matchesStoredRequest being checked before the ERROR and PROCESSED branches is the right order: a mismatched request must not get the stored failure either.

One question not tied to a line: step 2 reinstates MaskedValueSerializer for the Jackson 3 mapper, so a command carrying credentials stores them masked. If such a command ever reaches this framework, matchesStoredRequest compares masked text against masked text, and two different secrets that mask identically would count as the same request and be replayed. No command in the stack has a masked field today, so this is forward-looking, but it is the one place where masking and request matching pull against each other. Is the intent that masked fields are never part of a command payload?

adamsaghy and others added 7 commits November 14, 2026 16:48
Command hooks now receive a CommandContext - the command, its response, its error and its
state - instead of separate arguments, so a before hook can mark a command as handled
(skipExecution) and have the dispatcher return a response it supplied, and can stop the
remaining hooks (skipHooks). The synchronous, async and disruptor dispatchers all honour it;
the disruptor dispatcher hands each command a fresh future before publishing its slot.

Each command module gets constants for its properties, and the hook switches move under the
module they belong to (fineract.command.audit.hook-*-enabled,
fineract.command.hook-*-enabled). The audit module reports at startup which of its hooks are
on. The JDBC store is renamed to its Jdbc*/Json* types and stores a CommandContext. The
webhook command hooks are adapted to the new hook interface.

Step 1 of 9 of FINERACT-2485.

Co-authored-by: Aleksandar Vidakovic <aleks@apache.org>
…ormat

The command store runs on Jackson 3 while Jersey can only run on Jackson 2
(jersey-media-json-jackson 4.0.2 has no Jackson 3 support). Jackson3ObjectMapperConfig defines
the application's Jackson 3 mapper to produce the same JSON as the web mapper: the same date,
time, ExternalId and Month formats, NON_NULL inclusion, and the two date-time defaults Jackson 3
turned off (WRITE_DATES_AS_TIMESTAMPS, WRITE_DURATIONS_AS_TIMESTAMPS). java.time.Month is
written by name, because Jackson 3 writes it as a number that a Jackson 2 client reads back as
the wrong month. Credential masking is reapplied: the Jackson 2 MaskedValueSerializer annotation
is invisible to Jackson 3.

JacksonWireFormatEquivalenceTest pins the agreement, and Jackson3ObjectMapperWiringTest checks
that this mapper, not Spring Boot's default, is the one injected.

Step 2 of 9 of FINERACT-2485.
UpdateWorkingDaysCommandHandler was the only command handler executing without a retry, so it
gave up on the first contention where every other command tries again. It gets its own
commandWorkingDaysUpdate instance, configured like the others.

The product mix and floating rate handlers named retry instances that were never configured,
so they fell back to resilience4j's defaults and retried on every exception. Their instances
are configured like the others, and the unused commandMixTaxonomyMappingUpdate instance, left
behind when the mix module was removed, is dropped.

CommandHandlerRetryCoverageTest walks every CommandHandler on the classpath and fails if one is
added without a retry, or names a retry instance that is not configured.

Step 3 of 9 of FINERACT-2485.
Every command declares its entity and action names, in the vocabulary of m_permission, and
m_command stores them in two new NOT NULL columns (rows written before are backfilled as
UNKNOWN). The audit search and the checker inbox are defined in these terms, and maker-checker
derives the approval permission from them, so a row without them is invisible to both.

Declared as abstract methods on Command rather than an optional interface: a command that
forgot them would be stored with none. The names follow m_permission, which is not always what
the class name suggests (CACHE/UPDATE, BUSINESS_DATE, COLLATERAL_PRODUCT, CHARTSLAB, MEETING for
attendance); notes and images compute the entity from their payload. HookableCommand becomes a
marker, and a command read back from the store is a StoredCommand carrying the recorded names.

Step 4 of 9 of FINERACT-2485.
The command store can now claim an idempotency key atomically and report what was stored for
it, which the idempotency module in the next step builds on.

- m_command.custom_idempotency_distinctor (NOT NULL, default '') holds what a key is unique
  together with, and a unique constraint on (idempotency_key, custom_idempotency_distinctor)
  makes a claim atomic. Rows from earlier builds that share a key keep the earliest one;
  the others get their id appended, so no record is deleted.
- m_command.response_status_code records the status of an answer that is not the command's
  own result (a failure, or an answer that is not JSON).
- CommandStore gains claim(), getReplayByKey(), matchesStoredRequest() and recordResponse();
  every lookup by key also takes the distinctor, and IdempotencyKeyLookupCoverageTest fails the
  build on one that does not. A stored result that cannot be read back fails loudly instead of
  answering null.
- Writes of the command row are retried on transient and recoverable data-access failures; a
  constraint violation is not retried, because the row is already there.

Until the idempotency module is added, a second request with a key that is already stored is
rejected by the constraint instead of being executed again.

Step 5 of 9 of FINERACT-2485.

Co-authored-by: Aleksandar Vidakovic <aleks@apache.org>
A repeat of an idempotency key is answered by reading the command's stored result back into
the object it was. Four collateral command results (client collateral create and update,
collateral product update, loan collateral delete) had a builder but no constructor Jackson
could use, so they could be stored but not read back. The collateral responses get no-args and
all-args constructors.

CommandHandlerResultRehydrationTest walks every CommandHandler on the classpath and reads its
result type back the way a replay does, with every nested type present, so a new result type
cannot regress this silently. CommandReplayRehydrationTest checks that a result read back
renders byte for byte as the original.

Step 6 of 9 of FINERACT-2485.
The new fineract-command-idempotency module answers a repeated request the way the legacy
processing does:

- The key is read from the Idempotency-Key header (header only, never a query parameter) and
  claimed before the command runs, so only one of several concurrent requests executes.
- A repeat of a completed command is answered with the command's own result, read back from
  the store and rendered by the same serializer; a repeat of a failed one with the original
  failure; a repeat of one still running with 425 Too Early. Every replay carries
  x-served-from-cache: true. A key reused with a different request is refused with 409.
- An answer that is not the command's own result is captured by a response filter, which logs
  the key, status, content type and size of an answer it cannot store, never its body.
- A key is unique together with a distinctor, chosen by
  fineract.command.idempotency.custom-distinctor: ENTITY_NAME_AND_ACTION_NAME (default, e.g.
  LOAN|CREATE) scopes it as the legacy processing does, NONE makes it unique per tenant.
- The audit hooks move a claimed command out of UNDER_PROCESSING, so an application with
  idempotency on refuses to start with auditing, or its after or error hook, switched off.

The key header moves from fineract.command.idempotency-key-header-name to
fineract.command.idempotency.key-header-name, keeping FINERACT_IDEMPOTENCY_KEY_HEADER_NAME.
New command-auditing chapter; the idempotency chapter describes the new processing.

Step 7 of 9 of FINERACT-2485.

Co-authored-by: Aleksandar Vidakovic <aleks@apache.org>
@adamsaghy
adamsaghy force-pushed the FINERACT-2485/step-7-idempotency-module branch from a5690b9 to 7730742 Compare November 14, 2026 16:54
@adamsaghy

Copy link
Copy Markdown
Contributor Author

Two points from the review summary:

  • Hook ordering: now spelled out next to the constants in IdempotencyCommandConstants. It lists 10-12 core, 20/21 here and 30 audit, and why the claim has to come before the audit before hook.
  • Masking vs matchesStoredRequest: no command in the framework carries a masked field today. If one ever does, comparing masked text would be wrong, as you say: two secrets that mask the same would match. That command would need its request compared on something unmasked, e.g. a hash, before it could take an idempotency key. I'd rather handle that when such a command appears than design for it now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants