Repository navigation
Conversation
d002637 to
983f239
Compare
galovics
left a comment
There was a problem hiding this comment.
Thanks @adamsaghy, I know this is a draft on top of #6231 so I'm not treating it as merge-ready, but a few things I'd like us to settle before any of it goes back into #6231.
The replay-from-the-result-object approach makes sense to me, and the startup failure for idempotency without auditing is a good call. My main worry is that the same "425 forever" problem you're guarding against at startup still exists at runtime, plus the retry change reaches way outside idempotency. Details inline.
fineract-command-idempotency/src/main/java/org/apache/fineract/command/idempotency/hook/IdempotencyCheckCommandHook.java:88
+ if (UNDER_PROCESSING.equals(replay.state())) {
+ // the original has not finished, so there is nothing to replay yet and retrying is the right thing to do
+ throw new CommandIdempotencyInProgressException(key, replay.response());
+ }The PR refuses to start with auditing off because a claimed key would stay UNDER_PROCESSING forever. But the same thing happens at runtime: the pod dies between claim and the after hook, or store(ctx) runs out of retries and goes into the fallback (DLQ file or just a log line). The row stays UNDER_PROCESSING and every retry with that key gets 425 forever, and a client who follows the docs ("retry, it's still running") will retry forever. I don't see any lease/timeout on the claim. Don't we need one, e.g. a claimed-at timestamp and after a configurable period treat the row as abandoned (either let it be re-claimed or answer with a definite error)? I'm not convinced we can ship this without it. Thoughts?
fineract-provider/src/main/resources/application.properties:603
+resilience4j.retry.configs.default.retryExceptions=${FINERACT_RETRY_EXCEPTIONS:org.springframework.dao.TransientDataAccessException,org.springframework.dao.RecoverableDataAccessException,java.sql.SQLTransientException,java.sql.SQLRecoverableException,...}
...
-resilience4j.retry.instances.processJobDetailForExecution.retryExceptions=${FINERACT_PROCESS_JOB_DETAIL_RETRY_EXCEPTIONS:...}Two things here. First, this isn't only the command handlers anymore. processJobDetailForExecution, recalculateInterest and postInterest inherit the new default too, so we're now retrying COB/interest posting on RecoverableDataAccessException / SQLRecoverableException. A connection dropped during commit is exactly the case where we don't know whether the transaction went through, and retrying an interest posting there is how you get it twice. I'd keep the widened list to the command store at least, or leave the recoverable ones out entirely. Second, all the per-instance FINERACT_*_RETRY_EXCEPTIONS env vars are gone. Anyone who set them in their deployment silently loses that config on upgrade. Can we keep them working, or at least call this out as a breaking change in the docs? Also I'd love this to be its own PR, it has very little to do with idempotency.
fineract-command-idempotency/src/main/java/org/apache/fineract/command/idempotency/web/IdempotencyResponseCaptureFilter.java:99
+ log.error("Unable to record the response for idempotency key {} (status {}); the answer was: {}", key, wrapper.getStatus(),
+ body, e);Woah, I'd be very careful with this. We're dumping the whole response body into the logs, and same at line 121 for non-JSON answers. These are financial API responses, they can carry client data, and the whole point of the masking work in this PR is to keep that kind of thing out of places it shouldn't be. Let's log the key, status, content type and size, not the body.
fineract-command-jdbc/src/main/resources/db/changelog/tenant/module/command/parts/0005_command_response_cache.xml:25
+ <changeSet author="fineract" id="command-00009" context="postgresql,mysql">
+ <comment>
+ A repeated idempotency key has to be answered exactly as the original request was answered, which means
+ replaying the stored response with the status code it was first sent with.Minor: the files go 0003 -> 0005 and the IDs go command-00005 -> command-00009. I assume these are leftovers from #6231's history, but let's make them sequential (0004 / command-00006) while nothing has been released. The comment is also stale now, with this follow-up the status code is only stored for failures and non-JSON answers.
…interface The hook interfaces now take a CommandContext rather than the command and its result as separate arguments. WebhookCommandHookAfter and WebhookCommandHookError landed on develop against the old signatures, so they are adapted here; the after hook still reads the response before registering the synchronization, so a webhook reports the result as it was when the command finished.
… wire format Commands are persisted on Jackson 3 while Jersey can only run on Jackson 2, so the two mappers have to produce the same bytes. Three Jackson 3 defaults differ and are pinned by tests: WRITE_DATES_AS_TIMESTAMPS and WRITE_DURATIONS_AS_TIMESTAMPS are off, and java.time.Month is written as a number with no flag to restore the name - a Jackson 2 client reads that back as an ordinal and gets the wrong month. It also reapplies credential masking. @JsonSerialize(using = MaskedValueSerializer.class) is a Jackson 2 annotation and invisible to Jackson 3, so without this external-service credentials would reach the command table in the clear.
A repeat of a completed command is answered by rehydrating the stored result and handing it back through the dispatcher, so the resource and the web layer render it exactly as they rendered the original. Nothing about the original response has to be recorded for the two to match, which removes the stored status code for successful commands. The response capture now fills gaps only, enforced in SQL (response IS NULL): the status and the rendered answer are recorded only for a command that produced no result of its own. A stored result that cannot be reconstructed is answered 409 stating why, rather than a 500 or an empty 200, and an answer that cannot be stored is logged so that it stays recoverable. Every write to the command row is retried, including claiming the key and recording the answer. The retry policy is defined once rather than repeated across 47 instances, and covers everything non-deterministic while ignoring constraint violations explicitly. Attempt tracking is removed.
The audit hooks own the command row: they move a claimed command out of UNDER_PROCESSING and store its result. With auditing off, a claimed key would stay UNDER_PROCESSING for ever and every repeat would be answered 425 - visible only from the second request. An application configured that way now refuses to start. Both features default to enabled.
A command is recorded as UNDER_PROCESSING from the moment its key is claimed until the audit hooks record how it finished. If nothing ever does - the process dies in between, or the store gives up writing the row after its retries - the row stays that way for good, and answering 425 Too Early to every repeat tells the caller to come back for an answer that is never coming. A caller following the documented advice then retries for ever. Past fineract.command.idempotency.claim-timeout such a claim is treated as abandoned and the repeat is refused with 409, saying why. Age is the only signal available: nothing in the row distinguishes a command that died from one that is merely slow, so the default is generous and the setting is documented as one to widen rather than tighten. The key is refused, not released. The command that claimed it may have reached the database before whatever stopped it, so running it again is the duplicate this mechanism exists to prevent; establishing whether it took effect is a separate matter, and a retry needs a new key. No schema change: created_at is written when the key is claimed, and the audit hooks only ever touch updated_at and executed_at.
983f239 to
feb9f2f
Compare
|
Thanks for the review @galovics — all four addressed in the latest push. Stale claims. You're right, and the startup guard only covered the configuration case, not the runtime one. Added a claim timeout ( Deliberately refused and not re-claimed. The command that claimed it may have reached the database before whatever stopped it, so running it again is exactly the duplicate this mechanism exists to prevent; whether it took effect has to be established separately, and a retry needs a new key. Age is the only signal available — nothing in the row distinguishes a dead command from a slow one — so the default is generous and documented as a setting to widen rather than tighten. No schema change was needed: Retry. Took your suggestion. Logging the body. Agreed, that was a bad trade — recoverability is not worth putting response bodies in the logs on this API. Both sites now log the key, status, content type and size only. The tests assert the body is absent and the metadata present, so this can't quietly come back. Numbering. Renumbered to |
m_command carries approved_at and approved_by_username but nothing that says which entity and action a command concerns, so the approval columns cannot be used: maker-checker derives the permission governing approval by joining the two names with _CHECKER, the way the legacy processing does. The audit search and the checker inbox are defined in the same terms - the inbox returns the actionNames and entityNames a requestor may approve - so a row without them is invisible to both. This adds entity_name and action_name, and the means of filling them. Declared as abstract methods on Command rather than an optional interface. The interface shape is what left 44 of 47 commands without names; a command that forgets them would be stored with none, and a row no audit or inbox can find is worse than a compile error. The columns are therefore NOT NULL, with rows written before the change backfilled as UNKNOWN. The names are the ones in m_permission, checked against the 953 pairs the seed data defines, and they are not always what the class name suggests: CacheSwitchCommand is CACHE/UPDATE, the entity is BUSINESS_DATE rather than BUSINESSDATE, collateral is COLLATERAL_PRODUCT and CLIENT_COLLATERAL_PRODUCT, chart slabs are CHARTSLAB, and attendance is an action on MEETING. Notes and images act on different entities depending on the request - a note belongs to whatever it is attached to, an image to a client or a member of staff - so those compute the name from the payload, which is set before the command is stored. Two consequences worth naming. HookableCommand loses its two methods and becomes a marker: they duplicated what every command now declares, and it only ever meant "notify webhooks". And an abstract Command cannot be the target of the mapper's read direction, so a row read back becomes a StoredCommand carrying the recorded names - there is no class left to ask. The migration test rolls back a fixed number of changesets, which has to grow with every part added after 0003, or it leaves the unique constraint in place and the re-seed collides.
…lay gaps An idempotency key is now unique together with a distinctor, stored beside it in the new NOT NULL column m_command.custom_idempotency_distinctor (default ''). The composite unique constraint replaces the key-only one; it is added before the old one is dropped, so the key is never left unconstrained. The configured CustomIdempotencyDistinctor computes the value when the key is claimed: ENTITY_NAME_AND_ACTION_NAME (the default, e.g. LOAN|CREATE) scopes a key the way the legacy processing does, NONE makes it unique per tenant. Every lookup by key - claim, replay, request comparison, recorded response - also matches the distinctor, and IdempotencyKeyLookupCoverageTest fails the build on a lookup that does not. Review fixes on the same path: - Four collateral command results had no constructor Jackson could use, so a repeat of those requests was refused with 409 instead of replayed. They are now constructible, and CommandHandlerResultRehydrationTest reads every handler's result type back the way a replay does. The 409 for a result that cannot be rebuilt no longer invites a retry under a new key: the command did take effect. - Idempotency refuses to start with the audit after or error hook switched off, not only with auditing off: without them a command never leaves UNDER_PROCESSING. - A claim is always stored with its time, even with the timestamp hook off, so the claim timeout can tell an abandoned claim from a running one.
…mmand The legacy command processing lets a retry under the same key execute again after a recoverable failure (FINERACT-2985): an exception marked RecoverableError, or one whose error codes are all in the idempotency-recoverable-error-codes global configuration. The new command processing replayed every failure for ever, so a client retrying a transient failure on a migrated endpoint could not get past it. A command that fails recoverably now moves its record to <key>_RE<id>, the scheme the legacy processing uses, hashing a key too long to take the suffix. The original key is free, so the retry is executed as a new command, and the failure stays on record under the moved key with the answer the client was sent. If the moved key is taken or the move fails, the failure stays under the original key, as for any other failure. The decision sits behind RecoverableCommandErrorClassifier in fineract-command, so the command modules stay independent of fineract-core; fineract-core supplies an adapter that delegates to the legacy RecoverableErrorClassifier. Without a classifier, nothing is recoverable. The key is moved on the command before the audit error hook runs, because that hook rewrites the whole row from the command.
The legacy command processing answers a repeat of an unfinished command 425 Too Early for as long as it stays unfinished. The new command processing turns that into 409 after fineract.command.idempotency.claim-timeout, so that a claim abandoned by a dead process does not answer 425 for ever. The timeout stays on by default, at 10 minutes. Setting it to 0, or leaving it empty, now switches it off and restores the legacy answer, for deployments whose clients rely on it. A negative value is refused at startup: a negative age is always exceeded, so even a command that is still running would be declared abandoned.
Umbrella — not for merging. This draft has been split into nine smaller PRs, one concern each, to make review manageable. Please review and merge those, in this order:
Each step is one commit on top of the previous one and builds and passes its tests on its own. Every PR targets
develop, so it also shows the commits of the earlier steps until they merge; each description links a compare view of that step's own changes.Relationship to this PR and to #6231
m_commandLiquibase parts are renumbered in step order (0003entity/action names,0004distinctor column and unique key,0005response columns; the separate distinctor migration is folded into0004), a stray comment and blank lines inapplication.propertiesare removed, andCustomIdempotencyDistinctorTypeno longer holds aSupplierfield, which error-prone'sImmutableEnumCheckerrejects in a full build. That last fix is only in the stack, not on this branch.Known gaps, unchanged