Repository navigation
[WIP] Timeout parameter for READ_LINEAR, async for in Python - #1960
Open
franzpoeschel wants to merge 8 commits into
Open
franzpoeschel wants to merge 8 commits into
franzpoeschel wants to merge 8 commits into
Conversation
added 4 commits
October 27, 2026 11:26
Linear reads of a Series whose writer was killed wait forever for a step that never comes. Add an explicit, timeout-aware stepping API to the Snapshots/OpaqueSeriesIterator machinery, keeping regular for loops blocking and unchanged: AdvanceStatus::TRY_AGAIN no step within the timeout, stream still open OpaqueSeriesIterator::tryNext(float) the only timeout point OpaqueSeriesIterator::next() synonym for operator++() Snapshots::beginLazy() do not open the first step yet tryNext() distinguishes a timeout (TRY_AGAIN) from the end of the data (OVER) and never closes the reader on a timeout. The iterator stays usable and can simply try again. A negative timeout is an API misuse and throws. The timeout is passed explicitly through the internal advance calls (ADIOS2File/Series/Iteration), so no Series-level state or public Series API is added. Backends without IO steps behave like their blocking calls; random-access workflows report RANDOMACCESS. The lazy constructor and a retry-aware initSeriesInLinearReadMode() allow the very first step to be opened by tryNext() with a timeout, while not redoing file opening or metadata parsing when retrying.
Keep a writer and reader open at the same time in a single thread and let the reader start before the writer has written the next step. The reader uses beginLazy() and tryNext() and deterministically runs into a TRY_AGAIN between two steps, then succeeds when the writer adds the step. Also check that TRY_AGAIN is not treated as end of data, that closing the writer yields OVER, and that next() is a blocking synonym for operator++().
Reuse the existing StatefulIterator Python adaptor instead of adding a
new iterator class:
- bind AdvanceStatus (including try_again),
- bind Snapshots.begin_lazy(), try_next() and current(),
- add AsyncIterator.py, following the existing monkey-patch style, and
install __aiter__/__anext__ on StatefulIterator in __init__.py,
- ship AsyncIterator.py via the copy_aux_py() list.
The minimal __anext__ polls the non-blocking try_next(0) and yields to
the event loop between attempts; no executor or GIL release for blocking
calls is needed for this scope.
Do not store the last advance status or initialization progress on the
iterator. Instead:
- pass the advance status through seek()/loopBody()/nextStep() by
reference while tryNext() runs, so operator++() stays unchanged;
- use the existing CurrentStep::Before_t as the marker for a lazily
created iterator that has not been initialized yet;
- leave a timed-out iterator either in Before_t (first step) or in
During_t with an empty iteration_idx (between steps), which are the
states it already has when waiting for a next step.
This removes m_lastAdvance and the three initialization flags. A timeout
during initialization is simply retried from scratch; opening the file
again is idempotent and the step parsing only runs after a successful
BeginStep. Callers must use the status returned by tryNext() (not
is_end()) to distinguish TRY_AGAIN from OVER. The test no longer asserts
that a timed-out iterator differs from end().
franzpoeschel
force-pushed
the
topic-linear-read-timeout
branch
from
October 27, 2026 10:30
46c7967 to
d3c55f6
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| components_without_extent::components_without_extent(); | ||
| } | ||
|
|
||
| TEST_CASE("read_linear_try_next", "[serial][adios2]") |
Mark tryNext() overrides, document stepTimeout for Doxygen, merge duplicate switch branches, mark the Python adaptor's current() as nodiscard, and avoid the NVHPC 25.11 std::visit crash when initializing deferred snapshot iterators.
Defer engine opening in READ_LINEAR mode so that the first BeginStep() can honor tryNext()'s timeout. A timed-out engine is parked in m_uninitializedEngine and resumed on the next call instead of being reopened. Add a pseudoserial SST test where the writer keeps its first step open until the reader's timed call returns.
no need to fully slop this up
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.