Introduce more flexible storeChunk() syntax, use to add ADIOS2 memory selection

.github/ci/spack-envs/gcc13_py312_mpich_h5_ad2/spack.yaml

@@ -6,7 +6,10 @@
 #
 spack:
   specs:
-  - adios2@2.10 +mpi
+  # Pin the oldest ADIOS2 release that still lacks resetting of memory
+  # selections (added upstream in 2.11.0, backported to 2.10.1). This exercises
+  # the code path that rejects memory selections for such versions.
+  - adios2@2.10.0 +mpi
   - hdf5 +mpi
   - mpich
 

CMakeLists.txt

@@ -408,6 +408,7 @@ set(CORE_SOURCE
         src/Format.cpp
         src/Iteration.cpp
         src/IterationEncoding.cpp
+        src/LoadStoreChunk.cpp
         src/Mesh.cpp
         src/ParticlePatches.cpp
         src/ParticleSpecies.cpp
@@ -419,6 +420,7 @@ set(CORE_SOURCE
         src/version.cpp
         src/auxiliary/Date.cpp
         src/auxiliary/Filesystem.cpp
+        src/auxiliary/Future.cpp
         src/auxiliary/JSON.cpp
         src/auxiliary/JSONMatcher.cpp
         src/auxiliary/Memory.cpp
@@ -858,6 +860,12 @@ if(openPMD_BUILD_TESTING)
                 test/Files_SerialIO/components_without_extent.cpp
                 test/Files_SerialIO/read_while_writing.cpp
                 test/Files_SerialIO/file_state_lifetime.cpp
+                test/Files_SerialIO/deferred_load_unrelated_flush.cpp
+                test/Files_SerialIO/container_buffer_size.cpp
+                test/Files_SerialIO/buffer_config_single_use.cpp
+                test/Files_SerialIO/builder_sentinels.cpp
+                test/Files_SerialIO/joined_dim_buffer_api.cpp
+                test/Files_SerialIO/memory_selection.cpp
             )
         elseif(${test_name} STREQUAL "ParallelIO" AND openPMD_HAVE_MPI)
             list(APPEND ${out_list}

docs/source/details/mpi.rst

@@ -31,6 +31,8 @@ Functionality                Behavior           Description
 ``::makeConstant`` [3]_      *backend-specific* declare, write
 ``::storeChunk`` [1]_        independent        write
 ``::loadChunk``              independent        read
+``::prepareLoadStore()``     independent        configure deferred I/O
+``DeferredComputation`` [5]_ **collective**     deferred store/load + flush
 ``::availableChunks`` [4]_   collective         read, immediate result
 ============================ ================== ================================
 
@@ -49,6 +51,14 @@ Functionality                Behavior           Description
 .. [4] We usually open iterations delayed on first access. This first access is usually the ``flush()`` call after a ``storeChunk``/``loadChunk`` operation. If the first access is non-collective, an explicit, collective ``Iteration::open()`` can be used to have the files already open.
        Alternatively, iterations might be accessed for the first time by immediate operations such as ``::availableChunks()``.
 
+.. [5] The experimental deferred I/O API (``RecordComponent::prepareLoadStore()`` followed by ``store()`` / ``load()``) returns handles of type ``auxiliary::DeferredComputation``.
+       By default, invoking such a handle via ``get()`` / ``operator()()`` triggers the automatic flush of the underlying Series (the data operation itself is enqueued by ``store()`` / ``load()``).
+       Since ``Series::flush()`` is collective, this makes invoking the handle a collective operation: every rank must invoke its handles in a consistent order, even if the number of ``store()`` / ``load()`` calls differs between ranks.
+       The per-``RecordComponent`` flush counter is used to skip a flush if the component has already been flushed by another component in the meantime, so a given invocation does not necessarily flush.
+       The returned handles are marked ``[[nodiscard]]`` and the automatic flush is only triggered by an explicit invocation, so handles cannot be dropped unnoticed.
+       Call ``unsafeNoAutomaticFlush()`` on the configuration object to disable the automatic flush and instead call ``Series::flush()`` explicitly and collectively at a suitable point.
+       The destructor of a still-valid handle does not flush; it only prints a warning. Buffers passed to ``store()`` / ``load()`` must stay valid until the next flush point unless owned by the operation.
+
 .. warning::
 
   The openPMD-api will by default flush only those Iterations which are dirty, i.e. have been written to.

docs/source/dev/design.rst

@@ -23,7 +23,7 @@ Therefore, enabling users to handle hierarchical, self-describing file formats w
 
 .. literalinclude:: IOTask.hpp
    :language: cpp
-   :lines: 50-81
+   :lines: 57-91
 
 Every task is designed to be a fully self-contained description of one such atomic operation. By describing a required minimal step of work (without any side-effect), these operations are the foundation of the unified handling mechanism across suitable file formats.
 The actual low-level exchange of data is implemented in ``IOHandlers``, one per file format (possibly two if handlingi MPI-parallel work is possible and requires different behaviour).

docs/source/usage/workflow.rst

@@ -127,6 +127,16 @@ Flush points are triggered by:
     Flush point guarantees affect only the corresponding iteration.
 *   Calling ``Writable::seriesFlush()`` or ``Attributable::seriesFlush()``.
 *   The streaming API (i.e. ``Series.readIterations()`` and ``Series.writeIteration()``) automatically before accessing the next iteration.
+*   Invoking a handle returned by the experimental deferred I/O API (``RecordComponent::prepareLoadStore()`` followed by ``store()`` / ``load()``) via ``get()`` / ``operator()()``, unless automatic flushing was disabled via ``unsafeNoAutomaticFlush()``.
+
+.. note::
+
+    The automatic flush performed by the deferred I/O handles is MPI-collective (see :ref:`details-mpi`).
+    Each rank must invoke such a handle the same number of times and in the same order, *even if* the number of ``store()`` / ``load()`` calls differs per rank.
+    The flush is skipped if the pertaining ``RecordComponent`` has already been flushed by another component in the meantime (tracked via per-component flush counters), so a given invocation does not necessarily flush.
+    To avoid this coupling, disable the automatic flush with ``unsafeNoAutomaticFlush()`` and call ``Series::flush()`` explicitly at a collective point.
+    The handles are returned with the ``[[nodiscard]]`` attribute and the flush is only triggered by explicitly invoking them; destroying a still-valid handle skips the flush and prints a warning.
+    Note that ``store()`` / ``load()`` enqueue immediately, so buffers not owned by the operation must stay valid until the next flush point.
 
 Attributes are (currently) unaffected by this:
 
@@ -142,8 +152,8 @@ Attributes are (currently) unaffected by this:
     For user-guided selection of such implementations, ``Series::flush`` and ``Attributable::seriesFlush()`` take an optional JSON/TOML string as a parameter.
     See the section on :ref:`backend-specific configuration <backendconfig>` for details.
 
-Deferred Data API Contract
---------------------------
+Verbose Logging
+---------------
 
 A verbose debug log can optionally be printed to the standard error output by specifying the environment variable ``OPENPMD_VERBOSE=1``.
 Note that this functionality is at the current time still relatively basic.

include/openPMD/Dataset.hpp

@@ -34,6 +34,18 @@ namespace openPMD
 using Extent = std::vector<std::uint64_t>;
 using Offset = std::vector<std::uint64_t>;
 
+/** Selection of a region of memory for storing chunks.
+ *
+ * Used to specify a non-contiguous memory region when storing
+ * data chunks. This allows writing data that is not contiguous
+ * in memory.
+ */
+struct MemorySelection
+{
+    Offset offset;
+    Extent extent;
+};
+
 class Dataset
 {
     friend class RecordComponent;

include/openPMD/Datatype.hpp

@@ -420,6 +420,11 @@ inline size_t toBits(Datatype d)
     return toBytes(d) * CHAR_BIT;
 }
 
+/** Check if a Datatype is a signed type
+ *
+ * @param d Datatype to test
+ * @return true if signed type (integer, floating point, complex), else false
+ */
 constexpr bool isSigned(Datatype d);
 
 /** Compare if a Datatype is a vector type
@@ -602,6 +607,13 @@ inline bool isSameFloatingPoint(Datatype d)
     return isSameFloatingPoint(d, determineDatatype<T_FP>());
 }
 
+/** Compare if two Datatypes are equivalent floating point types
+ *
+ * @param d1 First Datatype to compare
+ * @param d2 Second Datatype to compare
+ * @return true if both types are floating point and have same bitness, else
+ * false
+ */
 inline bool isSameFloatingPoint(Datatype d1, Datatype d2)
 {
     // template
@@ -629,6 +641,13 @@ inline bool isSameComplexFloatingPoint(Datatype d)
     return isSameComplexFloatingPoint(d, determineDatatype<T_CFP>());
 }
 
+/** Compare if two Datatypes are equivalent complex floating point types
+ *
+ * @param d1 First Datatype to compare
+ * @param d2 Second Datatype to compare
+ * @return true if both types are complex floating point and have same bitness,
+ * else false
+ */
 inline bool isSameComplexFloatingPoint(Datatype d1, Datatype d2)
 {
     // template
@@ -656,6 +675,13 @@ inline bool isSameInteger(Datatype d)
     return isSameInteger(d, determineDatatype<T_Int>());
 }
 
+/** Compare if two Datatypes are equivalent integer types
+ *
+ * @param d1 First Datatype to compare
+ * @param d2 Second Datatype to compare
+ * @return true if both types are integers, same signedness and same bitness,
+ * else false
+ */
 inline bool isSameInteger(Datatype d1, Datatype d2)
 {
     // template
@@ -708,13 +734,24 @@ constexpr bool isChar(Datatype d)
 template <typename T_Char>
 constexpr bool isSameChar(Datatype d);
 
+/** Compare if two Datatypes are equivalent char types
+ *
+ * @param d1 First Datatype to compare
+ * @param d2 Second Datatype to compare
+ * @return true if both types are chars with same signedness and size, else
+ * false
+ */
 constexpr bool isSameChar(Datatype d1, Datatype d2);
 
 /** Comparison for two Datatypes
  *
  * Besides returning true for the same types, identical implementations on
  * some platforms, e.g. if long and long long are the same or double and
  * long double will also return true.
+ *
+ * @param d First Datatype to compare
+ * @param e Second Datatype to compare
+ * @return true if the datatypes are equivalent
  */
 constexpr bool isSame(openPMD::Datatype d, openPMD::Datatype e);
 
@@ -726,15 +763,34 @@ constexpr bool isSame(openPMD::Datatype d, openPMD::Datatype e);
  */
 Datatype basicDatatype(Datatype dt);
 
+/** Convert a scalar Datatype to its vector variant
+ *
+ * @param dt Scalar Datatype to convert
+ * @return Vector Datatype (e.g., INT becomes VEC_INT)
+ */
 Datatype toVectorType(Datatype dt);
 
+/** Convert a Datatype to its string representation
+ *
+ * @param dt Datatype to convert
+ * @return String representation of the Datatype
+ */
 std::string datatypeToString(Datatype dt);
 
+/** Convert a string to a Datatype
+ *
+ * @param s String representation of a Datatype
+ * @return The corresponding Datatype
+ */
 Datatype stringToDatatype(const std::string &s);
 
-void warnWrongDtype(std::string const &key, Datatype store, Datatype request);
-
-std::ostream &operator<<(std::ostream &, openPMD::Datatype const &);
+/** Stream operator for Datatype
+ *
+ * @param os Output stream
+ * @param dt Datatype to output
+ * @return Reference to the stream
+ */
+std::ostream &operator<<(std::ostream &os, openPMD::Datatype const &dt);
 
 template <typename T>
 constexpr auto datatypeIndex() -> size_t

include/openPMD/Datatype.tpp

@@ -223,36 +223,52 @@ namespace detail
     template <typename T>
     constexpr bool is_char_v = is_char<T>::value;
 
-    template <typename T_Char1, typename T_Char2>
-    inline bool isSameChar()
+    struct IsChar
     {
-        return
-            // both must be char types
-            is_char_v<T_Char1> && is_char_v<T_Char2> &&
-            // both must have equivalent sign
-            std::is_signed_v<T_Char1> == std::is_signed_v<T_Char2> &&
-            // both must have equivalent size
-            sizeof(T_Char1) == sizeof(T_Char2);
+        template <typename T>
+        static constexpr bool call()
+        {
+            return is_char_v<T>;
+        }
+        template <size_t N>
+        static constexpr bool call()
+        {
+            return false;
+        }
+    };
+
+    constexpr inline bool isChar(Datatype dtype)
+    {
+        return switchType<IsChar>(dtype);
     }
 
-    template <typename T1>
-    struct IsSameChar
+    struct DtypeSize
     {
-        template <typename T2>
-        static bool call()
+        template <typename T>
+        static constexpr size_t call()
         {
-            return isSameChar<T1, T2>();
+            return sizeof(T);
         }
-
-        static constexpr char const *errorMsg = "IsSameChar";
+        static constexpr char const *errorMsg = "DtypeSize";
     };
 
+    constexpr inline size_t dtypeSize(Datatype dtype)
+    {
+        return switchType<DtypeSize>(dtype);
+    }
 } // namespace detail
 
 template <typename T_Char>
 constexpr inline bool isSameChar(Datatype d)
 {
-    return switchType<detail::IsSameChar<T_Char>>(d);
+    return isSameChar(d, determineDatatype<T_Char>());
+}
+
+constexpr bool isSameChar(Datatype d1, Datatype d2)
+{
+    return detail::isChar(d1) && detail::isChar(d2) &&
+        isSigned(d1) == isSigned(d2) &&
+        detail::dtypeSize(d1) == detail::dtypeSize(d2);
 }
 
 namespace detail
@@ -285,11 +301,6 @@ constexpr inline bool isSigned(Datatype d)
     return switchType<detail::IsSigned>(d);
 }
 
-constexpr inline bool isSameChar(Datatype d, Datatype e)
-{
-    return isChar(d) && isChar(e) && isSigned(d) == isSigned(e);
-}
-
 constexpr bool isSame(openPMD::Datatype const d, openPMD::Datatype const e)
 {
     return

include/openPMD/IO/ADIOS/ADIOS2File.hpp

@@ -20,6 +20,7 @@
  */
 #pragma once
 
+#include "openPMD/Dataset.hpp"
 #include "openPMD/IO/ADIOS/ADIOS2Auxiliary.hpp"
 #include "openPMD/IO/ADIOS/ADIOS2PreloadAttributes.hpp"
 #include "openPMD/IO/ADIOS/ADIOS2PreloadVariables.hpp"
@@ -107,11 +108,14 @@ struct WriteDataset
     static void call(Params &&...);
 };
 
+/** Buffered put operation with unique pointer */
 struct BufferedUniquePtrPut
 {
     std::string name;
     Offset offset;
     Extent extent;
+    /** Optional memory selection for non-contiguous memory regions */
+    std::optional<MemorySelection> memorySelection;
     UniquePtrWithLambda<void> data;
     Datatype dtype = Datatype::UNDEFINED;