Skip to content

Fix TaggedValue Iterable masquerade and Enum socket handling - #151

Closed
elinscott wants to merge 24 commits into
scinode:mainfrom
elinscott:fix-tagged-value-unwrap
Closed

elinscott wants to merge 24 commits into
scinode:mainfrom
elinscott:fix-tagged-value-unwrap

Conversation

@elinscott

@elinscott elinscott commented Apr 29, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Value handling across process boundaries, and — since the enum half of that turned out to be half a contract — where enum socket membership is decided at all.

TaggedValue Iterable masquerade (src/node_graph/socket.py)

isinstance(TaggedValue(1.5), Iterable) was wrongly True — even though actually iterating raised TypeError. TaggedValue.__new__ now dispatches to one of two private subclasses based on whether the wrapped value is iterable:

  • _TaggedScalar shadows __iter__ = None, which makes the Iterable ABC subclass hook return False.
  • _TaggedIterable keeps ObjectProxy's forwarding behavior.

Enum serialization roundtrip

Enums (incl. str-Enum / int-Enum) are now treated as structured leaves so they survive a process boundary:

  • src/node_graph/utils/struct_utils.py: structured_type_info recognizes Enum; coerce_structured_value rebuilds the member from its bare serialized value
  • src/node_graph/socket_spec.py: the spec attaches structured_type extras to enum sockets so coerce_inputs_from_spec can reconstruct the member.

Enum socket membership, decided once at assignment

Rebuilding the member on the way out of storage left membership being decided by that coercion — which runs twice, on two different representations. At build it sees the raw object; after serialization it sees the bare value. So a member of a different enum whose value matched was rejected at build and accepted at run, and which applied to a given input depended on where its graph sat rather than on the value. Membership is now decided once, where a value is set on a socket:

  • TaskSocket._set_socket_value canonicalizes and checks — the one point an entry graph, a deferred sub-graph and a link from an untyped parent all pass through. Read-back becomes reconstruction with nothing left to reject.
  • SocketSpec.__post_init__ applies the same rule to a socket's default, so a default outside the allowed set raises where it is written rather than being accepted in silence.
  • Literal[...] derives an enum socket plus its allowed subset, or a base type plus allowed values; link compatibility compares the allowed sets, so Literal[A] flows into Literal[A, B] and not the reverse. Previously a Literal annotation produced a socket that accepted anything, and every Literal looked alike to the link checker.
  • Two numbers match only when their types agree, so True is not 1; everything else matches on equality, so a value read back from storage still names its member.
  • A refusal names the socket and lists the allowed values.

Deserialize hook in materialize_graph (src/node_graph/utils/graph.py)

New _deserialize_inputs walks the input namespace and applies adapter.deserialize to each leaf before the graph body runs. Symmetric counterpart to the existing serialize-on-write path: a @task.graph body whose signature declares a primitive should receive a primitive even if the engine wrapped it for provenance.

Testing

  • TaggedValue regression coverage: a scalar wrapper is not Iterable and iter() raises while arithmetic/equality still proxy; list/tuple/dict/str/set wrappers stay iterable; the socket= tag survives both paths.
  • The enum-coercion engine test now feeds the flattened form a str/int-Enum takes across a serialization boundary, so the coercion has a real member to rebuild rather than passing an already-typed value straight through.
  • The membership rule is covered across all three graph shapes, with the accepted and refused cases beside each other: a foreign member whose value matches is accepted everywhere, while a value naming no member, a foreign member with no matching value, and a member whose name alone matches are refused at build. The name-only case is the control — a rule keyed on names rather than values would pass the others and fail that one.

@codecov

codecov Bot commented Aug 7, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.20557% with 39 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.09%. Comparing base (8d85e61) to head (9605a89).

Files with missing lines Patch % Lines
src/node_graph/graph.py 57.14% 12 Missing ⚠️
tests/test_enum_literal_sockets.py 95.29% 12 Missing ⚠️
src/node_graph/link.py 92.18% 5 Missing ⚠️
src/node_graph/utils/struct_utils.py 93.05% 5 Missing ⚠️
src/node_graph/socket.py 92.30% 2 Missing ⚠️
src/node_graph/utils/graph.py 90.00% 2 Missing ⚠️
src/node_graph/socket_spec.py 98.11% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #151      +/-   ##
==========================================
+ Coverage   89.68%   90.09%   +0.41%     
==========================================
  Files          81       82       +1     
  Lines        8984     9525     +541     
==========================================
+ Hits         8057     8582     +525     
- Misses        927      943      +16     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@elinscott

Copy link
Copy Markdown
Collaborator Author

Update

The structured_type overlay this PR added for enums incidentally forced the socket to be required. SocketMeta.required is typed Optional[bool] but defaults to True, and merge_meta prefers any non-None overlay value — so a bare SocketMeta(extras={...}) republishes required=True over the value already computed from the parameter's default.

Now fixed by passing required=None, so the overlay says nothing about requiredness.

Worth noting the same shape exists on main independently of this branch: any Annotated[Optional[int], SocketMeta(help="...")] comes out required, because that overlay defaults the same way.

elinscott added a commit to elinscott/node-graph that referenced this pull request Aug 14, 2026
elinscott added a commit to elinscott/node-graph that referenced this pull request Aug 14, 2026
Resolution notes, all of it integration between two open pull requests:

- socket.py, tests/test_socket.py: scinode#151 and scinode#171 add disjoint definitions in
  the same place; both are kept.
- utils/graph.py: scinode#151's `_deserialize_inputs` rebuilds the collected mapping.
  `dict(values)` downgrades scinode#171's `TaggedNamespace` to a plain dict, so a
  graph body receives a mapping without its socket handle and `reference()`
  raises `AttributeError`. Rebuild via `values.copy()`, which
  `TaggedNamespace` overrides to keep the handle.
elinscott and others added 20 commits August 17, 2026 08:19
…lize_graph

Two related changes that together let downstream consumers of
``@task.graph`` round-trip primitive socket values cleanly:

1. ``TaggedValue.__new__`` now dispatches on the wrapped value's
   actual iterability, returning a ``_TaggedScalar`` (with
   ``__iter__ = None``) for non-iterables and a ``_TaggedIterable``
   otherwise. Previously the bare ``wrapt.ObjectProxy`` always
   delegated ``__iter__`` to whatever was wrapped, which made
   ``isinstance(TaggedValue(1.5), collections.abc.Iterable)`` return
   ``True`` while ``iter(...)`` raised ``TypeError`` — a contradiction
   that bit AiiDA's ``clean_value`` (and any other ABC-based
   isinstance check) hard. Setting ``__iter__ = None`` is recognised
   by ``Iterable.__subclasshook__`` (via ``_check_methods``) as
   "method intentionally unset", so the subclass hook returns
   ``NotImplemented`` and ``isinstance`` correctly answers ``False``
   for wrapped scalars.

2. ``materialize_graph`` now calls ``adapter.deserialize(value, socket)``
   on each input just before invoking the user's ``@task.graph`` body.
   ``SerializationAdapter.deserialize`` was already in the base class
   API but never called; this closes the loop with the existing
   serialize-on-write pass. New helper ``_deserialize_inputs`` walks
   the input namespace + values dict in parallel and applies the
   adapter to each leaf. No-op for graphs without a serialization
   adapter (the base class deserialize is identity).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hoists the dynamic-namespace path-walking logic out of links_from_dict
into a reusable staticmethod so aiida-workgraph's Map-zone clone path
(_patch_cloned_tasks) can resolve dotted targets like ``pseudos.O``
into a dynamic namespace by materialising children on demand.

Fixes the AttributeError "'O' is not in this namespace" raised when a
Map zone clones a task whose dynamic-namespace input is populated by
links (not raw values) — the round-trip via ``to_dict`` was preserving
the namespace itself but losing its link-installed children.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
``_deserialize_inputs`` previously only recursed into dict-typed
namespace values. ``coerce_inputs_from_spec`` runs first in
``materialize_graph``, so by the time deserialize runs a dataclass /
Pydantic namespace is already a structured instance — not a dict —
and the adapter never gets to see it.

For adapters that auto-promote primitive fields during serialisation
(e.g. ``aiida-workgraph`` wrapping ``int`` → ``orm.Int``), the
structured instance crosses the boundary with wrapped fields and
downstream user code breaks (``range(self.ntyp)`` raises ``TypeError``).

Pass structured-instance values to ``adapter.deserialize`` alongside
the dict path; the adapter walks fields and unwraps as needed.
coerce_structured_value ran import_structured_type unconditionally at the
top, so an already-materialised dataclass/pydantic instance whose recorded
type path is not importable (e.g. a locally-defined type whose qualname
contains "<locals>") raised ModuleNotFoundError instead of passing through
untouched. Move the import into the enum branch and the dict-rebuild path,
which are the only branches that need the class.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The previous test passed Color.GREEN directly, so coerce_inputs_from_spec
was a no-op (the value was already a Color) and the body's isinstance
assert was trivially satisfied — the test passed with or without the enum
coercion. Feed the flattened form a str/int-Enum takes across a
serialization boundary (a bare "green" / 9) so the coercion must rebuild
the member before the body runs; add an int-Enum case. Both tests fail
when the enum branch of coerce_structured_value is removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
wrapt.ObjectProxy exposes __iter__ at the C level, so a scalar TaggedValue
would report isinstance(_, Iterable) is True and defer iteration to a
missing wrapped __iter__. Cover: scalar TaggedValue is not Iterable and
iter() raises TypeError while arithmetic/equality still proxy; list/tuple/
dict/str/set TaggedValue is Iterable and iterates; socket= kwarg preserved
on both scalar and iterable. The scalar cases fail on pre-fix socket.py.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Docstring claimed the helper was used by the engine's Map-zone clone path
(_patch_cloned_tasks), which does not exist in this repo; the only caller
is links_from_dict. Describe the actual in-repo caller and note the
downstream reuse as external.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
uv.lock was added in the helper-extraction commit with no pyproject.toml
change; main does not track it and none of this PR's code needs a new
dependency floor. Remove it to return to the base branch's state.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An enum-typed parameter came out as a required socket even when it had a
default, so an optional enum input could not be expressed in a signature
at all.

The structured_type overlay was a bare `SocketMeta`, whose `required`
defaults to `True`, and `merge_meta` prefers any non-`None` overlay
value — so the overlay republished `required=True` over the value
computed from the parameter's default. Passing `required=None` leaves
that decision to the signature.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pycodestyle's E713 matches the logical line rather than the syntax tree,
so the words "not found in" inside the message tripped it and pre-commit
failed on a file with no membership test in it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI's black hook reformatted this line, so pre-commit failed on a file the
branch had not otherwise changed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An enum-typed socket rejected a member of another enum carrying the same
value while accepting that same value once storage had flattened it, and
the difference followed the graph's shape rather than the value.

- Canonicalize and validate on the socket's value setter, which every
  graph shape passes through
- Accept a member of the declared enum, a member of any other enum with
  a matching value, or a bare value; reject anything else
- Reconstruct on read-back through the same helper instead of deciding
  a second time
- Name the socket and its allowed values in the error, without the
  value wrapper's repr
- Derive Literal into an enum or base-type socket plus the permitted
  subset, and compare subsets when checking a link

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Storing the member itself reached the engine's serializers as a raw Enum
and dropped the graph_inputs links a rebuilt value used to carry.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
elinscott and others added 4 commits August 17, 2026 11:13
A default reached a socket without the check an assigned value passes,
so a defaulted enum socket held the member where an assigned one held
the value, and a Literal default outside its own allowed set was taken.

- Canonicalize and check a default where the spec is built, so a
  defaulted socket reads the same as an assigned one on both sides of a
  round trip.
- Decide enum membership by the rule a Literal uses: an IntEnum whose
  members are 1 and 2 no longer takes True or 1.0.
- Name the socket in the message a run-time coercion raises.
- Keep the enum behind a Literal whose arguments all name members of one
  enum, whether written as members or as their values.
- Render an enum argument's py_type with its module path.
- Say in the link check what it reads and what it leaves to the run,
  and cover the two-hop and untyped-source shapes it does not reject at
  build.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An enum socket refused the value it was read back as: a storage node
that equals 'none' without being a str named no member, so a workgraph
loaded from the database no longer built.

- Compare two numbers by type, which is where True and 1 must part, and
  everything else by equality.
- Cover the read-back shape with a value that equals a member's value
  without sharing its type.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Enum membership was decided by a coercion that ran twice on two
representations, so a member of another enum with a matching value was
rejected at build and accepted at run; it is now decided once, where a
value is set on a socket.

- Canonicalize and check in `TaskSocket._set_socket_value`, the one
  point an entry graph, a deferred sub-graph and a link from an untyped
  parent all pass through.
- Check a socket's default by the same rule, in `SocketSpec.__post_init__`,
  so a default outside the allowed set raises where it is written rather
  than being accepted in silence.
- Derive `Literal` into an enum socket plus its allowed subset, or a base
  type plus allowed values; link compatibility compares the allowed sets,
  so `Literal[A]` flows into `Literal[A, B]` and not the reverse.
- Match two numbers only when their types agree, so `True` is not `1`,
  and match everything else on equality, so a value read back from
  storage still names its member.
- Name the socket and list the allowed values when a value is refused.
@elinscott elinscott changed the title Fix TaggedValue Iterable masquerade and Enum serialization roundtrip Fix TaggedValue Iterable masquerade and Enum socket handling Aug 17, 2026
elinscott added a commit to elinscott/node-graph that referenced this pull request Aug 17, 2026
elinscott added a commit to elinscott/node-graph that referenced this pull request Aug 17, 2026
Resolution notes, all of it integration between two open pull requests:

- graph.py: scinode#151 and scinode#166 reword the same docstring paragraph of
  `_resolve_or_create_input_socket`; scinode#151's wording is kept, and scinode#166's
  already-materialised-leaf early return is taken.
elinscott added a commit to elinscott/node-graph that referenced this pull request Aug 17, 2026
Resolution notes, all of it integration between two open pull requests:

- socket.py, tests/test_socket.py: scinode#151 and scinode#171 add disjoint definitions in
  the same place; both are kept.
- utils/graph.py: scinode#151's `_deserialize_inputs` rebuilds the collected mapping.
  `dict(values)` downgrades scinode#171's `TaggedNamespace` to a plain dict, so a
  graph body receives a mapping without its socket handle and `reference()`
  raises `AttributeError`. Rebuild via `values.copy()`, which
  `TaggedNamespace` overrides to keep the handle.
@elinscott

Copy link
Copy Markdown
Collaborator Author

Superseded by #177 (masquerade) and #178 (enum/Literal membership). Closing

@elinscott elinscott closed this Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant