Skip to content

esp32: Add ESP-NOW - #2472

Open
gabrielmancini wants to merge 10 commits into
atomvm:release-0.7from
iobee-io:esp32-espnow
Open

gabrielmancini wants to merge 10 commits into
atomvm:release-0.7from
iobee-io:esp32-espnow

Conversation

@gabrielmancini

@gabrielmancini gabrielmancini commented Nov 9, 2026 •

Copy link
Copy Markdown

Adds an espnow port driver on ESP32, behind CONFIG_AVM_ENABLE_ESPNOW (off by default), and the espnow module in avm_esp32.

  • Frames go to a peer or to broadcast: up to 1470 bytes with ESP-NOW v2 (ESP-IDF 5.4 or newer), 250 before.
  • They arrive as {espnow, rx, Mac, Data, #{rssi := Rssi}}, each one's binary copied once, in the Wi-Fi task.
  • send/3 is synchronous. It returns ok once the frame is out (for a unicast frame, the peer acknowledged it) or {error, fail}. The driver gives the radio one frame at a time, as the ESP-NOW guide recommends, so concurrent senders wait their turn.
  • Calls: open/1 (receiver, channel 0..14, pmk), close/1, add_peer/2,3 (optional channel and lmk), del_peer/2, send/3 (a MAC or broadcast), set_channel/2, get_channel/1, stats/1.
  • Events go through the VM listener and a 16-slot queue, with one slot kept for the send result.
  • Received bytes waiting for the scheduler are bounded by CONFIG_AVM_ESPNOW_RX_BUDGET (8192 by default). A frame past either limit is dropped and counted in stats/1.
  • Only one port can be open, linked to the caller before ESP-NOW starts. An abnormal caller exit terminates it, and the next open/1 reclaims ESP-NOW. A normal exit does not close it, so call close/1.

The Wi-Fi radio gets one owner, the network driver. ESP-NOW holds it through network_wifi_hold/1 and network_wifi_release/0 (include/network_wifi.h), and does not call esp_wifi_init or esp_wifi_start itself.

  • One lock, the network driver's radio lock (network_wifi_lock/0), serializes the network's start and stop with everything ESP-NOW does in the scheduler: open, close, peers and keys, sends.
  • With no network started, the radio comes up as a bare STA that does not connect, on the ESP-NOW channel.
  • network:start/1 and network:stop/0 can be called while ESP-NOW is open. Each radio restart installs the PMK and the encrypted peers' LMKs again. network:stop/0 leaves the radio on and back on the ESP-NOW channel.
  • While the STA is connected, frames go on the access point's channel. An AP-only network is refused while ESP-NOW is open ({error, ap_only}), and the other way round.

The API follows harmon25/atomvm_espnow (Apache-2.0) by @harmon25: open, send, add_peer, del_peer and the {espnow, rx, Mac, Data} message. Its driver is what our first range tests ran on. This one is new code, because ESP-NOW here has to share the Wi-Fi radio with the network driver and survive network:start/1 and network:stop/0. Thanks to @harmon25 for it.

The ESP32 v5.5.5 PPP build in the CI matrix also enables ESP-NOW, so the two network options are built together, and an ESP32-C3/IDF 5.2.8 build covers the older send-callback signature. The network programming guide gets a section.

Tested on hardware: ESP32 (T-PCIE, WROVER with the VM not using PSRAM) and ESP32 without PSRAM (T-Call). The numbers below were measured side by side (−26 to −30 dBm, 250-byte frames) with an earlier asynchronous version of send; they are kept as a range reference:

mode throughput frames/s lost
unicast, 1 in flight ~23.6 KB/s ~95 0
unicast, 4 in flight ~31.7 KB/s ~127 0
broadcast ~24.9 KB/s ~100 0

The tests also covered:

  • network:start/1 and network:stop/0 cycles with ESP-NOW open;
  • a device using ESP-NOW next to a full application (TLS MQTT, Modbus), which cost about 4 KB of free heap;
  • an earlier version of this driver in a small fleet since 2026-11-04.

The review-driven changes since then (radio lock, deferred queue teardown, private send completion) still need their hardware regression run. No QEMU test: QEMU has no Wi-Fi radio.

These changes are made under both the "Apache 2.0" and the "GNU Lesser General Public License 2.1 or later" license terms (dual license).

SPDX-License-Identifier: Apache-2.0 OR LGPL-2.1-or-later

@github-advanced-security

Copy link
Copy Markdown

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

  • The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
  • Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
  • You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.

@gabrielmancini

Copy link
Copy Markdown
Author

The failing esp-idf (esp32, v5.5.5, true) job looks like a QEMU flake, not this change: in the same job the "with memory checks" build ran all 26 tests and passed, and the second run hung in test_uart_mode with QEMU looping on Unhandled interrupt 5 on cpu 1 until the 180 s timeout. The same test passes on #2471 with the same configuration. Could a maintainer re-run the failed job? I don't have the rights to. Thanks!

@gabrielmancini

Copy link
Copy Markdown
Author

The second red job, build-and-test (riscv64-linux-gnu-gcc, 28, ...), is not this change either. test_gen_tcp:test_connect_parameters connects to www.atomvm.org from the runner and timed out ({connect,110}, ETIMEDOUT). This PR only touches the ESP32 platform, libs/avm_esp32, docs and CI. A re-run of both failed jobs should be enough.

@petermm

petermm commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

LLM review, seems like there is v2 of esp-now which allows more data:

Review: ESP-NOW support

Scope and recommendation

Request changes. Reviewed the combined diff HEAD~2..HEAD, including:

  • dc78a8eba2afc34f51732aa8d8e2cbe9cd826554 — esp32: Add ESP-NOW
  • 964b72a4cf2bf6941cb98d2cbd8d11cdc2375482 — espnow: Format with erlfmt

The second commit only reformats the try expression in espnow.erl; no behavioral issue was found in that commit. The feature commit introduces the findings below.

The requested oracle review was performed, and its principal findings were cross-checked against the runtime, scheduler, and local ESP-IDF 5.5.5 sources. This is a source review, not a hardware qualification: no firmware build, allocation-fault test, or two-device radio test was run. Proposed diffs below are review suggestions, not applied or tested fixes.

Severity: P1 = high-impact correctness issue to fix before merging; P2 = correctness issue under a narrower configuration or failure condition.

1. P1 — Abnormal port termination leaves live references to freed state

Location: src/platforms/esp32/components/avm_builtins/espnow_driver.c:234–244,325–327,462–467.

The port stores EspNowData directly in ctx->platform_data, but calls teardown() only for the explicit close command. When a native process is killed, scheduler_run() destroys its context without calling its native handler (src/libAtomVM/scheduler.c:288–299). context_destroy() frees platform_data without unregistering the driver's listener or Wi-Fi callbacks (src/libAtomVM/context.c:301–307).

Trigger: Open ESP-NOW, terminate the port with exit(Port, kill), then deliver an RX/TX callback or traverse the system listeners. The singleton and listener still point into freed memory. The queue, ESP-NOW initialization, and radio hold also remain live; reopening is blocked by the stale singleton.

Required fix: Tie teardown to the port's lifetime, including kill/linked-owner termination, through a runtime-supported resource monitor or a native-context cleanup facility. Make cleanup run exactly once and coordinate it with in-flight callbacks. A global driver-destroy callback alone would not cover individual port death. This is not safely fixed by a one-line change in the close branch.

Missing regression: Terminate an open port without espnow:close/1, while a second device sends frames. Wait for termination, reopen, and verify both unicast and broadcast work. Repeat explicit close under the same load as a control. Run with memory-error detection where practical.

2. P1 — Radio ownership and singleton transitions race on SMP

Location: network_driver.c:1874–1945,1979–1982,2064–2065,2235–2253; espnow_driver.c:257–301, under src/platforms/esp32/components/avm_builtins/.

The new shared flags, holder callback, and espnow_data singleton have no lock. The comment that everything runs in the scheduler does not serialize different native contexts: the scheduler releases schedulers_mutex before executing native handlers. ESP32's CMake configuration enables SMP by default on multicore targets.

Two concrete failing interleavings:

  1. stop_network() observes a holder and begins restarting the bare STA. Concurrent ESP-NOW close calls network_wifi_release(), clears the holder callback, and can stop the radio. stop_network() then calls wifi_holder_restarted() without rechecking it: a NULL function call is possible.
  2. start_wifi() starts Wi-Fi but has not yet set wifi_used_by_network. Concurrent close sees no network owner and deinitializes Wi-Fi. Startup subsequently marks the now-off radio as started and returns success.

Concurrent opens can also pass both singleton checks before either publishes ownership.

Required fix: Serialize complete radio ownership transitions and singleton acquisition/release, including their SDK operations. Atomics on individual flags or a local copy of the callback do not preserve the ownership invariant. Choose one lock/serialized owner and document its order relative to listener and callback synchronization.

Missing regression: On a dual-core target, repeatedly overlap open/open, close/network-start, and close/network-stop. Assert only one open succeeds and that the remaining radio user can still transmit after each transition. A single-core-only test cannot cover this finding.

3. P1 — AP-only networking disables ESP-NOW's selected interface

Location: espnow_driver.c:247–254; network_driver.c:1926–1928,2007–2016.

Every peer, including the automatically added broadcast peer, uses WIFI_IF_STA. AP-only network startup selects WIFI_MODE_AP, disabling STA. Conversely, opening ESP-NOW after AP-only startup just acquires the already-started radio without enabling STA. ESP-IDF requires the peer's selected interface to be enabled; the send API documents ESP_ERR_ESPNOW_IF for an interface mismatch.

Trigger: Either startup order of ESP-NOW plus an AP-only network. ESP-NOW cannot reliably send through its configured interface, contrary to the documented coexistence contract.

Required fix: Derive the Wi-Fi mode from both owners. AP-only network plus ESP-NOW needs APSTA, whether the holder or the network starts first. Keep the extra STA unassociated. Simply selecting APSTA is insufficient: event_handler() currently invokes esp_wifi_connect() on WIFI_EVENT_STA_START whenever !data->managed, even for an AP-only network.

Missing regression: For both startup orders, start an AP-only network and exchange actual broadcast and acknowledged unicast frames with a second device. Verify the AP remains usable and the extra STA does not attempt an unintended connection. Repeat with STA and APSTA as controls.

4. P2 — Failed network startup does not restore a held radio

Location: network_driver.c:1977–1988,2000–2065,2235–2253.

Starting a network stops the existing bare STA, but wifi_used_by_network is set only after startup succeeds. If registration, configuration, or esp_wifi_start() then fails, cleanup sees network_had_wifi == false and does not restart the holder's STA. ESP-NOW remains open while the radio it holds is stopped.

Trigger: Open ESP-NOW successfully, then inject a failure after network startup stops the bare STA. Even subsequent stop_network() does not restore it under the current condition.

Suggested narrow recovery change, after fixing the ownership synchronization:

--- a/src/platforms/esp32/components/avm_builtins/network_driver.c
+++ b/src/platforms/esp32/components/avm_builtins/network_driver.c
@@
     // A holder (ESP-NOW) keeps the radio on, as a bare STA again.
     if (wifi_holder_restarted != NULL) {
-        if (network_had_wifi && wifi_start_bare_sta() == ESP_OK) {
+        if (!wifi_started && wifi_start_bare_sta() == ESP_OK) {
             wifi_holder_restarted();
         }
     } else {

This repairs the skipped restart when cleanup runs; it does not replace the need for a defined startup-error rollback path or error handling if the recovery restart itself fails.

Missing regression: Fault-inject failure after the held radio is stopped but before wifi_used_by_network is set. After cleanup, verify the original ESP-NOW channel and an acknowledged frame exchange, not merely that the port still answers calls.

5. P2 — RX silently truncates valid ESP-NOW v2 datagrams

Location: espnow_driver.c:128–135.

ESP-IDF 5.5.5 defines a 1470-byte v2 maximum alongside the 250-byte v1 maximum. The receive callback clamps len to 250 and delivers the prefix as an ordinary complete frame, with neither an error nor a dropped-event increment. A 251-byte datagram and a real 250-byte datagram with that prefix become indistinguishable.

The public API intentionally supports only 250 bytes; reject unsupported sizes rather than silently changing datagram contents.

Suggested small fix:

--- a/src/platforms/esp32/components/avm_builtins/espnow_driver.c
+++ b/src/platforms/esp32/components/avm_builtins/espnow_driver.c
@@
 static void espnow_recv_cb(const esp_now_recv_info_t *info, const uint8_t *payload, int len)
 {
+    if (len > ESP_NOW_MAX_DATA_LEN) {
+        // Use the synchronized drop-counter operation introduced for stats.
+        struct EspNowData *data = espnow_data;
+        if (data != NULL) {
+            data->dropped++;
+        }
+        return;
+    }
     struct EspNowEvent event = { .kind = EspNowReceived };
     memcpy(event.mac, info->src_addr, MAC_LEN);
     event.rssi = info->rx_ctrl->rssi;
-    event.len = len < ESP_NOW_MAX_DATA_LEN ? len : ESP_NOW_MAX_DATA_LEN;
+    event.len = len;

The counter increment above illustrates the rejection path; it must use the synchronized counter and callback-lifetime scheme described below, rather than introduce another unsynchronized increment.

Missing regression: Send distinct 249-, 250-, 251-, and 1470-byte frames from a v2-capable sender. Compare every accepted byte; require oversized frames to be absent from RX messages and counted as dropped. A send-side rejection test alone does not exercise this callback.

6. P2 — Allocation failures can crash open/RX or abandon calls forever

Location: espnow_driver.c:195,325–327,436–440.

There are three separate exhaustion paths:

  • context_new() can return NULL, but open immediately dereferences it after registering listeners and initializing ESP-NOW.
  • A successful event-heap allocation does not reserve the separately allocated refcounted binary buffer. term_from_literal_binary() calls term_binary_data() without checking whether binary allocation failed (src/libAtomVM/term.h:2114–2118). Larger RX payloads can therefore crash instead of being dropped.
  • Reply-space allocation failure removes the request without replying or executing it. The Erlang wrappers use port:call/2, whose default timeout is infinite. A caller can hang permanently; a close request is discarded too.

Suggested small fix for context allocation:

--- a/src/platforms/esp32/components/avm_builtins/espnow_driver.c
+++ b/src/platforms/esp32/components/avm_builtins/espnow_driver.c
@@
     Context *ctx = context_new(global);
+    if (IS_NULL_PTR(ctx)) {
+        teardown(data);
+        return NULL;
+    }
     ctx->native_handler = espnow_consume_mailbox;
     ctx->platform_data = data;

This addresses only the NULL dereference and depends on making teardown safe. For RX, use checked binary construction before accessing the buffer, destroy the temporary heap on failure, and count the drop. For calls, use the runtime's established OOM/error or termination path instead of silently discarding a synchronous request.

Missing regression: Inject context-allocation, refcounted-binary-allocation, and reply-heap failures separately. Assert failed open leaves no singleton/radio hold; failed RX is dropped without a crash; synchronous calls resolve or fail rather than hang.

Additional issues and small suggestions

Pending queue teardown violates the FreeRTOS queue-set contract

espnow_driver.c:240–242 ignores xQueueRemoveFromSet() failure and deletes the queue anyway. ESP-IDF's FreeRTOS-Kernel/queue.c:3287–3313 explicitly refuses removal of a nonempty queue because pending queue-set notifications remain. Quiesce callbacks and coordinate draining both the member and its set notifications before deletion. Merely resetting the member queue is not sufficient.

This is a confirmed API-precondition violation, but not independently proven to cause an immediate crash: AtomVM compares unmatched sender handles rather than dereferencing them. Queue-handle reuse and close/reopen under traffic need targeted testing. Treat this as part of the teardown fix, not as evidence that SDK callback unregistration necessarily waits for every in-flight callback.

Dropped statistics are neither atomic nor valid across the full counter range

dropped is a volatile uint32_t, updated from both the Wi-Fi task and event-consumer task (espnow_driver.c:123,181); volatile does not make read-modify-write increments atomic. Lost increments are possible. do_stats() also encodes it with term_from_int(), which only encodes small integers. On 32-bit ESP32, 134,217,728 is outside the positive small-integer range and is reported incorrectly.

Use atomic updates/a synchronized snapshot and the existing boxed-integer construction API, reserving the additional heap words. Test concurrent increments and the values immediately below, at, and above the small-integer boundary. Do not create a new integer-conversion wrapper if an existing helper suffices.

Describe the actual queue limit, not receiver-side backpressure

The eight-entry queue bounds Wi-Fi-task-to-VM backlog. Once the listener runs, it forwards events into the receiver's mailbox without checking whether that receiver consumes them. The receiver mailbox is not bounded by eight events. Documentation currently attributes drops to a slow receiver and can imply a stronger memory bound than implemented.

--- a/libs/avm_esp32/src/espnow.erl
+++ b/libs/avm_esp32/src/espnow.erl
@@
-%% @doc `dropped': frames and send results lost because the receiver did not
-%% keep up (the driver queues 8).
+%% @doc `dropped': frames and send results lost by the driver, including
+%% callback-queue overflow. The driver queues 8 events before VM delivery;
+%% this does not bound the receiver's mailbox.

Update the equivalent paragraph in doc/src/network-programming-guide.md too. Receiver-side backpressure would be an API/design change, not a necessary implementation addition for this review.

Compile both callback signatures in CI

The only ESP-NOW-enabled CI configuration uses ESP32/IDF 5.5.5. The pre-5.5 send-callback branch is never compiled. Add a small enabled configuration on a pre-5.5 SDK and a single-core target, preserving existing disabled builds:

--- a/.github/workflows/esp32-build.yaml
+++ b/.github/workflows/esp32-build.yaml
@@
         include:
+        - esp-idf-target: "esp32c3"
+          idf-version: 'v5.2.8'
+          espnow: 'ON'
+          jit: false
         - esp-idf-target: "esp32p4"

Check the GitHub matrix expansion when applying this: the added entry augments the matching base combination; it need not create a duplicate build.

Suggested missing public-API smoke test

No tests were added by these commits. The following is a small proposed ESP32-only test module, to register in the appropriate opt-in device harness once ESP-NOW is enabled. It checks outbound size validation and explicit close/reopen without relying on receipt of radio acknowledgments. It does not replace the lifecycle, SMP, AP coexistence, RX, and fault-injection scenarios above.

--- /dev/null
+++ b/tests/libs/eavmlib/test_espnow.erl
@@
+-module(test_espnow).
+-export([test/0]).
+
+test() ->
+    {ok, Port} = espnow:open([{channel, 1}]),
+    try
+        {error, badarg} = espnow:send(Port, broadcast, <<>>),
+        {error, badarg} = espnow:send(Port, broadcast, binary:copy(<<16#A5>>, 251)),
+        {error, badarg} = espnow:set_channel(Port, 0),
+        ok = espnow:send(Port, broadcast, binary:copy(<<16#5A>>, 250)),
+        {ok, 1} = espnow:get_channel(Port)
+    after
+        ok = espnow:close(Port)
+    end,
+    {ok, Reopened} = espnow:open([{channel, 6}]),
+    try
+        {ok, 6} = espnow:get_channel(Reopened)
+    after
+        ok = espnow:close(Reopened)
+    end,
+    ok.

This test sketch needs the standard source license header and device-harness integration before inclusion; it has not been compiled or executed. The asymmetric channel choices ensure reopening cannot pass by accidentally reporting the previous port's channel.

Evidence and merge gate

  • Inspected both commits, surrounding runtime/native-process destruction, scheduler locking, listener dispatch, and ESP-IDF 5.5.5 ESP-NOW and FreeRTOS contracts.
  • git diff HEAD~2 HEAD --check passed.
  • No source changes were applied. No firmware or hardware test results are claimed.
  • Fix the three P1 findings before merge, including lifetime-safe callback/queue teardown. Address failed-start rollback, datagram integrity, and allocation failures, and provide device-level regression evidence for the documented coexistence promise.
  • Still unverified: callback quiescence guarantees of SDK unregister/deinit; encrypted-peer continuity across Wi-Fi stop/start; and whether SDK channel-setting behavior enforces every connected-STA restriction promised by the Erlang docs. These are validation requirements, not additional confirmed defects.

@bettio bettio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still have to do an API review. Some initial findings.

@bettio bettio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few additional comments:

  • Port death: see the inline comment at espnow_driver.c:327 for a fix that needs no runtime change.
  • AP-only network: every peer is on WIFI_IF_STA while an ap-only config runs WIFI_MODE_AP, so every send fails with ESP_ERR_ESPNOW_IF in both start orders. Either run APSTA while a holder exists and keep the STA unassociated (event_handler must then not call esp_wifi_connect on STA_START for an AP-only config), or refuse the combination with a clear error and document it.
  • RX frames above the limit (espnow_driver.c:133): drop and count them in dropped instead of truncating.
  • Check context_new for NULL (espnow_driver.c:325) and teardown on failure.
  • Drain the queue after unregistering the callbacks and before xQueueRemoveFromSet (espnow_driver.c:241), and check its result: it fails on a non-empty queue.
  • Drops are not caused by a slow receiver: they happen when the 8-slot queue between the Wi-Fi task and the scheduler fills, and the receiver's mailbox is unbounded. Reword espnow.erl:121 and the guide at line 560.
  • CI: only the 5.5.5 build compiles the driver, so the pre-5.5 espnow_send_cb is never built. Add espnow: 'ON' to a pre-5.5 matrix entry too, for example esp32c3 v5.2.8.

Two questions on the hardware tests, since the code relies on them: did espnow:set_channel/2 fail while the STA was connected (inline comment at espnow_driver.c:387), and did encrypted peers keep working across network:start/1 and network:stop/0?

Comment thread libs/avm_esp32/src/espnow.erl Resolved
Comment thread libs/avm_esp32/src/espnow.erl Resolved
Comment thread libs/avm_esp32/src/espnow.erl Resolved
Comment thread libs/avm_esp32/src/espnow.erl Resolved
@bettio

bettio commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

This can be a follow-up commit / change, so an additional commit on this PR history.

This supersedes my inline comment about the v2 frame size. Two parts: a different way to stage frames between the Wi-Fi task and the VM, which removes the fixed queue and the need for a slot-size option, and a teardown handshake that the driver needs regardless of which staging it uses.

Why

On a 32-bit target every binary of 32 bytes or more is a refc binary, so espnow_event_callback already mallocs and copies almost every frame a second time through term_from_literal_binary. The fixed slot avoids no allocation; it only moves it out of the Wi-Fi task, at the price of permanent RAM:

staging idle RAM per 1470-byte frame pending copies per frame
8 fixed slots, as in the PR 2,164 at v1, 11,844 at v2 0 staged, then 1,490 in the VM 2
refc binary filled in the Wi-Fi task about 340 1,490 1

refc_binary_create_refc is a plain malloc that touches no VM state, so the callback can allocate the binary the receiver will eventually see, copy the payload once, and queue a 16-byte item. The listener wraps it into a term, sends the message, and destroys its temporary heap; the receiver's mailbox copy then owns the binary, and it is freed when the receiver's GC drops the message. An atomic bytes-in-flight budget bounds what the Wi-Fi task may stage, so the "no more memory in a burst" property stays, as a configurable number rather than a slot count. The maximum frame length follows the ESP-IDF constant and nothing else needs an option. ESP-IDF's own driver allocates its dynamic RX buffers in that task, so the malloc is in good company.

One helper in libAtomVM

(Let's implement this as a separated commit, to make review easier).

The non-const branch of term_alloc_refc_binary without the allocation, next to it in term.c:

term term_from_refc_binary(struct RefcBinary *refc, Heap *heap, GlobalContext *glb)
{
    term *boxed_value = memory_heap_alloc(heap, TERM_BOXED_REFC_BINARY_SIZE);
    boxed_value[0] = ((TERM_BOXED_REFC_BINARY_SIZE - 1) << 6) | TERM_BOXED_REFC_BINARY;
    boxed_value[1] = (term) refc->size;
    boxed_value[2] = RefcNoFlags;
    boxed_value[3] = (term) refc;
    term ret = ((term) boxed_value) | TERM_PRIMARY_BOXED;
    refc->ref_count = 1; // this heap's MSO list owns it from here
    heap->root->mso_list = term_list_init_prepend(boxed_value + REFC_BINARY_CONS_OFFSET, ret, heap->root->mso_list);
    synclist_append(&glb->refc_binaries, &refc->head);
    return ret;
}

Driver state

#include <stdatomic.h>

#ifdef ESP_NOW_MAX_DATA_LEN_V2
#define ESPNOW_MAX_FRAME ESP_NOW_MAX_DATA_LEN_V2
#else
#define ESPNOW_MAX_FRAME ESP_NOW_MAX_DATA_LEN
#endif
#define ESPNOW_QUEUE_LEN 16

struct EspNowEvent
{
    uint8_t kind;
    uint8_t mac[MAC_LEN];
    int8_t rssi;
    uint8_t ok;
    struct RefcBinary *frame; // EspNowReceived only; never listed while staged
};

struct EspNowData
{
    QueueHandle_t queue;
    EventListener listener;
    GlobalContext *global;
    int32_t receiver;
    _Atomic uint32_t rx_bytes; // staged bytes, callback to delivery
    _Atomic uint32_t dropped;
};

static _Atomic(struct EspNowData *) espnow_data; // stored before the callbacks are registered
static _Atomic uint32_t espnow_in_callback;
static _Atomic uint8_t espnow_channel;           // read by espnow_wifi_restarted, which must not touch data

Callbacks, in the Wi-Fi task

They take no VM lock: malloc, memcpy, atomics and a zero-timeout queue send.

// NULL means the driver is gone: do nothing.
static struct EspNowData *callback_enter(void)
{
    atomic_fetch_add(&espnow_in_callback, 1); // seq_cst, ordered before the load
    struct EspNowData *data = atomic_load(&espnow_data);
    if (data == NULL) {
        atomic_fetch_sub(&espnow_in_callback, 1);
    }
    return data;
}

static void callback_leave(void)
{
    atomic_fetch_sub(&espnow_in_callback, 1);
}

static void espnow_recv_cb(const esp_now_recv_info_t *info, const uint8_t *payload, int len)
{
    struct EspNowData *data = callback_enter();
    if (data == NULL) {
        return;
    }
    if (len <= 0 || len > ESPNOW_MAX_FRAME) {
        atomic_fetch_add(&data->dropped, 1);
        goto leave;
    }
    if (atomic_fetch_add(&data->rx_bytes, len) + len > CONFIG_AVM_ESPNOW_RX_BUDGET) {
        goto drop;
    }
    struct RefcBinary *frame = refc_binary_create_refc(len);
    if (UNLIKELY(frame == NULL)) {
        goto drop;
    }
    memcpy(frame->data, payload, len);
    struct EspNowEvent event = { .kind = EspNowReceived, .rssi = info->rx_ctrl->rssi, .frame = frame };
    memcpy(event.mac, info->src_addr, MAC_LEN);
    if (LIKELY(xQueueSend(data->queue, &event, 0) == pdTRUE)) {
        goto leave;
    }
    free(frame); // never listed, count still 0
drop:
    atomic_fetch_sub(&data->rx_bytes, len);
    atomic_fetch_add(&data->dropped, 1);
leave:
    callback_leave();
}

espnow_send_cb goes through the same callback_enter/callback_leave pair and queues an item without a frame, as it does today.

Listener, in the scheduler

static EventListener *espnow_event_callback(GlobalContext *glb, EventListener *listener)
{
    struct EspNowData *data = GET_LIST_ENTRY(listener, struct EspNowData, listener);
    struct EspNowEvent event;
    if (xQueueReceive(data->queue, &event, 0) != pdTRUE) {
        return listener;
    }
    bool received = event.kind == EspNowReceived;
    size_t staged = received ? event.frame->size : 0; // read now: the frame may be freed below
    size_t size = term_binary_heap_size(MAC_LEN)
        + (received ? TUPLE_SIZE(5) + TERM_BOXED_REFC_BINARY_SIZE + term_map_size_in_terms(1) : TUPLE_SIZE(4));
    Heap heap;
    if (UNLIKELY(memory_init_heap(&heap, size) != MEMORY_GC_OK)) {
        if (received) {
            free(event.frame); // still unlisted
            atomic_fetch_sub(&data->rx_bytes, staged);
        }
        atomic_fetch_add(&data->dropped, 1);
        return listener;
    }

    term message;
    if (received) {
        term payload = term_from_refc_binary(event.frame, &heap, glb); // listed, count 1
        // {espnow, rx, Mac, Payload, #{rssi => Rssi}} built as today, with payload in place of the copy
    } else {
        // {espnow, sent, Mac, ok | fail} as today
    }
    globalcontext_send_message(glb, data->receiver, message); // the copy takes the count to 2
    memory_destroy_heap(&heap, glb);                           // the sweep takes it back to 1, the receiver's
    if (received) {
        atomic_fetch_sub(&data->rx_bytes, staged);
    }
    return listener;
}

If the receiver no longer exists the send copies nothing and the sweep frees the frame, so a dead receiver leaks nothing.

Teardown

This order is needed with the current staging too. On a dual-core target the Wi-Fi task can be inside a callback on the other core while teardown runs, and whether esp_now_unregister_recv_cb or esp_now_deinit waits for it is undocumented. The handshake makes it explicit; the wait is bounded by one callback, since callbacks never block, and on a single core it never triggers because the heap lock is a critical section, so the Wi-Fi task cannot be preempted mid-callback by the scheduler.

static void teardown(struct EspNowData *data)
{
    esp_now_unregister_recv_cb();
    esp_now_unregister_send_cb();
    esp_now_deinit();
    atomic_store(&espnow_data, NULL);                 // no new callback touches data from here
    while (atomic_load(&espnow_in_callback) != 0) {  // one already running finishes first
        vTaskDelay(1);
    }
    atomic_store(&espnow_channel, 0);
    sys_unregister_listener(data->global, &data->listener); // returns only after a running dispatch
    struct EspNowEvent event;
    while (xQueueReceive(data->queue, &event, 0) == pdTRUE) {
        if (event.kind == EspNowReceived) {
            free(event.frame); // staged only, never listed
        }
    }
    xQueueRemoveFromSet(data->queue, event_set); // succeeds: the queue is empty
    vQueueDelete(data->queue);
    network_wifi_release();
    free(data);
}

espnow_wifi_restarted reads espnow_channel and dereferences nothing: today it reads data->channel from the network port's context and is safe only because network_wifi_release precedes free(data) under the mutex.

@gabrielmancini

Copy link
Copy Markdown
Author

Thanks to both of you for the reviews. Pushed six commits on top, one per area, and tested them on two ESP32 boards (ESP-IDF 5.5.5).

  • c9d4f37 espnow: Reply always, name the errors, keep the broadcast peer: stack-heap replies, error atoms, broadcast refused in add_peer/del_peer, set_channel refused while the STA is associated, a received frame over the limit dropped and counted instead of truncated, context_new checked, the queue emptied before xQueueRemoveFromSet (whose result is now checked), the drop wording in espnow.erl and the guide, and ESP-NOW on in the esp32c3 v5.2.8 build for the pre-5.5 send callback.
  • 2ee936a network: Serialize the Wi-Fi radio, and refuse an AP-only network with ESP-NOW: the mutex and the stop_network condition. For the AP-only case I took the "refuse with a clear error" option: espnow:open/1 returns {error, ap_only} while an AP-only network runs; an AP-only network started while ESP-NOW is open does not start (network:start/1 returns before the driver starts Wi-Fi, so the network process stops with {start_port_failed, ap_only_held}). Documented in the guide. A network with both sta and ap works.
  • 12d39e1 espnow: Open in a call, link the port, survive a killed port.
  • c8b7d7e libAtomVM: Add term_from_refc_binary, as a separate commit as you asked, with your code.
  • 1d6d26d espnow: Copy frames once, send one at a time, tear down after the callbacks: your staging (CONFIG_AVM_ESPNOW_RX_BUDGET, 8192 by default), the teardown handshake, and one frame in flight. Two details beyond the sketch: the send result reaches the port as a message ({espnow_sent, Result}), so only the port's handler touches the send queue; and the receive callback leaves the queue's last slot to the send result (with one frame in flight there is at most one), whose message is built on a stack heap, so a waiting send/3 cannot lose it.
  • dbed1ca espnow: Install the keys again after the network restarts the radio, from your second hardware question (below).

Hardware answers.

  1. set_channel/2 while the STA is connected: it was not a hardware observation. Tested now: with the STA associated to an AP the driver refuses with {error, sta_connected} and the channel stays the AP's; while the STA was still trying to connect, esp_wifi_set_channel itself returned ESP_FAIL. Broadcast keeps working on the AP's channel while associated.
  2. Encrypted peers across network:start/1 and network:stop/0: they did not keep working. Stopping the radio clears the keys ESP-NOW installed in it while the peer list stays, so frames went out unencrypted: the receiver's radio acknowledged them (the sender got ok) and its ESP-NOW dropped them. The network driver now calls the holder each time it stops and starts a held radio (restarted(bool bare_sta)), and ESP-NOW sets its PMK and its encrypted peers' LMKs again; it keeps them for that, apart from the state teardown frees. Tested: encrypted round trips before and after a network start/stop.

What the hardware test covered (one board driving, one answering): unicast with and without an ack (ok, {error, fail}), broadcast, a 1470-byte frame and a 1471-byte badarg, concurrent senders, broadcast refused in add_peer/del_peer, peer_exists, not_found, already_open, badarg at open, exit(Port, kill) then reopen, an owner that crashes taking the port with it, encrypted traffic across a network start/stop, the channel restored after network:stop/0, the STA associated to the other board's AP (refused set_channel, broadcast), and the AP-only refusal in both directions. All passed after dbed1ca.

I could not run erlfmt locally this time; CI will tell if espnow.erl needs it.

An espnow port driver, behind CONFIG_AVM_ENABLE_ESPNOW (off by default),
and the espnow module: frames of up to 250 bytes to a peer or broadcast,
received as {espnow, rx, Mac, Data, #{rssi := Rssi}}, with the send result
as {espnow, sent, Mac, ok | fail}.

The network driver becomes the one owner of the Wi-Fi radio. ESP-NOW holds
it (network_wifi_hold/1): with no network the radio comes up as a bare
STA, and network:stop/0 leaves it on while ESP-NOW is open, so ESP-NOW
survives the network going down and up again.

Events go through the VM listener, queued 8 deep; a frame that does not
fit is dropped and counted (stats/1).

The API follows harmon25/atomvm_espnow (Apache-2.0).

Signed-off-by: Gabriel Mancini <gabriel.mancini@gmail.com>
Signed-off-by: Gabriel Mancini <gabriel.mancini@gmail.com>
From the review:

- Replies are built on a stack heap and always sent: a request whose
  reply did not fit was dropped, and its caller waited in port:call/2
  forever.
- ESP-NOW errors a caller can act on come as atoms (not_found,
  peer_exists, peer_list_full, interface_mismatch, channel_mismatch,
  no_mem), others through esp_err_to_term/2.
- add_peer/3 and del_peer/2 refuse broadcast: the broadcast peer is the
  driver's, and deleting it broke every later broadcast.
- set_channel/2 is refused with sta_connected while the STA is
  associated, instead of relying on esp_wifi_set_channel to fail; open
  leaves the channel of an associated STA alone too.
- A received frame longer than 250 bytes (ESP-NOW v2) is dropped and
  counted, not truncated.
- A failed context_new tears the driver down; the queue is emptied
  before it leaves the event set, which fails on a non-empty queue.
- The docs say where drops happen: the driver's queue, not the
  receiver's mailbox.
- CI builds ESP-NOW on ESP-IDF 5.2 too (esp32c3), for the pre-5.5 send
  callback.

Signed-off-by: Gabriel Mancini <gabriel.mancini@gmail.com>
…h ESP-NOW

From the review:

- network:start/1 returns before its port starts Wi-Fi, so on a
  dual-core target espnow:open/1 could run network_wifi_hold while
  start_wifi ran, both on unsynchronized flags. One mutex (a no-op
  without SMP) now covers network_wifi_hold, network_wifi_release,
  start_wifi and the radio part of stop_network.
- stop_network brings a held radio back as a bare STA whenever it is
  off, not only when the network used it: start_wifi stops the held
  bare STA before it marks the radio as the network's, and a failure in
  between left ESP-NOW open with the radio off.
- An AP-only network runs the radio in WIFI_MODE_AP, where ESP-NOW,
  whose peers are on the STA interface, cannot send. network:start/1
  returns {error, ap_only_held} for one while the radio is held, and
  network_wifi_hold refuses while one runs (ESP_ERR_NOT_SUPPORTED).

Signed-off-by: Gabriel Mancini <gabriel.mancini@gmail.com>
From the review:

- espnow:open/1 creates the port cheaply and opens ESP-NOW in its first
  call, {open, Options}, whose reply says why it did not:
  {error, already_open}, {error, ap_only}, badarg for a bad option, the
  ESP-IDF error otherwise. Before, every failure was badarg from
  open_port. A port that fails to open terminates.
- The port's context does not point at the driver's state: a port killed
  (exit/2, a linked owner that crashed) is destroyed without its handler
  running, and context_destroy freed the state the Wi-Fi task, the
  listener and the radio hold still used. The state keeps the port's
  process id; only that port drives ESP-NOW ({error, not_open} for
  others), and an open that finds the port gone tears the old state
  down and opens again.
- espnow:open/1 links the port to its caller, so a supervised owner
  that crashes takes the port down and can open ESP-NOW again.
- The channel restored after network:stop/0 is kept apart from the
  state, which the network driver's callback no longer reads.

Signed-off-by: Gabriel Mancini <gabriel.mancini@gmail.com>
The non-const branch of term_alloc_refc_binary without the allocation:
wraps a refc binary created and filled elsewhere, unlisted and with a
count of 0, into a term of a heap, whose MSO list takes it. A driver can
then fill the binary a receiver will see in its own callback, outside of
the scheduler, and copy each payload once (ESP-NOW, next commit).

Signed-off-by: Gabriel Mancini <gabriel.mancini@gmail.com>
…lbacks

From the review:

- Received frames: the Wi-Fi task fills, with one copy, the refc binary
  the receiver gets (term_from_refc_binary), and queues a small item.
  The fixed slots of 250 bytes (12 KB idle with v2 frames) are gone, and
  a frame of any length ESP-IDF delivers is kept: up to 1470 bytes with
  ESP-NOW v2. CONFIG_AVM_ESPNOW_RX_BUDGET (8192 bytes) bounds what waits
  for the scheduler; past it frames are dropped and counted.
- send/3 waits for the send callback and returns ok or {error, fail}:
  the driver gives the radio one frame at a time, as the ESP-NOW guide
  recommends, since nothing in the callback says which frame it is
  about, and answers each caller from its frame's callback. The
  {espnow, sent, Mac, Result} message is gone. Frames up to 1470 bytes
  on ESP-IDF 5.4 or newer.
- Teardown: the callbacks are unregistered and the state unpublished
  first, then teardown waits for a callback already running (counted on
  entry), then the listener, the queue and its staged frames, then the
  radio. The channel restored after network:stop/0 is an atomic that
  teardown does not free.

Signed-off-by: Gabriel Mancini <gabriel.mancini@gmail.com>
Answering the review's question with a hardware test: encrypted peers did
not survive network:start/1 and network:stop/0. Stopping the radio clears
the keys ESP-NOW installed in it; the peer list stays, so frames went out
unencrypted and the receiver dropped them, while the sender saw the MAC
acknowledgement and got ok.

The network driver now calls the holder each time it stops and starts a
held radio (a network starting, and stopping back to a bare STA), and
ESP-NOW sets its PMK and its encrypted peers' LMKs again. It keeps them
for that, apart from the state teardown frees.

The guide says how an AP-only network is refused while ESP-NOW is open:
network:start/1 returns before the driver starts Wi-Fi, so the network
process stops with {start_port_failed, ap_only_held}.

Signed-off-by: Gabriel Mancini <gabriel.mancini@gmail.com>
Signed-off-by: Gabriel Mancini <gabriel.mancini@gmail.com>
@gabrielmancini

Copy link
Copy Markdown
Author

Rebased on release-0.7 (24 commits behind). The only conflict was the CHANGELOG, where both entries were kept. network_driver.c and term.h merged cleanly with the new socket/select changes. Checked an ESP32 build with CONFIG_AVM_ENABLE_ESPNOW=y and erlfmt on espnow.erl. The code is unchanged, so this is still waiting on the re-review of the six commits from 11-11.

@petermm

petermm commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

AMP code:

Latest review: ESP-NOW — PR #2472

Scope and recommendation

Request changes. Reassessed PR #2472 at b6ed4e56 on 2026-11-17. Oracle checked all seven findings from the preceding review at b21536a5 against the latest source and relevant runtime and ESP-IDF 5.5.5 contracts. All seven remain applicable; none was fixed or disproven. F3 is configuration-dependent, and the runtime consequences described below have not been independently reproduced on hardware.

The ESP-NOW driver and Erlang wrapper are unchanged between those heads. Intervening PPP, resource/select-event, and term changes do not resolve the findings. The original review also covered the feature updates since 964b72a4.

Two P1 concurrency defects remain. The smaller corrections below include proposed diffs where a local change is appropriate. These are review suggestions, not applied or tested fixes. The concurrency and teardown changes need a coherent ownership design rather than isolated one-line patches.

All espnow_driver.c locations refer to src/platforms/esp32/components/avm_builtins/espnow_driver.c at the reviewed head.

What the update resolves

  • AP-only networking: explicitly rejected in either startup order. This is a valid revised contract; automatically enabling APSTA is not required.
  • Original radio-transition races: wifi_mutex serializes network start/stop and hold/release. It does not protect all ESP-NOW state readers or SDK peer operations.
  • Failed network startup: normal network-port cleanup can restore a held bare STA, provided the recovery SDK call succeeds.
  • RX truncation and allocation: complete SDK-supported frames are accepted; refcounted payload allocation is checked. The new binary-adoption/MSO ownership path looks consistent.
  • Context allocation and reply construction: context creation is checked and replies use stack heaps. This does not establish universal OOM safety: the existing mailbox-delivery path still allocates.
  • Counter races, documentation, and CI coverage: increments are atomic, receiver-mailbox limits are accurately described, and an enabled ESP32-C3/IDF 5.2.8 build covers the older send-callback signature.
  • Original kill-induced dangling pointer: the context no longer owns the driver's state. State survives port death until the next open reclaims it, but that reclamation remains unsafe under SMP.

F1. P1 — Singleton readers race with reclamation and close

Locations: espnow_driver.c:500–508,767–768,415–446.

After the owner port dies, two schedulers can both load the retained espnow_data, observe that its port no longer exists, and call teardown(old). This permits double listener removal, queue deletion, and free. An opener or another newly created ESP-NOW port can also load the pointer just before explicit close frees it, then dereference data->port after free.

Atomic publication protects the pointer value, not the object's lifetime. The callback counter covers Wi-Fi callbacks, not scheduler-side readers. The radio mutex is acquired too late to protect this lookup/reclamation path. The scheduler releases its scheduling lock before invoking native handlers (src/libAtomVM/scheduler.c:268–299), so different ports can enter these paths concurrently. These are source-derived SMP races, not independently reproduced hardware crashes.

Required fix: Serialize scheduler-side singleton lookup/use, stale-state reclamation, initialization, and close under a lifecycle mutex or equivalent ownership protocol. A CAS claiming teardown alone does not protect readers already holding the pointer. Preserve callback/listener synchronization and define the lock order with radio ownership.

Missing regression: On a dual-core target, kill the owner and synchronize two concurrent reopen attempts. Require exactly one success and no double cleanup. Separately race open against explicit close, including uninitialized ports. Repeat with memory-error detection where practical.

F2. P1 — Restart can restore an obsolete LMK over a successful re-key

Locations: espnow_driver.c:289–308,611–632.

The restart callback snapshots encrypted-peer keys under espnow_keys_lock, then performs SDK operations after releasing it. Peer mutation does not hold wifi_mutex. A permitted sequence is:

  1. Restart snapshots peer M's old LMK.
  2. The port deletes M and successfully re-adds it with a different LMK.
  3. Restart fetches the new peer, overwrites its LMK with the old snapshot, and calls esp_now_mod_peer().

Both public peer operations can return ok, while the SDK ends with the old key and the cache remembers the new one. The gap between successful add_peer() and remember_lmk() also allows restart to miss a newly accepted peer.

Required fix: Serialize each complete SDK peer mutation plus cache update against the radio stop/start/restore transaction. Include init/deinit and sends in the ownership design so they cannot enter halfway through restoration. Use a sleeping mutex; do not hold a FreeRTOS critical section across SDK calls.

Missing regression: Pause restart after its snapshot, delete/re-add a peer using an asymmetric new LMK, then resume. Check the final SDK key and an encrypted exchange. Also pause between SDK add and cache update. The stale-key overwrite follows from the source; actual on-air behavior still needs device validation.

F3. P2 — Accepted encrypted peers can exceed restoration-cache capacity

Locations: espnow_driver.c:142,265–274,299–307.

ESP_NOW_MAX_ENCRYPT_PEER_NUM is 6 in the IDF 5.5.5 header. However, its Wi-Fi Kconfig and ESP-NOW guide permit 17 encrypted peers and default to 7 on ESP32; WIFI_INIT_CONFIG_DEFAULT() passes that configured capacity to Wi-Fi. remember_lmk() silently stops recording after six without rejecting an otherwise successful add. If a seventh peer is accepted, it is absent from restoration. The capacity mismatch is verified from source; SDK acceptance and resulting encrypted-traffic failure still need device validation.

Suggested correction: Use the SDK's total peer bound for the cache, leaving the SDK to enforce its configured encrypted-peer limit. This avoids relying on the contradictory six-peer macro:

--- a/src/platforms/esp32/components/avm_builtins/espnow_driver.c
+++ b/src/platforms/esp32/components/avm_builtins/espnow_driver.c
@@
-static struct EspNowKey espnow_lmks[ESP_NOW_MAX_ENCRYPT_PEER_NUM];
+static struct EspNowKey espnow_lmks[ESP_NOW_MAX_TOTAL_PEER_NUM];
@@
-    if (espnow_lmk_count < ESP_NOW_MAX_ENCRYPT_PEER_NUM) {
+    if (espnow_lmk_count < ESP_NOW_MAX_TOTAL_PEER_NUM) {
@@
-    struct EspNowKey lmks[ESP_NOW_MAX_ENCRYPT_PEER_NUM];
+    struct EspNowKey lmks[ESP_NOW_MAX_TOTAL_PEER_NUM];

This increases static and callback-stack storage; check the target's stack allowance. Alternatively size from the configured capacity or enforce a documented six-peer limit before accepting another encrypted peer. None of these alternatives replaces F2's synchronization fix.

Missing regression: On default ESP32, verify SDK acceptance of seven encrypted peers, restart the radio, and verify all seven keys and encrypted exchanges. Verify acceptance on hardware because the SDK header macro and configured capacity disagree. Also test a higher configured capacity where supported.

F4. P2 — A normal mailbox message can fabricate send completion

Locations: espnow_driver.c:654–664,750–776.

Any sender holding the port can send Port ! {espnow_sent, ok}. is_sent() accepts it as a listener completion. With send A outstanding and B queued, sent() falsely completes A and starts B before A's callback. A's actual callback can subsequently complete B, breaking the one-frame-at-a-time/callback-correlation contract.

This is a completion-correlation defect reachable through the port mailbox, not evidence of a remote-radio attack.

Required fix: Make the mailbox message only a wakeup and consume an actual synchronized driver-owned completion, or use private correlated completion tokens. Validating only ok | fail does not establish provenance. A small shape check is not a sufficient fix.

Missing regression: Inject the lookalike tuple while A is outstanding and B waits. Neither caller may finish, nor B reach the SDK, before the actual completion of A. Keep the legitimate callback path as a control.

F5. P2 — The caller can die before its ownership link is installed

Location: libs/avm_esp32/src/espnow.erl:82–89.

The wrapper links the port only after initialization returns. AtomVM's open_port/2 does not install that link automatically. If the caller is killed while waiting for initialization, the driver can finish with a live, unlinked owner port. Later opens return already_open; deferred reclamation cannot recover a port that still exists.

Also, a normal linked exit does not kill an ordinary non-trapping peer, contrary to the broad documentation promise that caller death closes the port.

Small suggested change for the initialization window:

--- a/libs/avm_esp32/src/espnow.erl
+++ b/libs/avm_esp32/src/espnow.erl
@@
         Port ->
+            link(Port),
             Opts = [{receiver, Receiver} | proplists:delete(receiver, Options)],
@@
-                    link(Port),
                     {ok, Port};

This narrows the initialization window, not the entire lifetime problem: the caller can still die between port creation and linking. Handle ownership-installation failure and review normal-exit behavior. If every owner exit must close the port, implement an ownership mechanism that observes every exit; otherwise narrow the promise. Monitoring is not a drop-in native-port fix: the current scheduler ignores native MonitorDownSignal. Ensure ownership cannot be stranded if the caller dies before installation.

Missing regression: Kill the caller while initialization is paused, resume, and verify another caller can open. Test normal exit separately from crash/kill.

F6. P2 — Member draining leaves queue-set notifications behind

Location: espnow_driver.c:427–437.

Teardown now drains data->queue before removing it, but direct member receives do not consume corresponding entries in the shared queue set. FreeRTOS requires selecting set notifications before receiving member items. Removal can now succeed while stale queue handles remain after deletion.

AtomVM compares unmatched handles rather than immediately dereferencing them, so an immediate use-after-free from this alone is not established. FreeRTOS removal does not purge the corresponding set notifications. Handle reuse and temporary stale-notification capacity consumption remain risks; lost events or persistent capacity exhaustion have not been established.

Required fix: Coordinate final draining and deletion with the event-poll owner, consuming both notifications and payloads without discarding unrelated drivers' events. Do not reset or blindly drain the shared set as a local workaround.

Missing regression: Repeatedly close/reopen with RX pending. Verify no stale notifications or lost set notifications, and verify another driver sharing the set continues to receive events.

F7. P2 — Dropped statistics still overflow the small-integer representation

Locations: espnow_driver.c:742–745,794.

Atomic updates fix lost increments, but term_from_int() still misrepresents counts at 134,217,728 and above on 32-bit ESP32. Use the existing boxed-integer helper and reserve its maximum heap size:

--- a/src/platforms/esp32/components/avm_builtins/espnow_driver.c
+++ b/src/platforms/esp32/components/avm_builtins/espnow_driver.c
@@
-    term_set_map_assoc(map, 0, globalcontext_make_atom(glb, ATOM_STR("\x7", "dropped")), term_from_int(atomic_load(&data->dropped)));
+    term dropped = term_make_maybe_boxed_int64(atomic_load(&data->dropped), heap);
+    term_set_map_assoc(map, 0, globalcontext_make_atom(glb, ATOM_STR("\x7", "dropped")), dropped);
@@
-        BEGIN_WITH_STACK_HEAP(PORT_REPLY_SIZE + TUPLE_SIZE(2) + term_map_size_in_terms(1), heap);
+        BEGIN_WITH_STACK_HEAP(PORT_REPLY_SIZE + TUPLE_SIZE(2) + term_map_size_in_terms(1) + BOXED_INT64_SIZE, heap);

Missing regression: Initialize the counter under controlled test conditions and check the public stats result at 134,217,727, 134,217,728, and UINT32_MAX. Assert exact nonnegative values.

Documentation suggestion

The PR description still describes the original 250-byte, asynchronous-send, eight-slot API. Current code and documentation implement SDK-dependent 250/1470-byte frame limits, synchronous sends, and a 16-slot event queue with one slot reserved for send completion and a separate RX byte budget. Refresh the PR description before merge.

If ownership remains link-based, clarify the normal-exit limitation in both libs/avm_esp32/src/espnow.erl:76–78 and doc/src/network-programming-guide.md:563–564 rather than promising cleanup on every caller death. This wording is appropriate only after fixing the ownership-installation window:

--- a/libs/avm_esp32/src/espnow.erl
+++ b/libs/avm_esp32/src/espnow.erl
@@
-%% to the caller: if the caller dies the port closes, and ESP-NOW can be
-%% opened again.
+%% to the caller: an abnormal caller exit terminates the port, and a later
+%% open reclaims its ESP-NOW state. A normal caller exit does not close it;
+%% call close/1 explicitly before returning normally.

Verification and merge gate

  • Fix F1 and F2 before merging. Address the narrower correctness findings and add regression evidence for the revised lifetime, synchronous-send, and encrypted-coexistence contracts.
  • Oracle's latest reassessment inspected the exact latest commit via git show, including supporting runtime changes, rather than treating the older checkout as the PR head. This was source/API inspection only: no local builds, automated tests, or hardware reproduction were performed. The earlier review recorded a passing git diff --check 964b72a4 upstream/pr-2472; that historical check is not build or runtime evidence for the latest head.
  • At the 2026-11-17 update, GitHub reported successful ESP32 build-matrix jobs, including ESP32/IDF 5.5.5 with PPP and ESP32-C3/IDF 5.2.8. CI was incomplete and not green: macOS Intel/OTP 27 and OTP 28 build-and-test jobs failed, and other jobs remained pending. Those failures have not been triaged or attributed to this feature.
  • The contributor reports hardware-observed encryption-key loss across Wi-Fi restart. This is contributor evidence, not an independently reproduced result. Device validation is still needed for callback quiescence, outstanding synchronous-send completion across restart, encrypted continuity, and restoration-error handling.
  • No automated regression tests were added in the updated feature diff. Test SDK-dependent 250/1470-byte boundaries, actual synchronous send results, and idle-receiver delivery without unrelated commands/timers.
  • The system queue-set listener already supplies the wakeup. The separate fork's dummy mailbox-message wakeup is not needed in this architecture.
  • This document contains proposed changes only. No implementation changes, commits, or GitHub comments were made.

@gabrielmancini

Copy link
Copy Markdown
Author

Thanks for the reassessment. All seven findings are addressed in one commit on top, a83da480 (espnow: One lock for the radio and ESP-NOW, a private send completion). The PR description is refreshed to the current API: frame limits, synchronous send, queue and RX budget, ownership.

Ownership design (F1, F2). Rather than adding a second lifecycle mutex next to the radio's, ESP-NOW now uses the network driver's radio lock for everything it does in the scheduler. It is exposed as network_wifi_lock/0 and network_wifi_unlock/0, and espnow_consume_mailbox takes it around each message: open, stale-state reclamation, close, add_peer plus the LMK cache update, del_peer plus the cache removal, sends, and the send completion.

  • The network driver already holds that lock while it stops and starts the radio and calls espnow_wifi_restarted. A restart therefore cannot interleave with a peer mutation, an open, a close or a send.
  • espnow_keys_lock (the critical section) is gone: every reader and writer of the key cache is under the same sleeping mutex, together with the SDK call that goes with it.
  • network_wifi_hold/1 and network_wifi_release/0 are now called with the lock taken.
  • Lock order is the radio lock, then the listeners lock. Teardown takes the listeners lock under the radio lock. No listener takes the radio lock: the ESP-NOW listener only sends messages, and the network driver takes wifi_mutex only in its port handler (start, stop) and its destroy callback.

F3. The cache is sized ESP_NOW_MAX_TOTAL_PEER_NUM, and the SDK enforces its configured encrypted limit. The restart callback no longer copies the cache to the stack, since it runs under the lock.

F4. The listener stores the result in an atomic completion field of the driver state and sends the port the bare atom espnow_sent as a wakeup. The handler atomic_exchanges the field: a lookalike message from any process finds no result and completes nothing. The tuple shape no longer carries any meaning.

F5. link(Port) comes before {open, Opts}, and a failed open unlinks. Your wording on normal exits is in both espnow.erl and the guide.

F6. Teardown no longer drains the member queue. After the callbacks and the radio are released, and with the listeners lock held so no listener runs, it marks the state closing:

  • an empty queue has no notification left in the set (each notification is consumed by the poll loop's select before the listener receives its item), so it is removed and deleted right away;
  • a non-empty queue keeps its listener. The poll loop delivers the remaining events through the set as usual, and the listener drops them. After the last one it removes the queue from the set, deletes it, frees the state, and returns NULL, which process_listener_handler handles after saving next.

The listener is now the first member of the state, because sys_listener_destroy frees the listener pointer at VM teardown.

F7. term_make_maybe_boxed_int64 is used, and the stack heap reserves BOXED_INT64_SIZE.

Evidence so far: source and build only. The ESP32/IDF 5.5.5 build with ESP-NOW and Ethernet enabled is clean, with no warnings, and erlfmt passes. The hardware regressions you listed come next on two boards:

  • concurrent reopen after an owner kill;
  • open racing close;
  • restart paused after the snapshot, with a re-key;
  • seven encrypted peers across a restart;
  • a lookalike espnow_sent with A outstanding and B queued;
  • a caller killed during open;
  • close/reopen with RX pending while another driver shares the set.

I will post the results here. Some of these need a test hook to pause at the right point, and I will keep such hooks out of this PR unless you would like them in.

@gabrielmancini

Copy link
Copy Markdown
Author

Hardware regressions for a83da480. I ran them on two dual-core ESP32 boards (SMP AtomVM, IDF 5.5.5, channel 1, PMK set) with a small test app: a partner board that echoes and floods on command, and a tester board. The same battery ran on the previous head b6ed4e56 to check that it detects the defects.

finding test b6ed4e56 a83da480
F3 6 encrypted fake peers, then the partner as the 7th encrypted peer (MAX_ENCRYPT_NUM=7); radio restart (network:start/1 + network:stop/0); encrypted ping/pong ping lost after restart ok
F4 two senders, 40 sends each, to a peer that never ACKs, while a third process sends the port espnow_sent and {espnow_sent, ok} every ms; control: 20 sends to the partner 14/80 returned ok 0/80 ok, control 20/20
F5 opener killed 0–19 ms into espnow:open/1, ×40; then open/1 must succeed within 1 s stranded: every later open is already_open 40/40
F5 an opener exits normally holding the port (not reached) already_open while held, close/1 from another process, then open ok, as documented
F1 owner killed, then two processes race open/1 (retrying already_open for 300 ms), ×60 (not reached) exactly one winner, 60/60
F1 close/1 and open/1 of the same port from two other processes, ×60 (not reached) 60/60
F6 partner flooding broadcast (2 ms); 150 × open, 0–5 ms, close, with a sink receiver; then 50 more with a network STA running; then 3 × sta_connect/0 (not reached) 150 + 50 cycles ok; free heap 186.6 → 185.3 KB (a leaked queue per cycle would be ~75 KB); frames received after; dropped => 0; each reconnect's disconnected event arrives through the shared event set
F2 1,800+ del/add re-keys of the partner alternating two LMKs while another process does 8 radio restarts; then both sides on the final key, one more restart, encrypted ping ok (the race window was not hit) ok

Notes:

  • The F5 strand on the old head makes every later test in that run fail with already_open. So F1 and F6 have no "before" column: they never ran there.
  • F2 shows no difference without a pause hook. The fix is the lock design described above, not a value this stress could distinguish. I can add a pause hook behind a test-only Kconfig if you want a deterministic regression for it.
  • F7 is not exercised on hardware, since it needs 2^27 drops.

The test app is about 400 lines of Erlang plus two start modules. It needs two boards, so it does not fit the QEMU suite. I can push it as an example or attach it if that would help.

From the review of 2026-11-17:

- The radio lock (network_wifi_lock/0, the network driver's wifi_mutex)
  is taken around everything the scheduler does with ESP-NOW: open,
  close, reclaiming a killed port's state, peers and their keys, sends.
  The network driver holds it while it restarts the radio and calls
  espnow_wifi_restarted, so two ports can no longer both reclaim the same
  state, an open cannot use a state a close is freeing, and a peer added
  again with a new LMK cannot be overwritten by a restart with the old
  one (F1, F2). network_wifi_hold/1 and network_wifi_release/0 are called
  with it taken. Lock order: the radio lock, then the listeners lock.
- The LMK cache has room for every peer (ESP_NOW_MAX_TOTAL_PEER_NUM): the
  SDK takes as many encrypted peers as
  CONFIG_ESP_WIFI_ESPNOW_MAX_ENCRYPT_NUM (7 by default), not the 6 of
  ESP_NOW_MAX_ENCRYPT_PEER_NUM (F3).
- The send result is left by the listener in the driver state, and the
  atom espnow_sent only wakes the port up: a lookalike message from any
  process finds no result and completes nothing (F4).
- espnow:open/1 links the port before ESP-NOW starts, and the docs say a
  normal caller exit does not close it (F5).
- Close no longer empties the queue itself, which left its notifications
  in the shared event set. Under the listeners lock, an empty queue goes
  at once; one with events left goes from the listener, after it takes
  the last one through the event set as usual (F6). The listener is the
  first member of the state, which sys_listener_destroy frees.
- stats/1 boxes a dropped count past the small integers (F7).

Signed-off-by: Gabriel Mancini <gabriel.mancini@gmail.com>
@gabrielmancini

Copy link
Copy Markdown
Author

Formatting-only amend: clang-format 16 removed one blank line left in espnow_consume_mailbox. The commit referred to above as a83da480 is now d217abb1, with the same content otherwise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants