Skip to content

tee: own page caches using memfd for 2+ outputs - #15854

Closed
oech3 wants to merge 1 commit into
uutils:mainfrom
oech3:tee-memfd
Closed

oech3 wants to merge 1 commit into
uutils:mainfrom
oech3:tee-memfd

Conversation

@oech3

@oech3 oech3 commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Closes #15803 .
Note that this causes large performance drop:

> yes|tee /dev/null|pv>/dev/null
 133GiB 0:00:02 [66.8GiB/s]
> yes|tee-rw /dev/null|pv>/dev/null
10.1GiB 0:00:02 [5.09GiB/s]

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

GNU testsuite comparison:

GNU test failed: tests/tee/tee. tests/tee/tee is passing on 'main'. Maybe you have to rebase?

@oech3
oech3 force-pushed the tee-memfd branch 4 times, most recently from 5681e0d to e0a00ac Compare October 26, 2026 04:24
@oech3

This comment was marked as resolved.

@oech3
oech3 marked this pull request as draft October 26, 2026 04:36
@oech3 oech3 changed the title tee: restrict fast-path to 1 output tee: own page caches using memfd for 2+ outputs Oct 26, 2026
@oech3
oech3 force-pushed the tee-memfd branch 11 times, most recently from 77d41f4 to b860187 Compare October 26, 2026 09:46
@oech3
oech3 marked this pull request as ready for review October 26, 2026 09:46
Comment thread tests/by-util/test_tee.rs Resolved
@oech3
oech3 force-pushed the tee-memfd branch 8 times, most recently from 15cd37b to 7027117 Compare October 26, 2026 14:43
@oech3
oech3 marked this pull request as draft October 29, 2026 00:26
@oech3 oech3 changed the title tee: own page caches using memfd for 2+ outputs tee: restrict fast-path to 1 writer case Oct 29, 2026
@oech3

oech3 commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

Actually, this does not help. memfd must be truncated to 0 if writes contains a pipe (common situation) which made tee slower than read/write.

@oech3
oech3 force-pushed the tee-memfd branch 2 times, most recently from 56f6e40 to eb64e60 Compare October 29, 2026 00:44
@oech3
oech3 marked this pull request as ready for review October 29, 2026 00:46
@oech3

oech3 commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

But it might worth to try to not reuse memfd.

@xtqqczze

Copy link
Copy Markdown
Collaborator

I think we can still have a fast path with multiple writers when:

  • The file descriptor refers to a memfd.
  • The producer applied the following seals: F_SEAL_WRITE, F_SEAL_GROW, and F_SEAL_SHRINK.

@oech3

oech3 commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

No (at least with if memfd was reused). Seals are unrelated at here since memfd is already provate resource (post-seals are actually unnecessary syscalls).

@oech3
oech3 marked this pull request as draft October 29, 2026 01:00
@xtqqczze

Copy link
Copy Markdown
Collaborator

I'm not suggesting applying seals necessarily. My point is that if these seals are already present, we can treat the source as immutable.

@oech3

oech3 commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

Post-sealing means it cannot be recycled. If we can keep fast-path by someway, I'll drop(memfd) after use instead of sealing.

@xtqqczze

Copy link
Copy Markdown
Collaborator

I’m saying we can check whether the producer has already applied the seals, not apply them ourselves.

@oech3

oech3 commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

It seems unrelated at here.

@oech3

oech3 commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

One of a option is post-sealing under debug build only to refuse wrong logic at tests.

Both of ftruncateing memfd and unreusing memfd (needed to detach page caches) were slower than read/write anyway.

@oech3

This comment was marked as outdated.

@oech3 oech3 changed the title tee: restrict fast-path to 1 writer case tee: own page caches using memfd for 2+ outputs Oct 29, 2026
@oech3
oech3 force-pushed the tee-memfd branch 2 times, most recently from ce1eed9 to c72e35a Compare October 29, 2026 02:53
@oech3

oech3 commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor Author

I think there is no fast-path for multiple outputs wihout using infinite RAM.
We can reuse memfd if writers does not contain pipe, or restrict fast-path to 1st 1 MiB, but it is too limited, or read/write is faster.
Closing. I'll open a simple cleanup PR when CI is green.

@oech3 oech3 closed this Oct 29, 2026
@oech3
oech3 deleted the tee-memfd branch October 29, 2026 04:50
@oech3
oech3 restored the tee-memfd branch October 30, 2026 13:12
@oech3
oech3 deleted the tee-memfd branch October 30, 2026 13:47
@oech3
oech3 restored the tee-memfd branch October 30, 2026 13:49
@oech3
oech3 deleted the tee-memfd branch October 30, 2026 14:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tee: zero-copy lets stdout and FILE differ if input changes after tee exits

2 participants