[ak-grg]

sort only uses the locale collator when the locale is UTF-8 (and not C/POSIX), but comm and join decided with should_use_locale_collation(), which only checked that the locale name isn't C/POSIX. So with LC_ALL=en_US or ru_RU.KOI8-R, sort wrote byte order, comm/join compared with the collator, and lines in both files were reported as unique to each.

should_use_locale_collation() now also requires a UTF-8 encoding, and init_locale_collation() calls it instead of repeating the check, so sort, comm, join and ls all get the same answer. ls doesn't change in practice: in non-UTF-8 locales it already fell back to byte comparison.

Tests use the example from the issue in comm and join, under en_US and ru_RU.KOI8-R. The locale is read from LC_ALL, so they don't need it installed and run on every platform.

Closes #15802

I used Claude to help with this change. I've gone through the diff and ran the comm, join, sort and ls tests, fmt and clippy myself.

[codspeed]

Merging this PR will not alter performance

1 warning, 1 note

⚠️ 118 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

ℹ️ 1 benchmark has no flamegraph

The flamegraph wasn't generated: most of the time was spent in runtime internals, which would leave nothing useful to analyze.

See affected benchmarks

✅ 401 untouched benchmarks
⏩ 70 skipped benchmarks¹

---

_{Comparing ak-grg:collate-utf8-only-15802 (9f3eb59) with main (0f0a079)}

Footnotes

1. 70 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[sylvestre]

needs to be rebased and you should look at the perf regression please