Retain complete index keys

[ryanduguid]

Summary

Tests for the capped index queue and the published key set from #1618.

• test_index_coverage.py checks the exact processing order for an uncapped queue and for caps of 2 and 4, with and without streams. The expected orders are what index_media() produced before it also returned the published keys, so the rework doesn't change which media get processed or in what order.
• It also checks that the published keys include every video and stream, including those beyond the cap.
• c5ab922 / 285f85e register every stream key before the stream allowance is applied.

The source-directory removal guard has moved out of this PR, as requested. It will come back separately with more testing.

Tests

The full Django suite passes locally (67 tests, Python 3.14 on Linux).

[tcely]

Because videos and streams are both essentially unlimited sized lists, we try to be as memory-efficient with them as is possible.

[meeb]

I would concur this needs some additional details. Have I missed something or does this just check a directory is in the download root during delete_all_media_for_source() with some bonkers AI tests?

[meeb]

Right, so this PR is actually about not following symlinks?

[tcely]

@meeb The diff being a bit confused is my fault. Some of the changes for the key tracking are already on the base branch. I'm happy to add those behind a setting. The rest of the changes do show in the diff view of this pull request.

---

I've changed the base back to main and opened #1618 for the changes that were already on the old base testing branch.

[ryanduguid]

Thanks both, and thanks @tcely for the key-tracking rework.

• c5ab922 fixes one thing in it: with islice(), register_key() only ran on the streams that fit under the cap, so the rest were missing from the published keys and the index test failed. It now uses your earlier suggestion, tuple(map(register_key, streams))[: allowed_streams].
• On the directory deletion, you were right that resolve() would break symlinked source directories. 6b1f068 drops that validation and restores the original asserts. The only remaining change is just before rmtree(): it skips the removal (and logs an error) if the directory isn't below DOWNLOAD_ROOT, using the same lexical check as the safe_join() in the source form, so symlinks aren't followed. Database cleanup is unchanged.
• The AST-based test is gone. The new test calls delete_all_media_for_source.func directly, as you suggested.

The full Django suite passes locally (69 tests).

[tcely]

@ryanduguid Regarding c5ab922:

Yes, your new test helpfully pointed out that the lazy evaluation via islice wasn't enough. I was in the middle of editing on the test branch when I saw that commit was pushed. 🙂

---

@ryanduguid

It is probably best to split your deletion changes off into another pull request. They will need more thought and testing.

I would be happy to merge this with the enhanced tests after those changes are no longer part of this pull request.

Please also give some thought to how you would like to document the way this feature works when both settings are used. The wiki source is available as a submodule for your convenience. If you write some changes and include the new / modified files under docs then I will take care of committing those for you, if you ask.

[meeb]

Ah right, thanks @tcely makes more sense now. Nothing to add beyond your own comments for now.

[ryanduguid]

@tcely done in efa0d00:

• The deletion guard is out of this PR. I'll bring it back as a separate PR once it has had more thought and testing, including your resolve(strict=True) / is_relative_to() suggestions against symlinked source directories.
• The index test now checks the exact processing order (uncapped, cap 2 and cap 4, with and without streams) against what index_media() produced before the rework, as well as the published key set.

On documentation: happy to write the wiki section explaining how MAX_ENTRIES_PROCESSING and the cleanup setting work together, once #1618 settles the setting's name (CLEANUP_WHICH or WHAT_MEDIA_TO_CLEANUP). I'll add it under docs/ and ask you to commit it.

[tcely]

@ryanduguid Thanks. Once a setting name is decided, I will merge that pull request first and update your branch from main.

[ryanduguid]

@tcely thanks. With #1618 settling on INDEXING_CLEANUP_UNAVAILABLE_POLICY, I've added the wiki update in 50502e1 as docs/Settings.md: a copy of the wiki's Settings page with a new INDEXING_CLEANUP_UNAVAILABLE_POLICY section and two notes under MAX_ENTRIES_PROCESSING explaining how they interact. Could you commit it to the wiki? If the setting name changes before #1618 merges, I'll update it.

[tcely]

@ryanduguid The wiki changes should be live already. Thanks for your contributions.