Skip to content

Draft Support for Very Large Collection - #21852

Open
pcanal wants to merge 26 commits into
root-project:Arlesiennefrom
pcanal:very_large_collection
Open

pcanal wants to merge 26 commits into
root-project:Arlesiennefrom
pcanal:very_large_collection

Conversation

@pcanal

@pcanal pcanal commented Apr 8, 2026

Copy link
Copy Markdown
Member

Not ready for review -- just running the CI for a checkpoint.

@pcanal
pcanal marked this pull request as draft April 8, 2026 20:11
@pcanal pcanal self-assigned this Apr 8, 2026
@github-actions

github-actions Bot commented Apr 9, 2026 •

Copy link
Copy Markdown

Test Results

    20 files      20 suites   2d 22h 34m 24s ⏱️
 3 886 tests  3 886 ✅  0 💤 0 ❌
68 738 runs  68 662 ✅ 76 💤 0 ❌

Results for commit d344fa1.

♻️ This comment has been updated with latest results.

@pcanal
pcanal force-pushed the very_large_collection branch from 2919c36 to c364f50 Compare April 10, 2026 21:29
@pcanal
pcanal force-pushed the very_large_collection branch 6 times, most recently from 6891a9f to f3f405d Compare April 23, 2026 22:09
@pcanal
pcanal force-pushed the very_large_collection branch 4 times, most recently from c8692de to 59db0dc Compare May 5, 2026 15:33
@pcanal
pcanal force-pushed the very_large_collection branch 2 times, most recently from b5f2099 to 2e2c703 Compare June 19, 2026 22:24
@pcanal pcanal closed this Jun 20, 2026
@pcanal pcanal reopened this Jun 20, 2026
@pcanal
pcanal force-pushed the very_large_collection branch from 2e2c703 to 408b5f4 Compare June 20, 2026 17:21
It is staying signed for historical reason since we do not plan to support collection larger
than std::numeric_limit<std::int64_t>::max() as onfile we reserve the sign bit as a marker.
note: std::signbit(int) not avail on Windows
NOTE: this might be backward incompatible
The change in TStreamerInfo class version changes the compressed
size of the TStreamerInfo record in the file.

In this test, the file is updated/added-to, consequently the
keys/baskets that are written after the first TStreamerInfo
record are moved slightly and this changes the compressed size
of the TTree object on file.
This is use in particular for TClass to register the
upcoming object type.
@pcanal
pcanal force-pushed the very_large_collection branch from c248d7b to d344fa1 Compare November 11, 2026 16:41
@pcanal
pcanal marked this pull request as ready for review November 11, 2026 16:41

@jblomer jblomer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome!

Comment thread core/meta/src/TClass.cxx
// we fail.
{
TClass__GetCallingNewRAII callingNew(defConstructor);
p = gCling->ClassInfo_New(GetClassInfo(),nElements, arena);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to update this, too, since it takes an int parameter

Comment thread core/base/inc/TBuffer.h
char *Buffer() const { return fBuffer; }
char *GetCurrent() const { return fBufCur; }
Int_t BufferSize() const { return fBufSize; }
Size_t BufferSize() const { return fBufSize; }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requires updating TMapRec (see MapFile::Update()).

Comment thread core/base/inc/TBuffer.h
Size_t BufferSize() const { return fBufSize; }
void DetachBuffer() { fBuffer = nullptr; }
Int_t Length() const { return (Int_t)(fBufCur - fBuffer); }
Size_t Length() const { return (Size_t)(fBufCur - fBuffer); }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Requires updating TDNDData (plus a follow-up fix in TGDNDManager::HandleSelectionRequest()).
  • Int_t pos in TClass::Load().
  • MPSendRecv passes a buffer length to TSocket::SendRaw().
  • TContainerConverters has a line Int_t currentpos = b.Length().
  • In TStreamerInfo::ReadBuffer there is a line Int_t bufpos = b.Length().
  • Fix the return statement in RNTupleSerializer::SerializeStreamerInfos.
  • Fix bufOffset in TTreeReaderValueFastBase::Adjust().
  • Maybe also add an overflow- checked LengthAsInt() for its use in TKey.

Comment thread core/cont/inc/TVirtualCollectionProxy.h Outdated
kCustomAlloc = BIT(5) ///< The collection has a custom allocator.
};

using size_type = UInt_t;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SizeType_t? (Not sure what the consistent spelling is in this context).

int res = unittestByteCount();
res += testReadWriteObjectAny();
return res;
} No newline at end of file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing newline at the end of the file.

return ru.ru_maxrss / 1024.0;
#endif
}
#define TIME_SUBTEST(timing, label, call) \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Align backslashes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants