Repository navigation
chore: Update to arrow/parquet 60.0.0 and object_store 0.14.2 - #25335
Merged
Merged
Conversation
Pin arrow/parquet crates to arrow-rs rev 901e69f913bb8a1738009218898959e8f4cd7ec8 via [patch.crates-io] until arrow 60.0.0 is released. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tore 0.14, etc) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # datafusion/core/tests/user_defined/user_defined_aggregates.rs # datafusion/core/tests/user_defined/user_defined_scalar_functions.rs # datafusion/core/tests/user_defined/user_defined_window_functions.rs # datafusion/ffi/src/udaf/mod.rs # datafusion/optimizer/src/optimizer.rs
…anges - data_buffers() now returns &Arc<[Buffer]> and GenericByteViewArray::new_unchecked takes Arc<[Buffer]> - ParquetMetaData::column_index()/offset_index() replaced by page_index() returning the new PageIndex struct - PageIndex cannot be constructed outside the parquet crate (apache/arrow-rs#10824): patch serialized index bytes in statistics_order_tests and drop the synthetic page index from single_column_metadata until upstream exposes a constructor - New ColumnOrder/SortOrder variants: INT96 now reports SortOrder::INT96_TIMESTAMP (still treated as untrusted for min/max), and ColumnOrder union fields 2/3 are now recognized
…adapt to API changes
# Conflicts: # Cargo.lock # datafusion/common/src/dfschema.rs # datafusion/physical-expr/src/planner.rs
- Use PageIndexBuilder (apache/arrow-rs#10842) to restore the page index tests in statistics_order_tests.rs - Update RunEndEncoded Display expectation (apache/arrow-rs#10840) - Update decimal formatting expectation (apache/arrow-rs#10869) - Update memory size expectations Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
github-actions
Bot
added
sql
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #25335 +/- ##
==========================================
- Coverage 82.36% 82.35% -0.01%
==========================================
Files 1137 1137
Lines 432962 432944 -18
Branches 432962 432944 -18
==========================================
- Hits 356610 356563 -47
- Misses 54825 54846 +21
- Partials 21527 21535 +8 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
alamb
changed the title
60.0.0 and object_store 0.14.2
Contributor
|
Looks like smooth upgrade, thanks @alamb |
Contributor
Author
|
Thank you very much for the review @comphead @adriangb and @kevinjqliu -- as I have been trying to get this release through for a while I am going to throw this into the merge queue. 🫡 -- and now hopefully I can work on some non upgrade related PRs |
alamb
enabled auto-merge
September 18, 2026 20:29
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Sep 18, 2026
Contributor
Author
🤔 but they all seem to have passed |
Contributor
Author
|
haohuaijin
pushed a commit
to haohuaijin/arrow-datafusion
that referenced
this pull request
Sep 19, 2026
…pache#25335) ## Which issue does this PR close? - Related to apache/arrow-rs#9880 - replaces / closes apache#24366 - closes apache#23693 ## Rationale for this change Update to the latest arrow/parquet release, `60.0.0`. See the upstream [CHANGELOG](https://github.com/apache/arrow-rs/blob/60.0.0/CHANGELOG.md) for details. ## What changes are included in this PR? 1. Update to arrow/parquet `60.0.0` and `object_store` `0.14.2` 2. Adjust code for upstream API differences (see inline comments) 3. Update test expectations for upstream behavior changes ## What is the testing strategy for this PR? Existing CI. ## Are there any user-facing changes? New dependency versions. Also, following the upstream change, a few DataFusion APIs now use arrow's new [`Metadata`](https://docs.rs/arrow/latest/arrow/datatypes/struct.Metadata.html) type (thanks @emilk !) instead of `HashMap<String, String>` --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
nuno-faria
pushed a commit
to nuno-faria/datafusion
that referenced
this pull request
Sep 20, 2026
## Which issue does this PR close? - N/A ## Rationale for this change Arrow 60 allows sharing the backing-buffer collection of a `StringViewArray` with a single Arc ref-count bump. apache#25335 took advantage of this in most places but missed a few spots; this fixes those locations to avoid allocating a new collection. ## What changes are included in this PR? See above. ## What is the testing strategy for this PR? Tests pass. ## Are there any user-facing changes? No.
alamb
pushed a commit
to pydantic/datafusion
that referenced
this pull request
Sep 22, 2026
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes #N/A. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. Please explain the problem you are trying to solve in terms of the user-visible behavior, rather than the implementation. For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the implementation. "COUNT(DISTINCT) returns wrong results when the column contains nulls" is the user-visible problem. --> In commit #094ad31, @alamb fixed CI failure by temporarily ignoring the two quick-xml RustSec advisories in the audit workflow and documented that the ignores should be removed once object_store upgrades to quick-xml >= 0.41.0. Now that PR apache#25335 has been merged, we can remove these temeporary ignores. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here, but it is sometimes worth providing a summary of the individual changes in this PR. --> Remove the two temporary audit ignores in security_audit.sh. ## What is the testing strategy for this PR? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code Briefly describe how this PR is tested, and point to the specific tests you added. For example: 'This new feature is covered by the `sqllogictest` cases added in `foo.slt`'. If this PR does not add tests, explain why. For example, if the change is already covered by existing tests, please mention it. You should also check the `codecov` bot reply on this PR to confirm the changed code is exercised. --> Just CI update ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please add the `api change` label. --> No Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>
lilic
added a commit
to influxdata/arrow-datafusion
that referenced
this pull request
Sep 24, 2026
datafusion/common/src/utils/memory.rs as of apache/datafusion main 9951f0b (2026-09-24): apache#24319 introduced the counter and apache#24364, apache#24837, apache#25335 refined it. Counts each buffer once across a sequence of batches; replaces the fork-local helpers.
15 tasks done
Omega359
pushed a commit
to Omega359/arrow-datafusion
that referenced
this pull request
Oct 11, 2026
…pache#25335) ## Which issue does this PR close? - Related to apache/arrow-rs#9880 - replaces / closes apache#24366 - closes apache#23693 ## Rationale for this change Update to the latest arrow/parquet release, `60.0.0`. See the upstream [CHANGELOG](https://github.com/apache/arrow-rs/blob/60.0.0/CHANGELOG.md) for details. ## What changes are included in this PR? 1. Update to arrow/parquet `60.0.0` and `object_store` `0.14.2` 2. Adjust code for upstream API differences (see inline comments) 3. Update test expectations for upstream behavior changes ## What is the testing strategy for this PR? Existing CI. ## Are there any user-facing changes? New dependency versions. Also, following the upstream change, a few DataFusion APIs now use arrow's new [`Metadata`](https://docs.rs/arrow/latest/arrow/datatypes/struct.Metadata.html) type (thanks @emilk !) instead of `HashMap<String, String>` --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Omega359
pushed a commit
to Omega359/arrow-datafusion
that referenced
this pull request
Oct 11, 2026
## Which issue does this PR close? - N/A ## Rationale for this change Arrow 60 allows sharing the backing-buffer collection of a `StringViewArray` with a single Arc ref-count bump. apache#25335 took advantage of this in most places but missed a few spots; this fixes those locations to avoid allocating a new collection. ## What changes are included in this PR? See above. ## What is the testing strategy for this PR? Tests pass. ## Are there any user-facing changes? No.
Omega359
pushed a commit
to Omega359/arrow-datafusion
that referenced
this pull request
Oct 11, 2026
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes #N/A. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. Please explain the problem you are trying to solve in terms of the user-visible behavior, rather than the implementation. For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the implementation. "COUNT(DISTINCT) returns wrong results when the column contains nulls" is the user-visible problem. --> In commit #094ad31, @alamb fixed CI failure by temporarily ignoring the two quick-xml RustSec advisories in the audit workflow and documented that the ignores should be removed once object_store upgrades to quick-xml >= 0.41.0. Now that PR apache#25335 has been merged, we can remove these temeporary ignores. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here, but it is sometimes worth providing a summary of the individual changes in this PR. --> Remove the two temporary audit ignores in security_audit.sh. ## What is the testing strategy for this PR? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code Briefly describe how this PR is tested, and point to the specific tests you added. For example: 'This new feature is covered by the `sqllogictest` cases added in `foo.slt`'. If this PR does not add tests, explain why. For example, if the change is already covered by existing tests, please mention it. You should also check the `codecov` bot reply on this PR to confirm the changed code is exercised. --> Just CI update ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please add the `api change` label. --> No Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
60.0.0(August 2026) arrow-rs#9880object_storeto0.14.1#23693Rationale for this change
Update to the latest arrow/parquet release,
60.0.0. See the upstream CHANGELOG for details.What changes are included in this PR?
60.0.0andobject_store0.14.2What is the testing strategy for this PR?
Existing CI.
Are there any user-facing changes?
New dependency versions.
Also, following the upstream change, a few DataFusion APIs now use arrow's new
Metadatatype (thanks @emilk !) instead ofHashMap<String, String>