Repository navigation
feat(hive): add Iceberg-to-HMS schema and record conversion helpers - #904
Merged
Merged
Conversation
MisterRaindrop
force-pushed
the
feat/hive-conversion-layer
branch
from
September 2, 2026 06:34
69b9934 to
8a4041c
Compare
Creating an Iceberg table through HMS means rendering its schema as Hive DDL. Add `TypeToHiveString` for one `Type` and `SchemaToHiveColumns` for a whole `Schema`; nested struct / list / map render recursively. `timestamptz` follows iceberg-java and emits `timestamp with local time zone`, since engines reading through HMS take their column types from this DDL. Java's sub-Hive-3 fallback is not reproduced: there is no Hive-version probe here and the vendored IDL is Hive 4.0.1. Iceberg v3 types that no reference maps to a real Hive type (timestamp_ns, variant, geometry, geography, unknown) return kNotImplemented rather than guessing a spelling.
HMS speaks `Database` and `Table`; Iceberg speaks `Namespace`, `TableIdentifier` and `Schema`. Add the conversion between them, plus the parameter keys (`metadata_location`, `table_type`, `EXTERNAL`) that iceberg-java and iceberg-rust use to recognise an Iceberg table, and the LazySimpleSerDe descriptor that keeps Spark and Trino reading it. The helpers produce plain structs rather than Thrift objects, so HmsClient stays the only translation unit that includes the generated bindings. A database owner is carried by `hive.metastore.database.owner` and `hive.metastore.database.owner-type`, the keys both references use; a bare `owner` is the table owner key. `owner-type` requires an owner and must name an HMS PrincipalType -- USER, GROUP or ROLE, compared case-insensitively.
wgtmac
force-pushed
the
feat/hive-conversion-layer
branch
from
October 9, 2026 13:52
8a4041c to
c5541b4
Compare
Member
|
Thanks for the work on this PR and sorry for the delay! I pushed a small update to align the conversion logic with Java to avoid further delay:
|
wgtmac
approved these changes
Oct 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Next step of the HiveCatalog split (follows #796): two conversion layers between Iceberg's catalog model and the HMS record shape. No Thrift, no I/O, fully unit tested.
hive_schema— renders an IcebergSchemaas the Hive DDL column list HMS stores, nested types included.timestamptzusestimestamp with local time zone, following iceberg-java; the sub-Hive-3 fallback is not implemented.variantmaps tounknown, following Java. Other unsupported v3 types returnkNotImplemented.hive_utils—Namespace/TableIdentifier↔ HMSDatabase/Table, the Iceberg-on-HMS parameter keys (metadata_location,table_type,EXTERNAL), and the LazySimpleSerDe descriptor Spark and Trino expect. It also translatesgc.enabledtoexternal.table.purge. Plain structs rather than Thrift objects, soHmsClientstays the only translation unit that includes the generated bindings.Database owner uses
hive.metastore.database.owner/hive.metastore.database.owner-type. Table owner useshive.metastore.table.owner. A bareownerremains an ordinary table property. Missing owner and location defaults are resolved by the catalog layer.Nothing calls any of this yet —
HiveCatalogremains a stub;HmsClient's Thrift wrappers and the catalog CRUD paths come next.43 new unit tests;
hive_catalog_testnow runs 60 cases.