Skip to content

Initial Snowflake support - #146

Open
tim-band wants to merge 10 commits into
mainfrom
snowflake
Open

tim-band wants to merge 10 commits into
mainfrom
snowflake

Conversation

@tim-band

@tim-band tim-band commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Can connect to a Snowflake database like so:

snowflake://TIM.BAND%40NHS.NET@ATKJNCU-LDS_DEV/OLIDS_SHARE_PSEUDO/OLIDS_PSEUDO?warehouse=SNOWFLAKE_LEARNING_WH&authenticator=externalbrowser

We can get the tables with the new flag datafaker make-tables --views-to-tables, but the PK and FK constraints don't come across.

Fixes #148

Comment thread pyproject.toml
ty = "^0.0.43"
colorama = "^0.4.6"
snowflake-sqlalchemy = "^1.11.1"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we add snowflake-connector-python = { version = ">=4.8,<5", optional = true, extras = ["secure-local-storage"] } here?

When I ran poetry run datafaker make-tables after setting SRC_DSN, I was prompted to install snowflake-connector-python[secure-local-storage] via pip install snowflake-connector-python[secure-local-storage].

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we going to make snowflakes as dependencies rather than optional? Anyone that uses the tools needs snowflakes whether they are using it or not at the moment.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's probably not too heavy a dependency? If so we'll need to fix it but it makes a huge difference to installability. As it is you can just pipx install it, but I can't find any way of pipx installing any optional dependencies.
If we have a nice way to do it I'm happy to split these things out.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove the optional, and keep snowflake-sqlalchemy/snowflake-connector-python as required dependencies. Making them truly optional would need a guarded-import fix (try/except ImportError) in places that import the snowflake modules.

@yhong123

Copy link
Copy Markdown
Collaborator

When running poetry run datafaker make-tables --force --views-to-tables, tables: in orm.yaml are populated.

Comment thread datafaker/dialects.py Outdated
element: snowflake_types.TIMESTAMP_LTZ, compiler: Any, **kw: Any
) -> str:
"""Render TIMESTAMP_LTZ as TIMESTAMP_LTZ."""
return compiler.process(element, **kw)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

compile_timestampltz_ddb is registered via @compiles(snowflake_types.TIMESTAMP_LTZ, "snowflake"), and inside it calls compiler.process(element, **kw) on the same element. Since this function is the dispatch target for (TIMESTAMP_LTZ, "snowflake"), that call re-enters itself — infinite recursion. Any DDL/query involving a TIMESTAMP_LTZ column against a Snowflake destination will raise RecursionError, for example when running the make-tables command.

Shall we call the type compiler’s native visit_TIMESTAMP_LTZ method directly, bypassing @compiles dispatch?

Suggested change
return compiler.process(element, **kw)
return compiler.visit_TIMESTAMP_LTZ(element, **kw)

Comment thread datafaker/dialects.py
_element: snowflake_types.TIMESTAMP_LTZ, compiler: Any, **kw: Any
) -> str:
"""Render TIMESTAMP_LTZ as DateTime."""
return compiler.process(DateTime(timezone=False), **kw)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The generic fallback renders TIMESTAMP_LTZ as DateTime(timezone=False). Since TIMESTAMP_LTZ is timezone-aware by definition, should this be DateTime(timezone=True) instead?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A really tricky one. This is Local TimeZone; how on earth do we model that? This is what I thought was the least bad option, but I'm totally open to different ideas.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we set all timezone to UTC for example? So everything is converted to UTC

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tim-band , what do you think of Nik's suggestion? Or should we just stick with your existing implementation?

Comment thread datafaker/db_utils.py
Comment on lines +187 to +191
case _:
tables = []
logger.error(
"--parquet-dir cannot be used with %s, only duckdb.", dialect_name
)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When --parquet-dir is passed with a non-duckdb dialect, this logs an error but continues normally (tables = []), unlike the DSN-malformed path above which does raise Exit(1). Should this fail hard instead, so it doesn’t look like a silent success in CI/batch runs?

Comment thread datafaker/proposers/intervals.py Outdated


def _not_copying_incompatible_fk(column: Column, fk_col: RelatedColumn) -> bool:
"""Return True if ``column`` and ``fk_col`` are incompatible foreign key types."""

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docstring says this returns True when the columns are incompatible, but the call site (line 587) keeps entries where it returns True, and the logic matches compatible pairs. Docstring looks inverted from the actual/intended behavior.

Comment on lines 331 to 333
src_table = column.table
dialect = engine.dialect
col = literal_column(f'"{column_name}"')
src_table = column.table # preserves schema for schema-qualified databases

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

src_table = column.table is assigned twice in a row (331 and 333)

@yhong123

yhong123 commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

Found one blocking bug: infinite recursion compiling TIMESTAMP_LTZ against the Snowflake dialect, it crashes on the PR’s own core use case, for example when running the make-tables command. Please see the review above for the suggested fix.

Tim Band and others added 6 commits November 4, 2026 12:36

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Snowflake timestamps cannot be written to destination

3 participants