Repository navigation
Description
🌀 Claude: Three ways a prepared query gives a wrong answer and doesn't raise. CodeRabbit found these while reviewing #323. I reproduced each one on main at 447791a (0.7.1), with DataFusion 55.0.0, using these helpers:
import numpy as np, xarray as xr
from xarray_sql import Context
def ds(v):
return xr.Dataset({"v": ("i", v)}, coords={"i": np.arange(len(v))})
def multi(x): # two groups of dims, so from_dataset makes a table for each
return xr.Dataset(
{"s": ("i", x), "t": (("i", "j"), np.ones((3, 2)))},
coords={"i": np.arange(3), "j": np.arange(2)},
)1. A rejected registration still replaces the Dataset that writes go to. from_dataset records the Dataset before DataFusion registers its table. So when DataFusion refuses a name that's taken, the record points at a Dataset no table reads, and later writes don't reach the query:
ctx = Context(); ctx.from_dataset("a", ds(np.arange(3.0)))
ctx.from_dataset("a", ds(np.zeros(3))) # raises: the name is taken
q = ctx.prepare("SELECT v FROM a ORDER BY i")
q(a=ds(np.array([7.0, 8, 9])))["v"] # [0.0, 1.0, 2.0]2. Registering a split Dataset again isn't counted. register_schema replaces the old schema silently, so a prepared query of one of its tables keeps reading the old one:
ctx = Context(); ctx.from_dataset("g", multi(np.arange(3.0)), table_names={("i",): "s1"})
q = ctx.prepare("SELECT s FROM g.s1 ORDER BY i")
ctx.from_dataset("g", multi(np.array([7.0, 8, 9])), table_names={("i",): "s1"})
ctx.sql("SELECT s FROM g.s1 ORDER BY i") # [7.0, 8.0, 9.0]
q()["s"] # [0.0, 1.0, 2.0], with no error3. A write of the wrong size writes some of the tables. The checks compare dims by name, not by size. So a dim of another size, with no coordinate on it, fails only during the copy, after earlier tables are written:
ctx = Context()
ctx.from_dataset("a", xr.Dataset({"v": ("i", np.arange(3))}))
ctx.from_dataset("b", xr.Dataset({"v": ("k", np.arange(3))}))
q = ctx.prepare("SELECT a.v + b.v AS s FROM a JOIN b ON a.i = b.k")
q(b=xr.Dataset({"v": ("k", np.arange(10, 13))}), a=xr.Dataset({"v": ("i", np.arange(5))}))
# ValueError: could not broadcast input array from shape (5,) into shape (3,)
ctx.sql("SELECT v FROM b ORDER BY k") # [10, 11, 12]: b was written#323 fixes all three:
from_datasetrecords a Dataset only after its table is registered.- Registering a split Dataset again counts each table its old schema had.
writechecks each dim's size before it writes anything.
Activity
Keep datasets and replacements true to the tables