Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -165,13 +165,13 @@ train_interactive.py
tmp_fig.py
log.txt

# Private MCH subpackage (versioned in a separate private repo)
# Private out-of-tree ingestion subpackage (versioned in a separate repo)
raddb/mch/

pyproject.toml

# Local AI-assistant working notes (machine-local paths — not for the public repo)
CLAUDE.md

# Local-only MeteoSwiss demo notebook
# Local-only network-specific demo notebook (not part of the public tutorials)
tutorial/06_mch_pipeline.ipynb
44 changes: 22 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,10 @@
| Citation | [![DOI](XXX)](XXX) | [**Documentation**](https://raddb.readthedocs.io/en/latest/)

RadDB archives xarray **DataTree** radar volumes as compact Parquet files and
gives you a small, fluent interface to load, filter, crop, extract cross-section and
gives you a small, fluent interface to load, filter, crop, extract cross-sections and
plot them. It is **network-agnostic**: any DataTree with the standard
[xradar](https://docs.openradarscience.org/projects/xradar/) coordinate layout
(NEXRAD, ODIM, IRIS, …) can be archived and analysed.
(ODIM, IRIS, …) can be archived and analyzed.

## Storage model

Expand Down Expand Up @@ -56,7 +56,7 @@ Core runtime dependencies: `numpy, pandas, polars, geopandas, shapely, pyproj, x
```python
import raddb

db = raddb.RadDB(archive_dir="/data/raddb", crs=32614)
db = raddb.RadDB(archive_dir="/data/raddb", crs=3067)
```

A **projected CRS is mandatory to write** an archive and never needed to read one.
Expand All @@ -67,27 +67,27 @@ There is no default: the wrong projection is silently wrong.

```python
# From saved DataTree files on disk (.zarr / .nc); the LUT is auto-generated:
db.archive(datatree_dir="/data/NEXRAD_datatree") # radar inferred per file
db.archive(datatree_dir="/data/FMI_datatree") # radar inferred per file

# ...or archive in-memory DataTrees directly:
db.archive(datatree=dt, radar="KTLX")
db.archive(datatree=[dt1, dt2], radar="KTLX")
db.archive(datatree={"KTLX": [dt1], "KMLB": [dt2]}) # multi-radar
db.archive(datatree=dt, radar="FANJ")
db.archive(datatree=[dt1, dt2], radar="FANJ")
db.archive(datatree={"FANJ": [dt1], "FKOR": [dt2]}) # multi-radar
```

Pass `filter=` to decide which gates ever reach the disk — the main control on
archive size:

```python
db.archive(
datatree=dt, radar="KTLX", filter={"var": "DBZH", "logic": ">", "threshold": 20}
datatree=dt, radar="FANJ", filter={"var": "DBZH", "logic": ">", "threshold": 20}
)
```

### Open

```python
rdf = db.open(time_period=("2024-06-12", "2024-06-13"), radars="KTLX")
rdf = db.open(time_period=("2024-06-17", "2024-06-18"), radars="FANJ")
print(rdf) # summary: gates, radars, time range, columns

len(rdf), rdf.columns(), rdf.radars()
Expand All @@ -98,7 +98,7 @@ rdf.crs(), rdf.geographic_crs()
```

`columns=` and `filters=` are pushed down into the scan, so only the rows you
asked for are ever materialised.
asked for are ever materialized.

### Filter and convert

Expand All @@ -117,22 +117,22 @@ dt = rdf.to_datatree() # back to xarray
```

Filters are `{"var", "logic", "threshold"}` dicts, where `logic` is one of
`==`, `!=`, `>`, `>=`, `<`, `<=`. `crs` is an EPSG int (e.g. `32614`), a
`==`, `!=`, `>`, `>=`, `<`, `<=`. `crs` is an EPSG int (e.g. `3067`), a
CRS object, or `None`.

### Crop to an area of interest

```python
box = rdf.crop_by_bbox(extent=[636_504, 676_504, 3_891_333, 3_931_333])
poly = rdf.crop_by_polygone("catchment.geojson")
disc = rdf.crop_around_point((656_504, 3_911_333), distance=20_000) # metres
box = rdf.crop_by_bbox(extent=[485_859, 525_859, 6_732_099, 6_772_099])
poly = rdf.crop_by_polygon("catchment.geojson")
disc = rdf.crop_around_point((505_859, 6_752_099), distance=20_000) # meters
# rdf.interactive_crop() # draw an AOI on a Jupyter map
```

### Cut a cross-section

```python
cs = rdf.extract_cross_section(p1=(626_504, 3_911_333), p2=(686_504, 3_911_333))
cs = rdf.extract_cross_section(p1=(475_859, 6_752_099), p2=(535_859, 6_752_099))
```

### Plot
Expand All @@ -141,7 +141,7 @@ cs = rdf.extract_cross_section(p1=(626_504, 3_911_333), p2=(686_504, 3_911_333))
rdf.plot_ppi(sweep=1, variable="DBZH", save="ppi.png")
rdf.plot_rhi(azimuth=270, variable="DBZH")
rdf.plot_cappi(altitude=3000, variable="DBZH")
cs.plot_cross_section(variable="DBZH", save="xsec.png")
cs.plot_vcs(variable="DBZH", save="xsec.png")
```

Each plot draws into one `Axes` and returns the matplotlib artist, so you compose
Expand All @@ -160,16 +160,16 @@ rdf.filter({"var": "DBZH", "logic": ">", "threshold": 20}).crop_by_bbox(
```python
db.inventory() # radars, volume counts, time ranges, size
db.inventory(detailed=True) # + LUT info, stored variables, day-by-day counts
db.inventory(datatree_dir="/data/NEXRAD_datatree") # DataTree files not archived yet
db.inventory(datatree_dir="/data/FMI_datatree") # DataTree files not archived yet
```

### LUT accessors (archive-bound)

```python
db.list_radars() # radars present in the archive
db.get_lut("KTLX") # the static LUT (polars)
db.get_radar_info("KTLX") # site location / sweep geometry
db.add_lut_projection("KTLX", epsg=32614)
db.get_lut("FANJ") # the static LUT (polars)
db.get_radar_info("FANJ") # site location / sweep geometry
db.add_lut_projection("FANJ", epsg=3067)
```

## Module structure
Expand All @@ -182,14 +182,14 @@ raddb/
├── lut.py # LUT generation / geo projection
├── aoi.py # AOI / crop / cross-section geometry
├── discovery.py # find_datatree_files + filename-time parsing
├── helper.py # filters, radar-name normalisation, timers
├── helper.py # filters, radar-name normalization, timers
└── viz/ # plot.py (PPI/RHI/CAPPI/cross-section), interactive.py
```

## Notes

- **Projected coordinates / `crs`.** Generating a LUT with a projection (e.g.
`crs=32614`) and the projected accessors (`extent`, `to_geopandas`) use `pyproj`,
`crs=3067`) and the projected accessors (`extent`, `to_geopandas`) use `pyproj`,
which needs the PROJ database. A `PROJ_DATA` / `PROJ_LIB` inherited from another
environment (a conda base env, a system PROJ) points at a proj.db of the wrong
PROJ version and makes every projection fail with *"no database context
Expand Down
2 changes: 1 addition & 1 deletion docs/source/03_quickstart.rst
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ Every operation returns a **new** ``RadDB``, so calls chain:
.crop_around_point(point=(-97.278, 35.333), distance=25_000, crs=4326)
)

Alongside ``crop_around_point`` there are ``crop_by_bbox``, ``crop_by_polygone``
Alongside ``crop_around_point`` there are ``crop_by_bbox``, ``crop_by_polygon``
(shapely geometry, GeoDataFrame or a ``.shp`` / ``.geojson`` path) and
``extract_cross_section`` for a vertical slice along an arbitrary line.

Expand Down
5 changes: 3 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,9 @@ dynamic = ["version"]

[tool.setuptools.packages.find]
include = ["raddb*"]
# raddb.mch is the private MCH subpackage (gitignored, separate repo) —
# excluded here as defense in depth for wheels built from a full checkout.
# raddb.mch is a private, out-of-tree ingestion subpackage (gitignored, kept in
# its own repo) — excluded here as defense in depth for wheels built from a
# checkout that has it present locally.
exclude = ["raddb.mch*", "raddb.tests*"]

[tool.setuptools_scm]
Expand Down
8 changes: 3 additions & 5 deletions raddb/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@

RadDB archives xarray DataTree volumes as Parquet files with an efficient
LUT-based layout. It is network-agnostic: any DataTree with the standard
xradar coordinate layout can be archived and reconstructed.
xradar coordinate layout can be archived and reconstructed (FMI, NEXRAD, ...).

MCH/METRANET-specific ingestion code lives in the private ``raddb.mch``
subpackage (gitignored in the public repository; never imported here).
Network-specific ingestion code — readers for a national archive's own raw
format — belongs in a separate package and is never imported here.
"""

from __future__ import annotations
Expand Down Expand Up @@ -101,7 +101,6 @@
from raddb.viz.plot import (
plot_cappi,
plot_cross_section,
plot_latent_scatter,
plot_ppi,
plot_rhi,
plot_vcs,
Expand Down Expand Up @@ -182,7 +181,6 @@
"plot_cappi",
"plot_vcs",
"plot_cross_section",
"plot_latent_scatter",
]

_root_path = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
Expand Down
30 changes: 15 additions & 15 deletions raddb/aoi.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@

The centroid tables are **polars** frames (the package-wide default); the spatial
predicate itself runs on plain numpy arrays via shapely, so no geometry objects
are ever materialised for the ~1.7M gates of a full LUT.
are ever materialized for the ~1.7M gates of a full LUT.
"""

from __future__ import annotations
Expand Down Expand Up @@ -133,7 +133,7 @@ def _lut_centroids(base_path: str | Path, radars: list[str], epsg=None) -> pl.Da
base_path : str or Path
RadDB archive base directory.
radars : list of str
Radar letters whose LUTs to load (e.g. ``["L", "P"]``).
Radar names whose LUTs to load (e.g. ``["L", "P"]``).

Returns
-------
Expand Down Expand Up @@ -201,8 +201,8 @@ def _load_one_centroid_table(base: Path, radar: str, epsg: int) -> pl.DataFrame:
# than EPSG lookups) keeps reprojection working even where the PROJ database is
# unavailable — the same tactic raddb.lut.add_lut_projection uses for WGS-84.
# The LV95 string is the standard 7-parameter CH1903+ definition; it agrees with
# the grid-based EPSG:2056 transform to well under a metre, negligible at the
# kilometre scale of AOI gate selection.
# the grid-based EPSG:2056 transform to well under a meter, negligible at the
# kilometer scale of AOI gate selection.
_PROJ4 = {
4326: "+proj=longlat +datum=WGS84 +no_defs",
2056: (
Expand Down Expand Up @@ -250,7 +250,7 @@ def _reproject_to_aoi(geom, crs: int | str | None, aoi_epsg: int):
"""
if crs is None:
return geom
# Normalise common spellings of "already the AOI CRS" → no pyproj needed.
# Normalize common spellings of "already the AOI CRS" → no pyproj needed.
if crs in (aoi_epsg, str(aoi_epsg), f"EPSG:{aoi_epsg}", f"epsg:{aoi_epsg}"):
return geom

Expand Down Expand Up @@ -500,7 +500,7 @@ def _read_geometry_file(path: Path):

``.shp`` via pyshp, ``.geojson`` via json. Polygons define an AOI to crop
with; lines define a vertical cross-section. The returned CRS is what the
file declares — callers must honour it rather than assuming LV95.
file declares — callers must honor it rather than assuming LV95.
"""
if not path.exists():
raise FileNotFoundError(f"Geometry file not found: {path}")
Expand Down Expand Up @@ -541,7 +541,7 @@ def _read_geojson(path: Path):
geoms = [shape(data)]
else:
raise ValueError(
f"Unrecognised GeoJSON object type {kind!r}; expected a "
f"Unrecognized GeoJSON object type {kind!r}; expected a "
"FeatureCollection, a Feature, or a bare geometry.",
)
if not geoms:
Expand Down Expand Up @@ -608,7 +608,7 @@ def _prj_crs(prj_path: Path):
# (distance-along-line, altitude) plane
# <- radDB_spatial_plot.get_rad_gdb_vert_cross_section
# Adapted to the current data model (projected x/y + altitude from the LUT) and
# fully vectorised (numpy point-to-segment prefilter replaces the KDTree).
# fully vectorized (numpy point-to-segment prefilter replaces the KDTree).
# ============================================================================

_CS_LUT_BASE_COLS = [
Expand All @@ -633,7 +633,7 @@ def _lut_cs_table(

Half-dimensions (prototype convention):
- ``dR``: half the radial gate spacing, derived per sweep from the LUT's
range grid (500 m -> 250 m for MCH radars);
range grid (e.g. 500 m -> 250 m at 500 m range sampling);
- ``dA`` (= dE): half the across-beam extent, ``range * tan(beamwidth/2)``
— grows with range.
"""
Expand All @@ -651,7 +651,7 @@ def _lut_cs_table(
if t is None:
available = set(pq.read_schema(lut_path).names)
xc, yc = f"x_{int(epsg)}", f"y_{int(epsg)}"
# The LUT stores two different x/y: metres from the radar, and the
# The LUT stores two different x/y: meters from the radar, and the
# projected pair. The section geometry works in the projected one,
# which is why it takes the plain `x`/`y` names here; the
# radar-relative pair rides along as x_rel/y_rel and is renamed back
Expand Down Expand Up @@ -802,7 +802,7 @@ def _beam_profile(base_path, sub: pd.DataFrame, epsg: int):
"""Per-gate beam profile from the ``v_plane`` lattice, or ``None``.

Returns ``(d_near, d_far, z_near, z_far, half_thickness)`` — the gate's
beam-centre altitude at its near and far range edge, as a function of ground
beam-center altitude at its near and far range edge, as a function of ground
distance, plus half its vertical extent there. This is the curved-beam
geometry the plots draw, replacing the flat ``u * tan(el)`` climb.
"""
Expand Down Expand Up @@ -837,7 +837,7 @@ def _endpoint_d_z(pt_xy: np.ndarray, sub: pd.DataFrame, origin: tuple[float, flo
rx = pt_xy[:, 0] - sub["x"].to_numpy(dtype=np.float64)
ry = pt_xy[:, 1] - sub["y"].to_numpy(dtype=np.float64)
az = np.deg2rad(sub["azimuth"].to_numpy(dtype=np.float64))
# Along-beam offset of the endpoint from the gate centre.
# Along-beam offset of the endpoint from the gate center.
u = rx * np.sin(az) + ry * np.cos(az)
d_center = 0.5 * (d_near + d_far)
span = d_far - d_near
Expand Down Expand Up @@ -874,7 +874,7 @@ def _cross_section_gates(
to polars when joining them onto the data frame.

Returns one row per crossed gate with its cross-section geometry: the gate
centre ``(d_center, z_center)`` and ``cs_polygon`` — the 4-corner shapely polygon
center ``(d_center, z_center)`` and ``cs_polygon`` — the 4-corner shapely polygon
in the (distance-along-line [m], altitude [m ASL]) plane, built by
offsetting the chord perpendicularly by ±dA (the vertical half-beamwidth
extent). ``d`` is measured from ``p1``.
Expand All @@ -889,7 +889,7 @@ def _cross_section_gates(
raise ValueError("cross-section line is degenerate (< 1 m long).")
half_bw_tan = float(np.tan(np.deg2rad(beamwidth_deg / 2.0)))

# --- vectorised point-to-segment prefilter (replaces the KDTree) ---
# --- vectorized point-to-segment prefilter (replaces the KDTree) ---
px = cs_t["x"].to_numpy(dtype=np.float64)
py = cs_t["y"].to_numpy(dtype=np.float64)
diag = np.hypot(cs_t["dR"].to_numpy(dtype=np.float64), cs_t["dA"].to_numpy(dtype=np.float64)) * 1.05
Expand Down Expand Up @@ -946,7 +946,7 @@ def _cross_section_gates(
)

out = sub.copy()
# Only the gate centre and its footprint are published. The chord endpoints
# Only the gate center and its footprint are published. The chord endpoints
# d_near/d_far and z_near/z_far are what the polygon is built from, so
# emitting them as well restated `cs_polygon` in scalar form.
out["d_center"] = 0.5 * (d_near + d_far)
Expand Down
45 changes: 3 additions & 42 deletions raddb/discovery.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,59 +7,20 @@
- **Archived POL search** (output side): locate ``*_POL.parquet`` files in a
time range inside an existing RadDB archive.

Everything here is pure filesystem + pandas — no pyart / radar_api
dependency — so discovery works in any environment. (Raw METRANET
scanning lives in the private ``raddb.mch.discovery`` module.)
Everything here is pure filesystem + pandas — no pyart dependency — so
discovery works in any environment. Scanning a national archive's own raw
format belongs in the package that reads it, not here.
"""

from __future__ import annotations

import datetime
import re
from collections import defaultdict
from pathlib import Path

import pandas as pd

from raddb.helper import ensure_utc

# ============================================================================
# METRANET filename helpers (shared with raddb.mch)
# ============================================================================


def _parse_volume_time(stem: str) -> datetime.datetime:
"""Parse the timestamp from a METRANET filename stem.

Works for any ``XXXYYJJJHHMM...`` stem (3-char prefix + 2-digit year +
day-of-year + hour + minute), e.g. ``MLA2419423300U`` or
``HZT2124010000L``. Returns 1970-01-01 when the stem cannot be parsed.
"""
try:
y, j, h, m = (
int(stem[3:5]),
int(stem[5:8]),
int(stem[8:10]),
int(stem[10:12]),
)
return datetime.datetime(2000 + y, 1, 1) + datetime.timedelta(
days=j - 1,
hours=h,
minutes=m,
)
except Exception:
return datetime.datetime(1970, 1, 1)


def _group_files_by_volume(paths: list[str]) -> dict:
"""Group sweep files by volume (based on filename stem)."""
vols = defaultdict(list)
for p in paths:
stem = Path(p).stem
vols[stem].append(p)
return dict(vols)


# ============================================================================
# DataTree file discovery (input side)
# ============================================================================
Expand Down
Loading
Loading