[richling-ucar]

Native-grid and CLM land diagnostics, merged with main

This PR brings native-grid (ne30pg3) support and the CLM land diagnostics (from ww-clm-diags) onto current main, with the bug fixes found by a 64-run regression suite on Casper. The full results, plots and per-file comparisons are in the LDF Merge Test Log.

What this PR does

• Native grids. Native ne30pg3 history becomes native time series and climatologies. They are then gridded to the baseline's lat/lon grid with xESMF and a weights file (bilinear by default, conservative on request), or plotted on the mesh with UXarray (unstructured_plotting). Native and lat/lon cases, and native cases and observations, can be compared either way: in UXarray mode the lat/lon side is remapped onto the native side's mesh (ESMF conservative weights, uxarray bilinear fallback).
• CLM land diagnostics. Land runs are recognised from clm2 in hist_str. Area × landfrac weighting, separate time-series and climatology years, regional climatologies and time series from regions_lnd.yaml, the LMWG table and website.
• Pipeline entry points. Each case can start from history, time series or climatology. cam_ts_done is now calc_cam_ts (old key still accepted). Existing time series are reused per variable.
• Plot whatever data there is. When one case lacks a file, plots show the cases that have it instead of being skipped.
• Meridional means over 5°S–5°N. meridional_mean now averages over the latbounds band (default 5°S–5°N), which main sets but ignores, averaging over all latitudes. All meridional-mean plots change.
• LENS2 per variable. The CESM2 LENS2 range behind the global-mean time series comes from each variable's lens2_file in the variable defaults, found in the config's lens2_data_loc (as obs_file and obs_data_loc), instead of a hard-coded directory.
• Climatology years in file names and metadata. Climo and regridded files are named after their climatology years (…_climo_0011-0013.nc), so a run over other years never reuses a file made for the old ones. Time series files record ts_yrs, climo and regridded files climo_yrs. Older names are still read.
• main eba1f07 merged in, keeping main's plotting features (NCL colormaps, per-plot settings, colormap_2d, surface wind stress).
• Environment adf_v1.0.5: uxarray 2025.6.0 (needed for native maps), plus pylint 4.1.3, astroid 4.3.4 and pytest 9.1.1 as developer tools. Every other pin is unchanged.

Testing

Check	Result	Notes
Suite 1 · pipeline entry points (S00–S10)	11 of 11	No unexpected differences from main. Only the meridional band and the new one-side-only time-series plots differ.
Suite 2 · native grid, remapping, UXarray (N01–N11)	12 of 12	Gridded native climatologies match ncremap to about 1e-7; AMWG tables identical (N01–N03).
Suite 3 · land (L01–L14)	14 of 14	f09, gridded native and UXarray native; native table means within 0.9% of f09.
Suite 4 · against observations (S00b, S11–S13, N12, N13, with N08, N09a)	8 of 8	Every value identical to main in obs mode (S11–S13 against S00b); only the meridional band and the one-sided global-mean plots differ. Native ts and climo entries match N08 to 1.8e-6. Atmosphere only: no land observations to test against.
Suite 5 · every native scenario gridded and on the mesh (N14–N24, L15–L23)	20 of 20	Each native pairing, from every entry point and in both directions, plotted both ways. Against the run covering the same grids from another entry point, every shared plot is pixel-identical and every regridded value identical. Mirrors agree (N09b/N14 TS mean difference 0.57/0.56).
Meridional means on the mesh (N04, N09a, N09b, L07)	4 of 4	Native vs native matches the gridded runs; native vs obs or f09 plots the test case alone; land on the mesh plots both cases over 5°S–5°N.
Land reference: L01 against the land scientists' code (L00, ww-clm-diags c6db27e)	matches	Time series, climatologies, land-table numbers and every plotted curve identical. Differences are ours by design: coastal cells kept, plain-text units, colour-bar arrows, case legend. The land branch crashes in polar maps and needs a mesh for f09 regional plots.
Extended run (years 1–3, then 1–5, same folders)	pass	Only the new years' time series made; new year-tagged climo and regridded files; climatology and table equal the 1–5 run's.
Unit tests (lib/test/unit_tests)	349 passed	In the adf_v1.0.5 env, on f736c77. With CI's packages only (PyYAML, pytest): 97 pass, 152 skip.
CI lint script (pylint, every modified lib/ file)	22 of 22	Threshold 9.5, in a venv with CI's packages only.
black 25.11.0 on lib/	clean	49 files unchanged (CI's pinned black).

The latest clean rerun (all 38 runs, on 1a9fbb7) started from wiped output folders, in an env built from env/conda_environment.yaml. S00 ran on main in its own env; S00 and S01 agree in every value, so the env change moves no result.

Bugs fixed

#	Bug	Fix	On main?
1	Gridded native time series read five years as about 75 days (time re-encoded in hours, bounds left in days).	grid_to_latlon, three save_to_nc, use_time_bounds_midpoint	partly
2	Every UXarray map failed: uxarray 2025.03.0 has no to_raster.	uxarray 2025.6.0	env
3	Native test vs non-native reference crashed the map, vector and polar scripts.	warn-and-skip checks (now remapped onto the mesh instead)	no
4	The same case crashed zonal_mean.	missing-reference checks	no
5	Polar maps crashed without plot_press_levels.	defaults to an empty list	yes
6	Global-mean time series crashed when the LENS2 file could not be read.	LENS2 left out with a warning	yes
7	Regional plots failed on native lndgrid climo.	_region_native	no
8	UXarray runs with no regrid step made no maps.	AdfData._native_climo_fallback	no
9	Regional plots used the baseline's mesh for both cases.	each case on its own grid	no
10	A plot was dropped when one case lacked data.	plot the cases that have it	behaviour
11	get_regrid_file raised for variables with no reference.	returns an empty list	no
12	Native maps crashed on all-zero fields.	“No Valid Data Points” panel	no
13	Regional legends hid the curves.	one shared case legend	no
14	main's create_clean_grid fails on bounds with a time dimension.	fixed here; also needs a PR to main	main only
15	Single-case line plots: the title overlapped the case name, and the note said “no baseline data” for a baseline on another grid.	constrained layout; “no comparable baseline data”	no
16	Derived ASA (albedo, %) kept the W/m² units of FSR, so plots labelled it “W/m^2”.	a formula that changes units sets them (“%”, “permil”)	partly
17	Polar maps on the mesh drew missing cells as 0% and printed a nan mean.	missing cells left blank; NaN-safe means	no
18	Polar maps against observations crashed the run when an obs variable is on its own dimensions (land MODIS albedo on lsmlat/lsmlon).	skip with a warning, as the lat/lon map does	no
19	Maps on the mesh against a remapped lat/lon side crashed: cell_weights dropped the uxarray grid with isel(time=0).	grid taken first; reference carries the test case's area/landfrac	no
20	Obs on the mesh: the copied area kept the model's time labels and aligned to NaN against a one-step obs file.	area, landfrac written time-independent	no
21	Vector maps: an early return skipped the remaining pairs; crashes on the mesh without pressure levels or plot_press_levels.	restructured like global_latlon_map; 30 plots byte-identical	partly
22	Taylor diagram against a lat/lon side on the mesh: out of memory (native test) or KeyError: 'lat' (lat/lon test vs native reference).	reference loaded per case on the right grid	no
23	Native land maps drew the ocean as 0 in the test, baseline and difference panels (only the % panel left it blank).	prep_ux_for_plot no longer fills NaN with 0	no

Two more were in the test setup (a config that read its own output, stale files between runs). Details for each are in the test log.

Release notes

• cam_ts_done is now calc_cam_ts, with the opposite meaning, matching calc_cam_climo: calc_cam_ts: true makes time series from the history files, as cam_ts_done: false did. Replace cam_ts_done: X with calc_cam_ts: not X. The old key still works and prints a deprecation note; giving both with contradicting values stops the run.
• Existing time series are reused per variable. With calc_cam_ts: true, files already in cam_ts_loc are kept and only the years they do not cover are made, each stretch in a file of its own.
• Runs on pre-made climatologies. With calc_cam_ts: false and calc_cam_climo: false a case needs no time series; scripts that need them leave that case out.
• New optional config keys, documented in config_cam_baseline_example.yaml. In diag_basic_info: unstructured_plotting, regions_file, lens2_data_loc. Per case: climo_start_year/climo_end_year (with ts_start_year/ts_end_year as other names for start_year/end_year), and for native-grid cases mesh_file, weights_file, regrid_method and, for CLM, latlon_file. At the top level: region_list and regional_var_list.
• LENS2 needs lens2_data_loc. A config that runs global_mean_timeseries and wants the LENS2 range must set lens2_data_loc (on Derecho/Casper: /glade/campaign/cgd/cas/islas/CESM_DATA/LENS2/global_means/annualmeans). New lens2_file entries cover ACTREL, AODVIS, CDNUMC, CLDHGH, CLDLOW, CLDMED, CLDTOT, FSDS, FSDSC, FSNT, FSNTC, SWCF, TGCLDIWP, TGCLDLWP and TS. The aerosol burdens (BURDEN*dn) have LENS2 files but no variable-defaults entry, so they no longer get the range.
• AMWG tables use the time-series years, as the land tables already did. With climo_start_year/climo_end_year narrower than the time series, the table still covers all the time-series years; the climatology years no longer steer its file choice or its one-year check.
• Climo and regridded file names carry the climatology years ({stem}_YYYY-YYYY.nc). Existing files with the older names are still read, but are never overwritten or reused by a run over other years: that run makes new year-tagged files. Time series files now record ts_yrs.
• Plots that change for main users: meridional means (5°S–5°N band), and single-case time-series, QBO, tape-recorder, zonal and meridional plots for runs where one case has no data, which used to be skipped.
• Environment adf_v1.0.5: rebuild with conda env create -f env/conda_environment.yaml. uxarray 2025.6.0 is required for native-grid maps.

Before this PR is ready

• Commit this round's work. The fixes above, the new unit tests and the env change are in five commits, 8e60e34 to c1e6bd2, pushed to the branch. The two root-level CLM configs below were left out.
• Commit fix 15. The single-case plot fix in lib/plotting_functions.py: committed and pushed as c711eb7.
• Commit the year-tagged file names and fixes 16 and 17, with their unit tests: pushed as f45d03d, b2a660d and e8a7c65.
• Rerun the full suite (now 38 runs with L00) on the year-tagged names: done on e8a7c65, all 38 pass.
• Skip regridding when the grids are identical. f09 to f09 conservative regridding changes the southernmost row (288 cells, up to 0.04% of the field maximum).
• Fix two unit strings in ldf_variable_defaults.yaml: ELAI m${-2}$ m${-2}$ and QBOT kg kg{-1}$ come out garbled in tables: fixed in 1a9fbb7.
• Decide on the root-level CLM configs. Kept, and updated to the new config keys (start_year/end_year, calc_cam_ts) in eb0ccfb.
• Document the new config keys in config_cam_baseline_example.yaml. Added, commented out: unstructured_plotting, regions_file, the per-case climo_start_year/climo_end_year, mesh_file, weights_file, regrid_method and latlon_file, and region_list/regional_var_list; lens2_data_loc is set.
• Sign-off on the uxarray pin. Moving to 2025.6.0 changes main's environment for every user.
• CI does not run the science tests. CI installs only PyYAML and pytest, so 141 tests skip, including every native-grid and land test, and scripts/ is never linted. Add a CI job with the science stack, or state that HPC runs are the test.
• New plots and website entries on main. Runs that used to skip time-series, zonal and meridional plots now make single-case versions.
• Size. 69 files and about 22,750 added lines. Consider separate PRs for the fixes that also affect main (1, 5, 6), native grid, land, and the plot-whatever-there-is rule.
• Open the small PR to main for fix 14 (create_clean_grid with time-dimensioned bounds).
• Rerun Suite 1 if main moves. main was still at eba1f07 on 2026-10-21.
• Hard-coded LENS2 path. Replaced by a per-variable lens2_file in the variable defaults, found in the config's lens2_data_loc. Land variables get none (the old search matched atmosphere files for them).
• Known bug, not fixed here: lmwg_table and main's amwg_table divide by N instead of √N for the standard error, so the standard error and 95% columns are too small.
• Multiple test cases. The land regional, global-mean and table scripts use only the first test case.
• GenTS back end does not reuse existing time series per variable, as the ADF back end now does.
• Gridded file metadata. Gridded files lack lat_bnds/lon_bnds and store area in km² where the f09 files use steradians.
• Release notes for cam_ts_done → calc_cam_ts, the new env, and the new config keys: written (section above), ready for the GitHub release.
• Update the wiki and LAND-DIAGS_README.md with the release-note changes.
• Choice of remap onto the mesh. In UXarray mode the lat/lon side is now remapped onto the native mesh (fdb06e7, 4868272; Suite 5), but the method is fixed in the code: ESMF conservative weights (made once per grid pair, cached in regrid_weights/), uxarray bilinear only when ESMF cannot run, and inverse-distance weighting if bilinear fails (GPCP 2.5°). Gridded mode lets each case choose its weights_file and regrid_method; mesh mode has no setting. Conservative keeps area means (fluxes, precipitation, global and regional means) and is the right default; bilinear is smoother (winds, temperature) and matches the bilinear gridded runs. Add a setting such as mesh_regrid_method: conserve | bilinear, one per run or per case, or keep conservative only?
• Land masking in the remap onto the mesh. A land field remapped from f09 onto the mesh averages the f09 cells each mesh cell overlaps, leaving out the missing ocean cells and renormalising the rest; neither grid's land fraction is used. Along coasts a mesh cell can take full weight from a sliver of f09 land, a mesh land cell whose f09 neighbours are all ocean comes out empty, and f09 land can land in cells the model counts as mostly ocean. Inland values and the averages (the on-mesh file carries the test case's area and landfrac, so both sides are weighted over the same cells) are unaffected. Weight the source by its land fraction and mask the result with the mesh's, as the gridded path does with latlon_file? Same coastal issue as the land question on coastlines.
• Misleading mesh warning. Gone: with the reference remapped onto the mesh, “declare mesh_file” is no longer printed (N09a, N19).
• Ocean on native land maps. Native (UXarray) lat/lon maps of land fields showed the ocean as 0 in the test and baseline panels and as “no change” in the difference, because prep_ux_for_plot filled missing cells with 0. Fixed (bug 23): missing cells stay blank in all four panels; checked on L07's maps, means and RMSE unchanged (92b1105).
• Reproducibility for reviewers. The test configs and data live on glade, outside the repo. Add the configs (or a short how-to) somewhere reviewers can see them.

Science questions · native grid

• Global means. UXarray runs weight by native cell area; gridded runs use cos(latitude) on the gridded field. Tables from native time series differ from f09 by at most 0.0035 in the mean. Which number goes in tables, and should a native run always compute its table on the native grid?
• Polar rows. Should winds in the rows next to the poles be trusted after bilinear remapping (up to 1.9 m/s difference in V between remaps), or masked in polar plots?
• Native zonal and meridional means. Are 2° bins right for ne30pg3 (about 1°), and should finer meshes get finer bins?
• Native vs observations. In UXarray mode obs are now remapped onto the mesh (N09a, N19, N22). Is that wanted, or should native-vs-obs always be gridded (N08, N12, N13)?
• Mixed pairs in UXarray mode. The lat/lon case is now remapped onto the mesh automatically, in either direction (Suite 5). Is that right as the default? The comparison is at native resolution, so plot-header numbers differ from the gridded runs'.
• Meridional band. The branch averages meridional means over 5°S–5°N; main ignores its own latbounds and averages globally. Which is wanted?
• TAUX/TAUY sign. scale_factor: -1 is never applied because new_unit makes the file look converted, so maps show CAM's sign. Which sign is wanted?
• Gridded file metadata. Does any downstream user need lat_bnds/lon_bnds or area in steradians in gridded files?

Science questions · land

• Land masking when gridding. The ADF masks with the source and target land masks; ncremap -P clm weights by the sub-grid mask. Table means differ by at most 0.88% (SOILWATER_10CM). Which should the ADF match?
• Regions across grids. A cell counts as inside a region when its centre is in the box, so two grids take in different land at the edges. TOTECOSYSC over CONUS is 1.72% higher on the native grid than on f09 for the same case, while global means are identical. Across grids (L14) that offset is about 17 times the real case difference. Grid both cases first, weight cells by overlap, use mask regions, or note it on the plots?
• Antarctica. ELAI, GPP, ET and QRUNOFF_TO_COUPLER are all missing there, so their panels are skipped. Show them as zero instead?
• Standard error. lmwg_table uses std/N instead of std/√N. Fix it here or with main's amwg_table?
• ENERGY_BALANCE land fraction. The derived ENERGY_BALANCE scales by a hard-coded cpl_landfrac = 0.29251054, which only fits one grid. Should it come from the case's own land fraction instead?
• Carbon isotope formulas. C13/C14_TOTSOMC_pm and C13/C14_TOTLITC_pm used to get the plain sum of their constituents; the merge computes them with the per-mil isotope formula from the NCL land diagnostics. Is that right, and do the formulas match Jain et al. 1996?
• Coastlines. The land scientists' code drops coastal cells when regridding lat/lon CLM (1,500 of 23,360 on f09); this branch keeps them. Is that the behaviour the LMWG wants, and is a land-aware regrid needed for other grid pairs?

Software questions

• Regrid step for UXarray runs. Both setups now work: with the regrid step (native files passed through) and without it (native climo read directly). Keep both, or require one?
• Implicit gridded/ folders. The ADF reads gridded/ whenever it exists, so pre-made native time series must come from a run that never gridded them. Make it an explicit setting?
• Writing next to inputs. The regrid stage writes gridded/ beside pre-made native climo, inside a folder the user supplied. Should the ADF only write under its own output folders?
• Mesh configuration. Meshes are given per test case and once for the baseline. Is that the shape we want, and what about obs on a mesh?
• Shared atm/land scripts. Keep is_lnd branches, or split into separate land scripts on shared helpers? CAM time-series plots also get no website category, unlike land.

Full test log: https://claude.ai/artifact/31HPszLwAiSNJqV5orBeYy

[richling-ucar]

Closes #354 and #397 as it is now outdated.