Bring in Land and Native Grid

LAND-DIAGS_README.md

@@ -0,0 +1,80 @@
+### Here's a quick start that you can use for Land Diagnostics
+#### Download the adf repo
+On casper:
+1. Navigate to the directory where you want the adf (e.g., `cd ~`)
+2. Clone the ADF
+`git clone https://github.com/NCAR/ADF.git`
+3. Set your personal repository as the upstream repo
+```
+cd ADF
+git remote add upstream https://github.com/<git_user_name>/ADF.git
+```
+4. Switch to the clm-diags branch
+`git switch -c clm-diags origin/clm-diags`
+
+#### Set up your computing environment
+5. Create the ADF conda environment, which the land diagnostics use too (it includes the uxarray version, 2025.03.0, they need). On NCAR's CISL machines (derecho and casper), run the following on the command line:
+```
+module load conda
+conda env create -f env/conda_environment.yaml
+conda activate adf_v1.0.3
+```
+(The environment's name is set by the `name:` line at the top of `env/conda_environment.yaml`; add `-n <name>` to `conda env create` to choose another.)
+
+Also, along with these python requirements, the `ncrcat` NetCDF Operator (NCO) is also needed.  On the CISL machines this can be loaded by simply running the following on the command line:
+
+```
+module load nco
+```
+
+## Running ADF diagnostics
+
+Detailed instructions for users and developers are availabe on this repository's [wiki](https://github.com/NCAR/ADF/wiki). 
+
+You'll have to add your username to the appropriate config file, but after that, for a quick try of land diagnostics
+
+`./run_adf_diag config_clm_unstructured_plots.yaml`
+
+This should generate a collection of time series files, climatology (climo) files, re-gridded climo files, and example ADF diagnostic figures, all in their respective directories.
+
+**NOTE:** If you get NCO failures at the generate timeseries stage that end up causing LDF to fail, see issue [#365](https://github.com/NCAR/ADF/issues/365) 
+
+When additional memory is needed sometimes need to run interactive session on casper:
+`execcasper -A P93300041 -l select=1:ncpus=4:mem=64GB`
+
+## TEST for Land Diags:
+
+For this branch there are (3) ways to run the ADF:
+
+1) On native grid with unstructured plotting via Uxarray
+2) On native grid but gridded to lat/lon
+3) On already lat/lon gridded input files (hist, ts, or climo)
+
+For (1), the config yaml file will be essentially the same, but with a couple of additional arguments:
+  - in `diag_basic_info` set the `unstructured_plotting` argument to `true`
+  - in each of the test and baseline section supply a mesh file in the `mesh_file` argument
+
+  Example yaml file: `config_clm_unstructured_plots.yaml`
+
+For (2), the config yaml file will need some additional arguments:
+  - in each of the test and baseline sections, supply the following arguments:
+
+    Weights file:
+
+    `weights_file: /glade/work/wwieder/map_ne30pg3_to_fv0.9x1.25_scripgrids_conserve_nomask_c250108.nc`
+
+    Regridding method:
+
+    `regrid_method: 'coservative'`
+    (Yes, spelled incorectly for a bug in xESMF)
+
+    Lat/lon file:
+
+    `latlon_file: /glade/derecho/scratch/wwieder/ctsm5.3.018_SP_f09_t232_mask/run/ctsm5.3.018_SP_f09_t232_mask.clm2.h0.0001-01.nc`
+
+  NOTE: The regridding method set in `regrid_method` MUST match the method in the weights file
+
+  Example yaml file: `config_clm_native_grid_to_latlon.yaml`
+
+
+:

README.md

@@ -97,7 +97,7 @@ This generates time series files, climatology (climo) files, re-gridded climo fi
 ### What the ADF produces
 
 A run goes through the stages below in order. Each writes to a location set in the `diag_basic_info`
-or case block of the config file. Time series can be skipped with `cam_ts_done: true`, the website with
+or case block of the config file. Time series can be skipped with `calc_cam_ts: false`, the website with
 `create_html: false`, and the other stages by removing their scripts from the config.
 
 | Stage | Output | Config key |

config_amwg_default_plots.yaml

@@ -164,11 +164,13 @@ diag_cam_climo:
     #       end at latest available year.
     end_year: 14
 
-    #Do time series files exist?
-    #If True, then diagnostics assumes that model files are already time series.
-    #If False, or if simply not present, then diagnostics will attempt to create
-    #time series files from history (time-slice) files:
-    cam_ts_done: false
+    #Calculate time series files?
+    #If true, or if simply not present, then diagnostics will create time series
+    #files from the history (time-slice) files in 'cam_hist_loc'.
+    #If false, then diagnostics will use the pre-made time series files in
+    #'cam_ts_loc' instead.  (This used to be 'cam_ts_done', which meant the
+    #opposite; it is still accepted.)
+    calc_cam_ts: true
 
     #Save interim time series files?
     #WARNING:  This can take up a significant amount of space,
@@ -242,11 +244,13 @@ diag_cam_baseline_climo:
     #       end at latest available year.
     end_year: 14
 
-    #Do time series files need to be generated?
-    #If True, then diagnostics assumes that model files are already time series.
-    #If False, or if simply not present, then diagnostics will attempt to create
-    #time series files from history (time-slice) files:
-    cam_ts_done: false
+    #Calculate time series files?
+    #If true, or if simply not present, then diagnostics will create time series
+    #files from the history (time-slice) files in 'cam_hist_loc'.
+    #If false, then diagnostics will use the pre-made time series files in
+    #'cam_ts_loc' instead.  (This used to be 'cam_ts_done', which meant the
+    #opposite; it is still accepted.)
+    calc_cam_ts: true
 
     #Save interim time series files for baseline run?
     #WARNING:  This can take up a significant amount of space:

config_cam_baseline_example.yaml

@@ -80,6 +80,22 @@ diag_basic_info:
     #isn't specified in the variable defaults file.
     obs_data_loc: /glade/campaign/cgd/amp/amwg/ADF_obs
 
+    #Location of the CESM2 LENS2 global, annual mean files plotted behind the
+    #global mean time series.  Each variable's file is named by "lens2_file" in
+    #the variable defaults; leave this out and no LENS2 range is drawn (unless
+    #a "lens2_file" is given as a full path):
+    lens2_data_loc: /glade/campaign/cgd/cas/islas/CESM_DATA/LENS2/global_means/annualmeans
+
+    #Plot native-grid (unstructured, e.g. ne30pg3) cases on their own mesh with
+    #UXarray, instead of gridding them to lat/lon first.  Each native case then
+    #needs a "mesh_file" (see the case settings below).  If false, or missing,
+    #native cases are gridded to lat/lon with their "weights_file":
+    #unstructured_plotting: false
+
+    #Region definitions for the regional (land) plots listed in "region_list"
+    #(see the end of this file).  If missing, ADF/lib/regions_lnd.yaml is used:
+    #regions_file: /some/path/to/regions_file.yaml
+
     #Location where re-gridded and interpolated CAM climatology files are stored:
     cam_regrid_loc: /glade/derecho/scratch/${user}/ADF/regrid
 
@@ -234,12 +250,20 @@ diag_cam_climo:
     #Note:  Leaving this entry blank will make time series
     #       end at latest available year.
     end_year: 14
+    #("ts_start_year" and "ts_end_year" are accepted as other names for these.)
+
+    #Years the climatology covers, within the time series years above.  If
+    #missing, the climatology covers all of the time series years:
+    #climo_start_year: 12
+    #climo_end_year: 14
 
-    #Do time series files exist?
-    #If True, then diagnostics assumes that model files are already time series.
-    #If False, or if simply not present, then diagnostics will attempt to create
-    #time series files from history (time-slice) files:
-    cam_ts_done: false
+    #Calculate time series files?
+    #If true, or if simply not present, then diagnostics will create time series
+    #files from the history (time-slice) files in 'cam_hist_loc'.
+    #If false, then diagnostics will use the pre-made time series files in
+    #'cam_ts_loc' instead.  (This used to be 'cam_ts_done', which meant the
+    #opposite; it is still accepted.)
+    calc_cam_ts: true
 
     #Save interim time series files?
     #WARNING:  This can take up a significant amount of space,
@@ -253,6 +277,26 @@ diag_cam_climo:
     #Location where time series files are (or will be) stored:
     cam_ts_loc: /glade/derecho/scratch/${user}/ADF/${diag_cam_climo.cam_case_name}/ts
 
+    #Native-grid cases
+    #-----------------
+    #ADF finds out from the files whether a case is on a native (unstructured)
+    #grid.  Such a case needs one of these, depending on "unstructured_plotting":
+    #
+    #ESMF mesh file, to plot the case on its own grid ("unstructured_plotting: true"):
+    #mesh_file: /glade/campaign/cesm/cesmdata/inputdata/share/meshes/ne30pg3_ESMFmesh_cdf5_c20211018.nc
+    #
+    #Weights file that grids the case to the baseline's (or observations')
+    #lat/lon grid ("unstructured_plotting: false").  The weights decide the
+    #remapping (bilinear, conservative, ...):
+    #weights_file: /glade/campaign/cesm/cesmdata/inputdata/cpl/gridmaps/ne30pg3/map_ne30pg3_TO_fv0.9x1.25_aave.240826.nc
+    #
+    #How the weights are applied: "conservative" (the default) or "bilinear":
+    #regrid_method: conservative
+    #
+    #CLM only: a file on the target lat/lon grid holding its "landmask", used
+    #to mask the gridding of land output:
+    #latlon_file: /some/path/to/latlon_landmask_file.nc
+
     #TEM diagnostics
     #---------------
     #TEM history file number
@@ -314,9 +358,9 @@ diag_cam_climo:
     #    - 14
     #    - 14
 
-    #cam_ts_done:
-    #    - false
-    #    - false
+    #calc_cam_ts:
+    #    - true
+    #    - true
 
     #cam_ts_save:
     #    - true
@@ -396,12 +440,20 @@ diag_cam_baseline_climo:
     #Note:  Leaving this entry blank will make time series
     #       end at latest available year.
     end_year: 14
+    #("ts_start_year" and "ts_end_year" are accepted as other names for these.)
+
+    #Years the climatology covers, within the time series years above.  If
+    #missing, the climatology covers all of the time series years:
+    #climo_start_year: 12
+    #climo_end_year: 14
 
-    #Do time series files need to be generated?
-    #If True, then diagnostics assumes that model files are already time series.
-    #If False, or if simply not present, then diagnostics will attempt to create
-    #time series files from history (time-slice) files:
-    cam_ts_done: false
+    #Calculate time series files?
+    #If true, or if simply not present, then diagnostics will create time series
+    #files from the history (time-slice) files in 'cam_hist_loc'.
+    #If false, then diagnostics will use the pre-made time series files in
+    #'cam_ts_loc' instead.  (This used to be 'cam_ts_done', which meant the
+    #opposite; it is still accepted.)
+    calc_cam_ts: true
 
     #Save interim time series files for baseline run?
     #WARNING:  This can take up a significant amount of space:
@@ -414,6 +466,12 @@ diag_cam_baseline_climo:
     #Location where time series files are (or will be) stored:
     cam_ts_loc: /glade/derecho/scratch/${user}/ADF/${diag_cam_baseline_climo.cam_case_name}/ts
 
+    #Native-grid baseline: the same settings as for the test case(s) above
+    #("mesh_file", "weights_file", "regrid_method", "latlon_file"):
+    #mesh_file: /glade/campaign/cesm/cesmdata/inputdata/share/meshes/ne30pg3_ESMFmesh_cdf5_c20211018.nc
+    #weights_file: /glade/campaign/cesm/cesmdata/inputdata/cpl/gridmaps/ne30pg3/map_ne30pg3_TO_fv0.9x1.25_aave.240826.nc
+    #regrid_method: conservative
+
     #TEM diagnostics
     #---------------
     #TEM history file number
@@ -568,7 +626,7 @@ plotting_scripts:
 #
 #A single entry of "all" processes every variable found in the test case input
 #files instead (the fields in the history files, or the variables in the
-#pre-made time series files when "cam_ts_done" is true).  Variables listed
+#pre-made time series files when "calc_cam_ts" is false).  Variables listed
 #alongside "all" are still added, which is how a derived variable that is in no
 #input file (e.g. PRECT) can be asked for:
 #
@@ -616,7 +674,21 @@ diag_var_list:
 #    - U850
 #    - V200
 #    - V850
-
+
+#Regions for the regional (land) climatology and time series plots
+#("regional_climatology", "regional_timeseries"), by name from "regions_file".
+#If missing, no regional plots are made:
+#region_list:
+#    - Global
+#    - CONUS
+#    - Amazonia
+
+#Variables for the regional plots (at most 15).  If missing, a default set of
+#land variables is used:
+#regional_var_list:
+#    - TBOT
+#    - PREC
+
 # Options for multi-case regional contour plots (./plotting/regional_map_multicase.py)
 # region_multicase:
 #     region_spec: [slat, nlat, wlon, elon]

config_cam_baseline_example_testENSO.yaml

@@ -165,11 +165,13 @@ diag_cam_climo:
     #       end at latest available year.
     end_year: 44
 
-    #Do time series files exist?
-    #If True, then diagnostics assumes that model files are already time series.
-    #If False, or if simply not present, then diagnostics will attempt to create
-    #time series files from history (time-slice) files:
-    cam_ts_done: false
+    #Calculate time series files?
+    #If true, or if simply not present, then diagnostics will create time series
+    #files from the history (time-slice) files in 'cam_hist_loc'.
+    #If false, then diagnostics will use the pre-made time series files in
+    #'cam_ts_loc' instead.  (This used to be 'cam_ts_done', which meant the
+    #opposite; it is still accepted.)
+    calc_cam_ts: true
 
     #Save interim time series files?
     #WARNING:  This can take up a significant amount of space,
@@ -244,9 +246,9 @@ diag_cam_climo:
     #    - 14
     #    - 14
 
-    #cam_ts_done:
-    #    - false
-    #    - false
+    #calc_cam_ts:
+    #    - true
+    #    - true
 
     #cam_ts_save:
     #    - true
@@ -327,11 +329,13 @@ diag_cam_baseline_climo:
     #       end at latest available year.
     end_year: 14
 
-    #Do time series files need to be generated?
-    #If True, then diagnostics assumes that model files are already time series.
-    #If False, or if simply not present, then diagnostics will attempt to create
-    #time series files from history (time-slice) files:
-    cam_ts_done: false
+    #Calculate time series files?
+    #If true, or if simply not present, then diagnostics will create time series
+    #files from the history (time-slice) files in 'cam_hist_loc'.
+    #If false, then diagnostics will use the pre-made time series files in
+    #'cam_ts_loc' instead.  (This used to be 'cam_ts_done', which meant the
+    #opposite; it is still accepted.)
+    calc_cam_ts: true
 
     #Save interim time series files for baseline run?
     #WARNING:  This can take up a significant amount of space: