Preparing MPAS CONUS-12km background

[chunhuazhou]

The goal is to create a CONUS domain with 12km resolution and to prepare for building the RRFSv2 test system.

The workflow follows the NSSL MPAS run, using the GSL-NSSL MPAS model that has RUC-LSM and some other changes. I copied / linked the real-time workflow from /misc/whome/role.wrfruc/HRRRv5 and made some changes to run it for the CONUS-12km domain, initialized from GFS/ GEFS at /lfs4/BMC/wrfruc/Chunhua.Zhou/JEDI-tests/HRRRv5-12km.

Under /lfs4/BMC/wrfruc/Chunhua.Zhou/JEDI-tests/HRRRv5-12km, use workflow.ens/hrrrv5.xml to run GEFS initialized ensemble forecast and workflow/hrrrv5.xml to run GFS initialized deterministic forecast.

Step 1: Download global 12km uniform meshes from https://mpas-dev.github.io/

$ wget https://www2.mmm.ucar.edu/projects/mpas/atmosphere_meshes/x1.4096002.tar.gz

$ wget https://www2.mmm.ucar.edu/projects/mpas/atmosphere_meshes/x1.4096002_static.tar.gz

Untar and get the following files:

x1.4096002.graph.info

x1.4096002.cvt.part.256

x1.4096002.grid.nc

x1.4096002.static.nc

Step 2: Create a CONUS 12km domain from the global 12km meshes - grid.nc

Using the file conus.custom.pts from MPAS-Limited-Area/docs/points-examples/ and modify it for the desired conus domain:

name: conus12km_mpas
Type: custom
Point: 40.0, -100.0
50.0, -145.0
50.0, -50.0
20.0, -65.0
20.0, -129.0

Use the create_region python script from MPAS-Limited-Area/ to get grid.nc file for the customized CONUS domain:

$ create_region conus.custom.pts x1.4096002.grid.nc

Output:
conus12km_mpas.grid.nc
conus12km_mpas.graph.info

Step 3: Run init_atmosphere_model and get conus12km_mpas.static.nc

$ cp /lfs4/BMC/wrfruc/Chunhua.Zhou/JEDI-tests/HRRRv5-12km/exec/init_atmosphere_model .
$ cp /lfs4/BMC/wrfruc/Chunhua.Zhou/JEDI-tests/skylab/mpas/MPAS-Model.master/namelist.init_atmosphere .
$ cp /lfs4/BMC/wrfruc/Chunhua.Zhou/JEDI-tests/skylab/mpas/MPAS-Model.master/streams.init_atmosphere .

Modify streams.init_atmosphere:

<streams>
<immutable_stream name="input"
                  type="input"
                  filename_template="conus12km_mpas.grid.nc">
                  input_interval="initial_only" />
<immutable_stream name="output"
                  type="output"
                  filename_template="conus12km_mpas.static.nc"
                  packages="initial_conds"
                  output_interval="initial_only" />
<immutable_stream name="surface"
                  type="output"
                  filename_template="x1.40962.sfc_update.nc"
                  filename_interval="none"
                  packages="sfc_update"
                  output_interval="86400" />
<immutable_stream name="lbc"
                  type="output"
                  filename_template="lbc.$Y-$M-$D_$h.$m.$s.nc"
                  filename_interval="output_interval"
                  packages="lbcs"
                  output_interval="3:00:00" />
</streams>

Modify namelist.init_atmosphere:

&nhyd_model
    config_init_case = 7
    config_start_time = '2010-10-23_00:00:00'
    config_stop_time = '2010-10-23_00:00:00'
    config_theta_adv_order = 3
    config_coef_3rd_order = 0.25
    config_interface_projection = 'linear_interpolation'
/
&dimensions
    config_nvertlevels = 55
    config_nsoillevels = 4
    config_nfglevels = 38
    config_nfgsoillevels = 4
/
&data_sources
    config_geog_data_path = '/lfs4/NAGAPE/hpc-wof1/ywang/MPAS/WPS_GEOG/'
    config_met_prefix = 'CFSR'
    config_sfc_prefix = 'SST'
    config_fg_interval = 86400
    config_landuse_data = 'MODIFIED_IGBP_MODIS_NOAH'
    config_topo_data = 'GMTED2010'
    config_vegfrac_data = 'MODIS'
    config_albedo_data = 'MODIS'
    config_maxsnowalbedo_data = 'MODIS'
    config_supersample_factor = 3
    config_30s_supersample_factor = 1
    config_use_spechumd = false
/
&vertical_grid
    config_ztop = 30000.0
    config_nsmterrain = 1
    config_smooth_surfaces = true
    config_dzmin = 0.3
    config_nsm = 30
    config_tc_vertical_grid = true
    config_blend_bdy_terrain = false
/
&interpolation_control
    config_extrap_airtemp = 'lapse-rate'
/
&preproc_stages
    config_static_interp = true
    config_native_gwd_static = true
    config_vertical_grid = false
    config_met_interp = false
    config_input_sst = false
    config_frac_seaice = false
/
&io
    config_pio_num_iotasks = 0
    config_pio_stride = 1
/
&decomposition
    config_block_decomp_file_prefix = 'conus12km_mpas.graph.info.part.'

Run init_atmosphere_model, using the following example run script (run_init.sh):

#!/bin/sh
#SBATCH --account=nrtrr
##SBATCH --qos=debug
#SBATCH --nodes=1-1
#SBATCH --tasks-per-node=1
#SBATCH -t 00:55:00
#SBATCH --job-name=init_atmosphere
#SBATCH -o out.init_atmosphere

source /lfs4/BMC/wrfruc/Chunhua.Zhou/JEDI-tests/HRRRv5-12km/module_files/modulefile.jet
ulimit -s unlimited
srun init_atmosphere_model

Output: conus12km_mpas.static.nc

Step 4: Use metis to partition the mesh for parallel run

Copy the source code from MPAS-DATA/metis-5.1.0.tar.gz Under /lfs4/BMC/wrfruc/Chunhua.Zhou/JEDI-tests/skylab/mpas/metis-5.1.0

$ module load gnu/13.2.0 intel/2023.2.0
$ make config
$ make

Example command to partition the mesh for using 120 cores:
$ gpmetis -minconn -contig -niter=200 conus12km_mpas.graph.info 120

Output: conus12km_mpas.graph.info.part.120

Step 5: Run a deterministic MPAS forecast, using workflow/hrrrv5.xml , with GFS GRIB2 data as IC/LBC

Main namelist entries for running the 12-km CONUS test:

&nhyd_model
    config_time_integration_order = 2
    config_dt = 60.0
    config_number_of_sub_steps = 3
    config_dynamics_split_steps = 3
    config_len_disp                 = 12000.0

Suggested number of cores: 120 cores as tested on Jet.

Step 6: Run 30-member ensemble forecast, using workflow.ens/hrrrv5.xml, with GEFS GRIB2 data as IC/LBC

The 1-hour forecast files (deterministic and 30-member ensemble, initialized at 12Z 05/20/2024, valid at 13Z), along with the RAP PrepBUFR observations:

Jet: /lfs4/BMC/wrfruc/Chunhua.Zhou/MPAS-DATA/conus12km_mpas/2024052013

Hera: /scratch2/BMC/wrfruc/chunhua/data/conus12km_mpas/2024052013

Another test case, initialized at 18Z 05/26/2024, valid at 00Z 05/27/2024, is also available:

Jet: /lfs4/BMC/wrfruc/Chunhua.Zhou/MPAS-DATA/conus12km_mpas/2024052700

Hera: /scratch2/BMC/wrfruc/chunhua/data/conus12km_mpas/2024052700

[guoqing-noaa]

Thanks @chunhuazhou for preparing the background and ensebmles.
This is a great start for us to explore creating a good mpasjedi test case.

Chunhua's MPAS runs used a new physics which cannot be handled by current mpasjedi in RDASApp.
So @Junjun-NOAA is working to repeat this case but with the community version MPAS model.

[ShunLiu-NOAA]

@guoqing-noaa 2024052700 case is good for RDAS test since there are more observations available at 00z. Let's focus on 00z case.
Could you let's know what is the reason that RDAS mpasjedi cannot handle the new physics in MPAS?

[guoqing-noaa]

@guoqing-noaa 2024052700 case is good for RDAS test since there are more observations available at 00z. Let's focus on 00z case. Could you let's know what is the reason that RDAS mpasjedi cannot handle the new physics in MPAS?

We can still use the 2024052700 case. But we have to use the community MPAS_model at the moment.
MPASJEDI bump localization crashed due to inconsistent physics settings. Please refer to this issue #87 for more information.

@Junjun-NOAA Please set up the case at 2024052700z. Thanks!

[guoqing-noaa]

FYI, We reran the 2024052700z case using the community model version (i.e. no GSL new physics) and so far the bumploc has been completed successfully. With this confirmation, @Junjun-NOAA is now working on generating 30 ensembles.

[guoqing-noaa]

Updates on this:
We used the old namelist.atmosphere (not the one from Chunhua's run) and we successfully ran the bumploc and a pure 3DEnVar run (see the below figure). So Chunhua's case also works!

But there is still one issue: if we want to test BJ's static BE, we would want to run 55 vertical levels. This is adapted from BJ's communication about static BE:
One thing I didn't mention is ... the interpolating the global B statistic from one mesh to another mesh works only in horizontal direction, but NOT in vertical direction. Thus we should configure the vertical coordinate the same. In all of my testing, I am using the default configuration of MPAS, that is 55 vertical levels with a 30 km model top. If needed, a procedure interpolating the B statistics between different vertical configurations should be developed further.

we will use GSL's MPAS version eventually, but at the moment, @chunhuazhou's setting has 60 vertical levels while @Junjun-NOAA 's rerun has 55 levels. It may be easier to test BE-related things with a 55-vertical level setting.
@TingLei-NOAA @ShunLiu-NOAA @hu5970 Do you also favor a 55 vertical level setting? Thanks!

[guoqing-noaa]

Close this issue as a MPAS conus12km 30 members case has been established, see issue #92 . Thank @chunhuazhou and @Junjun-NOAA for your work on this!

[guoqing-noaa]

@hongli-wang, please follow the above instructions from @chunhuazhou and this MPAS tutorial guide:https://docs.google.com/document/d/1ytJq-pFGKEkGsgGnp2z-RgBL_0tYYtgvWCvKgtqEzso/edit.

We use the community model default settings. If you want to have a quick setup, @Junjun-NOAA or @chunhuazhou can help you generate the grid.nc ans static.nc you need.

[hongli-wang]

@guoqing-noaa
Thanks. I will contact Chunhua and Junjun for assistance. One more question is on use of GEFS ensemble. It seems you are running JEDI with a regional MPAS ensemble with 30 members. If I heard correctly, you mentioned JEDI/MPAS can use GEFS ensemble. Is that correct?

[delippi]

@chunhuazhou, I'm currently trying to run some assimilation experiments of only wind observations using this test case. I'm getting a warning like:

WARNING: Variable uReconstructZonal not in input file.
WARNING: Variable uReconstructMeridional not in input file.

And then I get zero increments. I think this is why I can't get my wind assimilation tests to work properly--I've gotten temperature assimilation to work. Is it possible to add these 3D wind variables restart.nc background file?

[TingLei-NOAA]

@chunhuazhou and @delippi I suspect the wind analysis issue is related to the issue #148 @Junjun-NOAA @SamuelDegelia-NOAA . I will also refer to Donne 's info from there.