the 0 U/V increment in 3DEnVar case #148

Junjun-NOAA · 2024-09-05T14:11:36Z

It looks like the U/V increment is 0 at all vertical levels in 3DEnVar case.
Using the static B interpolated from BJ's global static B, the single-obs (temperature) 3DVAR, 3DEnVar and Hybrid DA experiments have been conducted. Below is temperature and U increment at level = 20. The U and V increment at all vertical levels are 0.

SamuelDegelia-NOAA · 2024-09-05T14:59:49Z

Hi @Junjun-NOAA, I can look into this. Are you using rrfs_mpasjedi_2024052700_Ens3Dvar.yaml (recently renamed from sonde_singeob_airTemperature_mpasjedi.yaml) to run 3DEnVar?

Junjun-NOAA · 2024-09-05T15:11:29Z

Hi @SamuelDegelia-NOAA Thanks for your help. I was using sonde_singeob_airTemperature_mpasjedi.yaml, and the run directory is /scratch1/BMC/wrfruc/jjhu/rundir/RDASApp/expr/mpas_2024052700, you can take a look at the details. Thanks

TingLei-NOAA · 2024-09-05T20:20:41Z

@Junjun-NOAA @SamuelDegelia-NOAA I found this issue also exists for the old mpas-jedi case which I missed.
Some thorough investigation is needed!

SamuelDegelia-NOAA · 2024-09-05T20:25:21Z

@Junjun-NOAA @TingLei-NOAA I wonder if this has to do with there being no 3D wind data in the background (and ensemble) restart files except for:

        float u(Time, nEdges, nVertLevels) ;
                u:units = "m s^{-1}" ;
                u:long_name = "Horizontal normal velocity at edges" ;
        float w(Time, nCells, nVertLevelsP1) ;
                w:units = "m s^{-1}" ;
                w:long_name = "Vertical velocity at vertical cell faces" ;

It is hard to track these u, w variables since their names are so short and common. But should they be used to compute uReconstructZonal and uReconstructMeridional somewhere that isn't getting applied in our yaml?

TingLei-NOAA · 2024-09-05T20:37:00Z

@SamuelDegelia-NOAA In my case, the ensemble members are the same as my background files (as you described, only that normal wind exists). I guess there should be some internal conversion for those winds.
But, it deserve a verification if the ensemble spread of zeros caused that issue. I am trying to test this by hard coded ensemble perturbations.

SamuelDegelia-NOAA · 2024-09-05T20:46:47Z

Sorry @TingLei-NOAA, I edited my comment a little bit. I shouldn't say that it is specifically a zero spread issue or just us missing something to do the internal conversion from the normal winds into u, v. For example, I see in rrfs_mpasjedi_2024052700_Ens3Dvar.yaml that the background state variables include u whereas it is not included in the ensemble member state variables.

TingLei-NOAA · 2024-09-05T20:56:00Z

@SamuelDegelia-NOAA Thanks for your clarification. But I think we can first try if we could rule out if that is a zero spread issue.

TingLei-NOAA · 2024-09-06T14:31:12Z

@delippi reported an issue with wind analysis. It is maybe related to the current issue
#41 (comment)

TingLei-NOAA · 2024-09-06T14:57:20Z

An update: when the ensemble perturbation was enforced to be 1 by hardwiring in the code. The wind increment in the increment output were as expected. so, now, my guess is that , for the wind related variable conversion issue, the uReconstructZonal, uReconstructMeridional in the ensembles are zeroes and caused 0 wind increment.

guoqing-noaa · 2024-09-06T15:03:47Z

@Junjun-NOAA @TingLei-NOAA I wonder if this has to do with there being no 3D wind data in the background (and ensemble) restart files except for:
        float u(Time, nEdges, nVertLevels) ;
                u:units = "m s^{-1}" ;
                u:long_name = "Horizontal normal velocity at edges" ;
        float w(Time, nCells, nVertLevelsP1) ;
                w:units = "m s^{-1}" ;
                w:long_name = "Vertical velocity at vertical cell faces" ;
It is hard to track these u, w variables since their names are so short and common. But should they be used to compute uReconstructZonal and uReconstructMeridional somewhere that isn't getting applied in our yaml?

@SamuelDegelia-NOAA Do you mean we only have U and W, no V?

delippi · 2024-09-06T15:32:18Z

@TingLei-NOAA, thanks for pointing me to this issue. This is indeed the same issue I'm having. When trying to only assimilate winds I get the following warnings in my log file:

WARNING: Variable uReconstructZonal not in input file.
WARNING: Variable uReconstructMeridional not in input file.

I'm thinking the variable just needs to be added to the backgrounds. Or, if that isn't possible and as @SamuelDegelia-NOAA stated, a recipe that converts the horizonal normal velocity at edges u variable to uReconstructZonal and uReconstructMeridional.

guoqing-noaa · 2024-09-06T16:19:15Z

Thank @Junjun-NOAA @TingLei-NOAA @delippi @SamuelDegelia-NOAA for great detective work!

It seems we have to update the restart files. @chunhuazhou , could you help with this?

We can try to modify the restart stream in streams.atmosphere as follows:

<immutable_stream name="restart"
                  type="input;output"
                  filename_template="restart.$Y-$M-$D_$h.$m.$s.nc"
                  io_type="pnetcdf,cdf5"
                  input_interval="initial_only" 
                  clobber_mode="replace_files"
                  output_interval="1:00:00" />

        <stream name="output" />
        <var name="pressure" />
        <var name="uReconstructZonal" />
        <var name="uReconstructMeridional" />
</stream>

There is a related PR here: https://github.com/JCSDA-internal/mpas-jedi/pull/1012

guoqing-noaa · 2024-09-06T16:22:31Z

@Junjun-NOAA Could you also try in your test cases? Thanks!

TingLei-NOAA · 2024-09-06T16:24:54Z

@guoqing-noaa Yes. I think there should be some setup in those stream files to make the current use of uReconstructMeridional,uReconstructZonal. We could dig more and might have further discussion with mpasjedi core team to decide what analysis variable should be used for RRFS.
For being now, by using "u" (changes made in the yaml) rather than uReconstructMeridional,uReconstructZonal, the ens3dvar created the increment for u.
@Junjun-NOAA That will be interesting to see if this workaround is also working for your case .
My changed yaml is test-ens3dvar.yaml in hera: /scratch2/NCEPDEV/fv3-cam/Ting.Lei/dr-mpas-bundle/rdas-mpas-tests/CONUS_15km_da_oneob_test.

Junjun-NOAA · 2024-09-06T16:32:41Z

@guoqing-noaa @TingLei-NOAA Thanks for digging into this issue. I will try your suggestions and see what we will get.

delippi · 2024-09-06T16:43:08Z

@TingLei-NOAA, it looks like your yaml only assimilates a single temperature ob.

TingLei-NOAA · 2024-09-06T17:00:35Z

@TingLei-NOAA, it looks like your yaml only assimilates a single temperature ob.

Yes. that is the current setup of Junjun 's case ,which show no cross-covariance worked as expected (0 increment in wind).
That will be interesting to see what quick work around is available in your case. That will be great if you could point to your run dir and we can see if there are some setup to help.

chunhuazhou · 2024-09-06T17:16:19Z

Thank @Junjun-NOAA @TingLei-NOAA @delippi @SamuelDegelia-NOAA for great detective work!

It seems we have to update the restart files. @chunhuazhou , could you help with this?

We can try to modify the restart stream in streams.atmosphere as follows:
<immutable_stream name="restart"
                  type="input;output"
                  filename_template="restart.$Y-$M-$D_$h.$m.$s.nc"
                  io_type="pnetcdf,cdf5"
                  input_interval="initial_only" 
                  clobber_mode="replace_files"
                  output_interval="1:00:00" />

        <stream name="output" />
        <var name="pressure" />
        <var name="uReconstructZonal" />
        <var name="uReconstructMeridional" />
</stream>
There is a related PR here: https://github.com/JCSDA-internal/mpas-jedi/pull/1012

I will update the test data and let you know when they are available. Thanks!

SamuelDegelia-NOAA · 2024-09-06T17:21:12Z

Great work all. Just to confirm, do we know if uReconstructZonal and uReconstructMeridional are prognostic variables, or do we also need to add uas an analysis variable?

Edit: This link states that u is the only prognostic wind variable, so it sounds like we also need it as an analysis variable. But we can update uReconstructZonal and uReconstructMeridional for post-processing purposes.

guoqing-noaa · 2024-09-06T17:40:20Z

@SamuelDegelia-NOAA @TingLei-NOAA
It is interesting to see what the differences will be between using edgeU vs using reconstructed u/v.
The NCAR group uses the reconstructed u/v and has not tried edgeU

guoqing-noaa · 2024-09-06T17:40:51Z

I will update the test data and let you know when they are available. Thanks!

Thanks, @chunhuazhou

Junjun-NOAA · 2024-09-06T17:57:09Z

@TingLei-NOAA Update from using u as an analysis variable, the increment of uReconstructMeridional and uReconstructZonal are generated. Thanks

TingLei-NOAA · 2024-09-06T18:13:53Z

@Junjun-NOAA Great! we could dig more on related topics. Thanks for identifying this problem and initiate this discussion.

guoqing-noaa · 2024-09-07T20:14:29Z

@chunhuazhou
Junjun has done a few tests and the previous method of adding reconstructed wind only works for non-restart files and does NOT work for restart files.

Junjun and I are now testing the mpasout files. This task has become a top priority now as quite a few tasks depend on this.

HuiLiu-NOAA · 2024-09-09T19:56:55Z

I checked the mpas-jedi tutorial case, and the BG file does have the uReconstructZonal and uReconstructMeridional variables; but the BG files in both the old and new mpas case do NOT have the 2 variables. I guess this issue is just due to the missing of these two variables in the BG and restart files (and so there is no way to compute the analysis increments). Adding these variables to the BG and restart files should resolve this issue, hopefully.

HuiLiu-NOAA · 2024-09-09T20:04:29Z

I guess the BG file in the old MPAS case was also generated from the restart files.

HuiLiu-NOAA · 2024-09-09T20:08:08Z

Consult with the mpas-jedi team about how to generate the BG files with the 2 variables would be helpful, I guess.

guoqing-noaa · 2024-09-09T20:30:23Z

@HuiLiu-NOAA @TingLei-NOAA @delippi @Junjun-NOAA

I generated a new case yesterday where we have reconstructed winds. If you want to give it a try, I can share the path. The only part is that the ensemble spread is small and the analysis increment max is about 0.46 K.

chunhuazhou · 2024-09-09T20:32:26Z

@chunhuazhou Junjun has done a few tests and the previous method of adding reconstructed wind only works for non-restart files and does NOT work for restart files.

Junjun and I are now testing the mpasout files. This task has become a top priority now as quite a few tasks depend on this.

Thanks for telling me this. I didn't get the restart files to have uReconstructZonal and uReconstructMeridional but did add them to the mpasout files. My forecast jobs (with mpasout) are in the queue. I will update it here when they are ready.

HuiLiu-NOAA · 2024-09-09T20:32:38Z

I am glad to try it.

…

On Mon, Sep 9, 2024 at 2:30 PM Guoqing Ge ***@***.***> wrote: @HuiLiu-NOAA <https://github.com/HuiLiu-NOAA> @TingLei-NOAA <https://github.com/TingLei-NOAA> @delippi <https://github.com/delippi> @Junjun-NOAA <https://github.com/Junjun-NOAA> I generated a new case yesterday where we have reconstructed winds. If you want to give it a try, I can share the path. The only part is that the ensemble spread is small and the analysis increment max is about 0.46 K. — Reply to this email directly, view it on GitHub <#148 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOF252HUA5Y7YNJC5IKPRPLZVYAPJAVCNFSM6AAAAABNWS54ASVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZZGAZDSNZYGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

guoqing-noaa · 2024-09-09T20:33:24Z

I'm working on generating another case (using 36h GEFS forecasts for ensemble BEC) using the rrfs-workflow, but this may take some time.

chunhuazhou · 2024-09-09T20:34:13Z

@HuiLiu-NOAA @TingLei-NOAA @delippi @Junjun-NOAA

I generated a new case yesterday where we have reconstructed winds. If you want to give it a try, I can share the path. The only part is that the ensemble spread is small and the analysis increment max is about 0.46 K.

@guoqing-noaa Guess my message came cross yours - did you already generated the ensemble mpasout files?

guoqing-noaa · 2024-09-09T20:35:11Z

@HuiLiu-NOAA @TingLei-NOAA @delippi @Junjun-NOAA
I generated a new case yesterday where we have reconstructed winds. If you want to give it a try, I can share the path. The only part is that the ensemble spread is small and the analysis increment max is about 0.46 K.

@guoqing-noaa Guess my message came cross yours - did you already generated the ensemble mpasout files?

:) Not yet. If you have those mpasout, I can use them. Thank you, @chunhuazhou

guoqing-noaa · 2024-09-09T20:37:04Z

@HuiLiu-NOAA
If you want to give it a try now, here is my run directory for the mpasjedi2024052700 case:
/scratch1/BMC/wrfruc/gge/RDASApp/expr/mpas_2024052700

TingLei-NOAA · 2024-09-09T20:38:32Z

I checked the mpas-jedi tutorial case, and the BG file does have the uReconstructZonal and uReconstructMeridional variables; but the BG files in both the old and new mpas case do NOT have the 2 variables. I guess this issue is just due to the missing of these two variables in the BG and restart files (and so there is no way to compute the analysis increments). Adding these variables to the BG and restart files should resolve this issue, hopefully.

By BG file, did you mean nicas parameter files like bumploc_200km6km_nicas_local_000036-000034.nc?
In my understanding, when they are generated using the current setup as i bumploc_connus15km.yaml, there won't variables information in those files. The "common" group will be used for every variables and the cross-covariance would be realized by the "multivariate strategy : duplicated".

HuiLiu-NOAA · 2024-09-09T20:39:35Z

will do.

…

On Mon, Sep 9, 2024 at 2:37 PM Guoqing Ge ***@***.***> wrote: @HuiLiu-NOAA <https://github.com/HuiLiu-NOAA> If you want to give it a try now, here is my run directory for the mpasjedi2024052700 case: /scratch1/BMC/wrfruc/gge/RDASApp/expr/mpas_2024052700 — Reply to this email directly, view it on GitHub <#148 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOF252HJCG5CNDZ4VSGBEUDZVYBINAVCNFSM6AAAAABNWS54ASVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZZGA2DCNJTGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

chunhuazhou · 2024-09-09T20:41:23Z

@HuiLiu-NOAA @TingLei-NOAA @delippi @Junjun-NOAA
I generated a new case yesterday where we have reconstructed winds. If you want to give it a try, I can share the path. The only part is that the ensemble spread is small and the analysis increment max is about 0.46 K.

@guoqing-noaa Guess my message came cross yours - did you already generated the ensemble mpasout files?

:) Not yet. If you have those mpasout, I can use them. Thank you, @chunhuazhou

They are almost ready on Jet:

deterministic forecast (ready) /lfs5/BMC/wrfruc/Chunhua.Zhou/w4/nco/stmp/conus12km/1.0.1/rrfs.20240526/23/fcst/ctl/mpasout.2024-05-27_00.00.00.nc
ensemble forecast files (initialized from GEFS 35-hour forecast, some are still in the queue) /lfs5/BMC/wrfruc/Chunhua.Zhou/w4/nco/stmp/conus12km/1.0.1/rrfs.20240526/23/fcst/mem*/mpasout.2024-05-27_00.00.00.nc

HuiLiu-NOAA · 2024-09-09T20:41:51Z

I looked at the bg.2022-05-26_19.00.00.nc in the run directory.

…

On Mon, Sep 9, 2024 at 2:39 PM TingLei-NOAA ***@***.***> wrote: I checked the mpas-jedi tutorial case, and the BG file does have the uReconstructZonal and uReconstructMeridional variables; but the BG files in both the old and new mpas case do NOT have the 2 variables. I guess this issue is just due to the missing of these two variables in the BG and restart files (and so there is no way to compute the analysis increments). Adding these variables to the BG and restart files should resolve this issue, hopefully. By BG file, did you mean nicas parameter files like bumploc_200km6km_nicas_local_000036-000034.nc? In my understanding, when they are generated using the current setup as i bumploc_connus15km.yaml, there won't variables information in those files. The "common" group will be used for every variables and the cross-covariable would be done by the "multivariate strategy : duplicated". — Reply to this email directly, view it on GitHub <#148 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOF252BYVALVCAFDHTMDGVDZVYBN7AVCNFSM6AAAAABNWS54ASVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZZGA2DIMBSGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

TingLei-NOAA · 2024-09-09T20:47:58Z

I looked at the bg.2022-05-26_19.00.00.nc in the run directory.
…
On Mon, Sep 9, 2024 at 2:39 PM TingLei-NOAA @.> wrote: I checked the mpas-jedi tutorial case, and the BG file does have the uReconstructZonal and uReconstructMeridional variables; but the BG files in both the old and new mpas case do NOT have the 2 variables. I guess this issue is just due to the missing of these two variables in the BG and restart files (and so there is no way to compute the analysis increments). Adding these variables to the BG and restart files should resolve this issue, hopefully. By BG file, did you mean nicas parameter files like bumploc_200km6km_nicas_local_000036-000034.nc? In my understanding, when they are generated using the current setup as i bumploc_connus15km.yaml, there won't variables information in those files. The "common" group will be used for every variables and the cross-covariable would be done by the "multivariate strategy : duplicated". — Reply to this email directly, view it on GitHub <#148 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOF252BYVALVCAFDHTMDGVDZVYBN7AVCNFSM6AAAAABNWS54ASVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZZGA2DIMBSGU . You are receiving this because you were mentioned.Message ID: @.>

OK, the background files. Got it! Thanks. I think as @guoqing-noaa is doing, adding those missing variables to background files and ensembles would make it work. The issue I want to find out is that the reconstruced wind should be converted from the edge-u through mpas-jedi's variable change and it seems working for analysis variable (otherwise it should report "error" or the app doesn't have the check and just let them 0s? )but, it seems not working for ensemble components. Those are what I hope to sort out.

HuiLiu-NOAA · 2024-09-09T20:59:46Z

Thanks @TingLei-NOAA for the clarification.
Maybe checking the MPAS (not mpas-jedi) code for the analysis variable changes might be helpful? I think this transform of edge-u to the reconstruced is not done in the mpas-jedi VariableChange class, may be in MPAS-solo codes.

Agree! I think mpas lib should be invoked from mpas-jedi to those "complicated" conversion if such conversion is available in the latter.

chunhuazhou · 2024-09-09T21:46:19Z

@HuiLiu-NOAA @TingLei-NOAA @delippi @Junjun-NOAA
I generated a new case yesterday where we have reconstructed winds. If you want to give it a try, I can share the path. The only part is that the ensemble spread is small and the analysis increment max is about 0.46 K.

@guoqing-noaa Guess my message came cross yours - did you already generated the ensemble mpasout files?

:) Not yet. If you have those mpasout, I can use them. Thank you, @chunhuazhou

They are almost ready on Jet:

deterministic forecast (ready) /lfs5/BMC/wrfruc/Chunhua.Zhou/w4/nco/stmp/conus12km/1.0.1/rrfs.20240526/23/fcst/ctl/mpasout.2024-05-27_00.00.00.nc

ensemble forecast files (initialized from GEFS 35-hour forecast, some are still in the queue) /lfs5/BMC/wrfruc/Chunhua.Zhou/w4/nco/stmp/conus12km/1.0.1/rrfs.20240526/23/fcst/mem*/mpasout.2024-05-27_00.00.00.nc

The mpasout files are ready on Jet.

guoqing-noaa · 2024-09-09T21:49:12Z

Thanks, @chunhuazhou!

HuiLiu-NOAA · 2024-09-10T05:31:19Z

@guoqing-noaa : I just tested your newest case and got reasonable wind analysis increments. Thanks!

TingLei-NOAA mentioned this issue Sep 6, 2024

Preparing MPAS CONUS-12km background #41

Closed

TingLei-NOAA mentioned this issue Sep 9, 2024

Enhancing understanding and application of variable changes in mpas jedi #159

Open

guoqing-noaa mentioned this issue Sep 10, 2024

update the mpas_2024052700 case to use the mpasout files #160

Open

the 0 U/V increment in 3DEnVar case #148

the 0 U/V increment in 3DEnVar case #148

Comments

Junjun-NOAA commented Sep 5, 2024

SamuelDegelia-NOAA commented Sep 5, 2024

Junjun-NOAA commented Sep 5, 2024

TingLei-NOAA commented Sep 5, 2024

SamuelDegelia-NOAA commented Sep 5, 2024 • edited Loading

TingLei-NOAA commented Sep 5, 2024

SamuelDegelia-NOAA commented Sep 5, 2024

TingLei-NOAA commented Sep 5, 2024

TingLei-NOAA commented Sep 6, 2024

TingLei-NOAA commented Sep 6, 2024

guoqing-noaa commented Sep 6, 2024

delippi commented Sep 6, 2024

guoqing-noaa commented Sep 6, 2024

guoqing-noaa commented Sep 6, 2024

TingLei-NOAA commented Sep 6, 2024 • edited Loading

Junjun-NOAA commented Sep 6, 2024

delippi commented Sep 6, 2024

TingLei-NOAA commented Sep 6, 2024

chunhuazhou commented Sep 6, 2024

SamuelDegelia-NOAA commented Sep 6, 2024 • edited Loading

guoqing-noaa commented Sep 6, 2024

guoqing-noaa commented Sep 6, 2024

Junjun-NOAA commented Sep 6, 2024

TingLei-NOAA commented Sep 6, 2024

guoqing-noaa commented Sep 7, 2024

HuiLiu-NOAA commented Sep 9, 2024

HuiLiu-NOAA commented Sep 9, 2024

HuiLiu-NOAA commented Sep 9, 2024

guoqing-noaa commented Sep 9, 2024

chunhuazhou commented Sep 9, 2024

HuiLiu-NOAA commented Sep 9, 2024 via email

guoqing-noaa commented Sep 9, 2024

chunhuazhou commented Sep 9, 2024

guoqing-noaa commented Sep 9, 2024

guoqing-noaa commented Sep 9, 2024

TingLei-NOAA commented Sep 9, 2024 • edited Loading

HuiLiu-NOAA commented Sep 9, 2024 via email

chunhuazhou commented Sep 9, 2024 • edited Loading

HuiLiu-NOAA commented Sep 9, 2024 via email

TingLei-NOAA commented Sep 9, 2024

HuiLiu-NOAA commented Sep 9, 2024 • edited by TingLei-NOAA Loading

chunhuazhou commented Sep 9, 2024

guoqing-noaa commented Sep 9, 2024

HuiLiu-NOAA commented Sep 10, 2024

SamuelDegelia-NOAA commented Sep 5, 2024 •

edited

Loading

TingLei-NOAA commented Sep 6, 2024 •

edited

Loading

SamuelDegelia-NOAA commented Sep 6, 2024 •

edited

Loading

TingLei-NOAA commented Sep 9, 2024 •

edited

Loading

chunhuazhou commented Sep 9, 2024 •

edited

Loading

HuiLiu-NOAA commented Sep 9, 2024 •

edited by TingLei-NOAA

Loading