Python converter for RRFS/ [3D]-RTMA MSONET prepBUFR data

[PraveenKumar-NOAA]

Description

Write a Python converter for the MSONET prepBUFR data that converts a BUFR file into an IODA file for the following subset:

| MSONET | A48117 | MESONET SURFACE REPORTS (COOPERATIVE NETWORKS) |

• Use Python API to convert a BUFR file into an IODA-formatted netCDF file
• Ioda obs validation using ioda-validate.x
• Perform validation for the observations using the input BUFR and output IODA file
• Add output(s) to the testoutput directory
• Add tests to the CMakeLists.txt files
• Have the tests pass

Requirements

• All important mnemonics required by the RRFS/ [3D]-RTMA users need to be known.

Acceptance Criteria

• It will be finished when all files have been checked and tests have passed.

[guoqing-noaa]

@PraveenKumar-NOAA I have a question. Are the data you mentioned in this PR included in the regular prepbufr files?
If yes, can the current bufr2ioda.x handle it?

Can this yaml file be able to used to convert the mesonet data?
https://github.com/rrfsx/rrfs-workflow/blob/main/parm/rrfs/prepbufr_mesonet.yaml

[PraveenKumar-NOAA]

@guoqing-noaa yes, MSONET data will be used here from the prepbufr file, and BUFR2IODA can handle it.
Use the following YAML to convert the MSONET prepbufr data into NetCDF format:
https://github.com/NOAA-EMC/RDASApp/blob/develop/rrfs-test/IODA/yaml/bufr_ncep_prepbufr_msonet.yaml

[guoqing-noaa]

@PraveenKumar-NOAA Thanks for more information.
So I wonder why we need an extra Python converter?

[PraveenKumar-NOAA]

@guoqing-noaa great question, thanks. The bufr2ioda converter which uses a query interface and is still under development. It is used to convert a BUFR file into a NetCDF file based on a description a user creates in a YAML or a Python script. We are testing its capability using both, a yaml and a Python script. It seems that Python based converter is more flexible for the queries, derived variables, and unit conversions.

[guoqing-noaa]

@PraveenKumar-NOAA
Thanks for the more information. Could you give us some specific examples of what the flexibilities are?
Correct me if I am wrong, but I don't think we want to "enhance" the prepbufr contents or add derived variables. I would think we want to keep the observations as-is.

[PraveenKumar-NOAA]

@guoqing-noaa please find below an example for adpupa prepbufr:
https://github.com/JCSDA-internal/ioda-converters/blob/develop/test/testinput/bufr_ncep_prepbufr_adpupa.yaml
https://github.com/NOAA-EMC/GDASApp/blob/develop/ush/ioda/bufr2ioda/bufr2ioda_adpupa_prepbufr.py

For further clarification, we can chat separately including bufr2ioda developers.

[PraveenKumar-NOAA]

Following validation plots were created for the pressure and wind observations using the RRFS test data, dated 2022052619:

pressure (POB):
IODA plot:

windEastward (UOB):
IODA plot:

There plots are in perfect agreement with the BUFR plots and IODA plots created using the ioda-converters YAML, which can be seen here:
#38