ATM: Runtime memory increases associated w/NetCDF reads

[LarissaReames-NOAA]

Description

When running ATM in either global or regional mode PET VMPeak (as output by the ProfileMemory=true setting in ufs.configure) jumps up each time a LBC file is read (in regional) or a climatological file is read (for global runs). We do not see similar memory increases for other model components when run in coupled mode (S2S).

To Reproduce:

1. Compile with current develop branch (or anything back to at least #45c8b2a) using either intel or GNU on either Hera or Jet
2. Run either control_c48 or regional_control regression test with ProfileMemory = true in ufs.configure
3. Output VMPeak values increase in large jumps around the time LBCs/Climo files are read in

Additional context

This behavior was revealed by @DeniseWorthen when investigating Issue #2320 . It's not clear that these memory issues are the direct cause of the long run failures reported there given that they are evident when the reported "working" #45c8b2a is used.

Output

Memory profile traces of select PETs from both global and regional runs for reference. These were all produced on Hera but runs on Jet produce similar results.

With intel/2021.5.0

With intel/2023.2.0