How the chunk sizes are selected in the fms of UFS

[TingLei-NOAA]

Hi, FMS folks,
I asked a similar question first on UFS forum and experts there clarified that the chunking is controlled in FMS layer.
(ufs-community/ufs-weather-model#1012). Any comments/suggestions from FMS experts are appreciated. The question with little modifications are below:
In the restart files generated by UFS, we recently found, the chunk size is "too small" for large model domains, as in the regional parallel runs at EMC,
netcdf fv3_dynvars {
dimensions:
xaxis_1 = 3646 ;
xaxis_2 = 3647 ;
yaxis_1 = 2527 ;
yaxis_2 = 2526 ;
zaxis_1 = 65 ;
Time = UNLIMITED ; // (1 currently)

........
float T(Time, zaxis_1, yaxis_2, xaxis_1) ;
T:_Storage = "chunked" ;
T:_ChunkSizes = 1, 7, 281, 406 ;
T:_Endianness = "little" ;
.............
Such small values of the chunk sizes make the recently implemented GSI parallel netcdf IO (for netcdf 4 based on hdf5) almost totally fail to show any speeding up.
However, when, for example, the chunk size were converted to values equal to their dimension sizes, the total wall clock time could be reduced 10 to 30 % ( from Eric Rogers) .
So, That will be great if , first, we can know what the current status is for the chunk size specification in FMS for UFS, and finally, if we can have an " optimal" chose for the chunk sizes in terms of the speeding performances of UFS model performance and the GSI (and, maybe, other postprocessing tools).
Thanks a lot!

[TingLei-NOAA]

An update: UFS developers pointed out that ncchksz in namelist fms2_io_nml is used in nf90_open ( default 64k). However, it is expected not only sizes but the shapes of the chunking could be specified to help the efficient IO to those netcdf files. I did some searching in fms and didn't find the chunking shapes are controlled by fms (seemingly selected by netcdf intrinsic functions according the sizes), is that right?
While, the shapes of the chunk are also important for the IO speeding , any classifications/suggestions on how we can realize the user specified chunking shapes and sizes ?

[rem1776]

@uramirez8707 Could you take a look at this io question when you get a chance?

[TingLei-NOAA]

@uramirez Thanks a lot for noticing this question!

[GFDL-Eric]

Hi @TingLei-NOAA,

Your findings are correct... FMS does not currently have a mechanism to control the chunking shape. Chunk shapes are defined in the nf90_def_var call as an optional argument. The function within FMS that calls nf90_def_var is called netcdf_add_variable and is located in the netcdf_io.F90 file of fms2_io. netcdf_add_variable does not currently accept or pass a chunksizes argument to nf90_def_var, so the "default" chunk sizes are used (n.b. although we didn't use the storage optional argument in the nf90_def_var call, chunking is used because an unlimited dimension is present). The determination of the "default" chunk sizes is a bit hazy, but ostensibly has something to do with the ncchksz parameter used in the nf90_open call. Obviously, without a way to specify chunksizes from FMS, changing the default chunking behavior might only be possible by increasing the ncchksz namelist parameter, a solution that I find to be rather clunky. An alternative solution would be to add an optional argument to netcdf_add_variable that would result in passing the dimension sizes in as the chunksizes argument, although there are hurdles with this method as well. First, a chunksize may be no larger than 4 GiB by NetCDF definition, so this could result in errors e.g. with the dimensions you referenced I think you would be over this 4 GiB threshold. The second issue is that this information would have to be passed presumably at the namelist level, and would be the default for all netcdf_add_variable calls. I'm not sure if doing this manner of chunking would affect the performance increases of doing it for these large files. Thirdly, I don't believe implementing this in all the places where netcdf_add_variable is called will be trivial... so if it does get the go ahead, it might take considerable time to implement. So aside from cranking up the ncchksz namelist parameter to a significantly higher value and seeing if that improves performance for your use case, there's not much to be done in the near term.

[TingLei-NOAA]

@GFDL-Eric Thank you so much for your rather complete description of the situation. I will share them with UFS developers and colleagues at EMC. Yes, to add a new function to specify the chunk shape and sizes in UFS/FMS are not trivial. Currently, we may try using nccopy to do the conversion, which would, in total, still give us a quicker throughout.