Understanding xr.open_dataset backend_kwargs

[rsignell-usgs]

With the latest version of h5py from conda-forge, if you use open_dataset with engine=h5netcdf you get a warning message when opening netcdf4 files, suggesting that you explicitly set decode_vlen_strings=True:

I verified that adding that arg to h5netcdf.File() worked, turning off the warning message, but when I tried using backend_kwargs I couldn't figure out how to get it to work:

import fsspec
import xarray as xr
import h5netcdf

url = 's3://noaa-nwm-retro-v2.0-pds/full_physics/2018/201801010000.CHRTOUT_DOMAIN1.comp'
ncfile = fsspec.open(url)

#  works:
f = h5netcdf.File(ncfile.open(), decode_vlen_strings=True)

# doesn't work
ds = xr.open_dataset(ncfile.open(), engine='h5netcdf', backend_kwargs={'decode_vlen_strings':True})

yielding:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-a0d1ef1cefa8> in <module>
----> 1 dset = xr.open_dataset(ncfile.open(), engine='h5netcdf', backend_kwargs={'decode_vlen_strings':True})

/home/conda/store/d05c9ede5b971baf68a3777b3053fe561bc2f06dd4b6433f0f277e17dbf5129b-pangeo/lib/python3.8/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables, backend_kwargs, use_cftime, decode_timedelta)
    570 
    571         opener = _get_backend_cls(engine)
--> 572         store = opener(filename_or_obj, **extra_kwargs, **backend_kwargs)
    573 
    574     with close_on_error(store):

TypeError: open() got an unexpected keyword argument 'decode_vlen_strings'
​```

[spencerkclark]

I think @kmuehlbauer may have astutely addressed this recently in #4893, which should be included in the next xarray release (set to happen soon!).

[kmuehlbauer]

@rsignell-usgs

Yes, as @spencerkclark pointed out, #4893 should fix this. And while using xarray.open_dataset you don't need to hassle with the backend_kwargs since decode_vlen_strings=True will be default.

[rsignell-usgs]

Thanks guys. Glad to know this will be addressed in the next release. I'm still confused why my attempt to set this via backend_kwargs didn't work, but it's certainly not a big deal...

[kmuehlbauer]

@rsignell-usgs The backend_kwargs decode_vlen_strings isn't yet known in xarray H5netcdfBackendEntrypoint. Each backend_kwargs has to be specified at the specific BackendEntrypoint. If one isn't specified it raises the error you are seeing.