Attempt to kerchunk classic netcdf file not working.

[noaaroland]

I'm trying to get started with some of the concepts used by this project. I was able to do some simple things with xarray and xpublish, but all my attempts to use kerchunk have failed so far.

For example, looking at this file:
wget https://data.pmel.noaa.gov/aclim/thredds/fileServer/B10K-K20_CORECFS/Level1/1970-1974/B10K-K20_CORECFS_1970-1974_average_Jel.nc

When I check it with ncdump -k it says "classic" so I try:

import fsspec as fs
from kerchunk.netCDF3 import NetCDF3ToZarr
so = {'mode':'r'}
u = 'B10K-K20_CORECFS_1970-1974_average_Jel.nc'
with fs.open(u, **so) as infile:
    ncchunks = NetCDF3ToZarr(infile, u)

I get the following error:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 83: invalid start byte

TIA

[mpiannucci]

You need to open it with "rb" mode because it is a binary file. You also may need to use the latest GitHub revision of the main branch because we had to submit a few PRs to properly process legacy netcdf3 files

[noaaroland]

Thank you...

Is this what you mean by "the latest version of the main branch'?

(cf) [rhs@sour ~]$ pip install git+https://github.com/fsspec/kerchunk
pip install git+https://github.com/fsspec/kerchunk
...
Installing collected packages: kerchunk
Successfully installed kerchunk-0.2.2

I started with 'rb' as below, but the mode is already set in the call internal to kerchunk.

import fsspec as fs
from kerchunk.netCDF3 import NetCDF3ToZarr
so = {'mode':'rb'}
u = 'B10K-K20_CORECFS_1970-1974_average_Jel.nc'
with fs.open(u, **so) as infile:
    ncchunks = NetCDF3ToZarr(infile, u)

File "/home/users/rhs/kerchunk/kerchunk/kerchunk/netCDF3.py", line 64, in init
super().init(
TypeError: netcdf_file.init() got multiple values for argument 'mode'

The mode is set here:

        with fsspec.open(filename, **(storage_options or {})) as fp:
            super().__init__(
                fp, *args, mmap=False, mode="r", maskandscale=False, **kwargs
            )

Using kerchunk source, and removing the mode from that line and passing in 'rb', returns an error:

  File "/home/users/rhs/miniforge-pypy3/envs/cf/lib/python3.11/site-packages/scipy/io/_netcdf.py", line 234, in __init__
    raise ValueError("Mode must be either 'r', 'w' or 'a'.")
ValueError: Mode must be either 'r', 'w' or 'a'.

So, I'm still stuck.

Thanks,
Roland

[mpiannucci]

Ah right, i remember now. NetCDF3ToZarr uses fsspec internally, so you only have to pass the filename and storage options are optional. I got stuck on this when first using the netcdf3 function too, its definitely inconsistent. I just downloaded your file and was successful using this code:

from kerchunk.netCDF3 import NetCDF3ToZarr
import ujson

fname = './B10K-K20_CORECFS_1970-1974_average_Jel.nc'

# NetCDF3ToZarr uses fsspec internally, so only pass the file name and storage options
chunks = NetCDF3ToZarr(fname)
with open(f'{fname}.zarr', 'w') as f:
    f.write(ujson.dumps(chunks.translate()))