Ability to specify the desired dtype when decoding a variable with scale_factor and add_offset

[eschalkargans]

Hello,

I create this discussion following a recent xarray update that changes the decoding of encoded data via attributes scale_factor and add_offset. Long story short, before, my uint16 was decoded to a float32, but now is decoded to float64. The float32 was desirable for my usecase.

More details on this comment I made on the issue: #2304 (comment)

_choose_float_dtype implementation in the associated PR: https://github.com/pydata/xarray/pull/8713/files#diff-ebbe6dc9f2f17dc4a9777193c923783413a3a6814f6e85d82ade7ca9f8ca65c9R396

My question would be: is it possible to specify the desired target dtype? Either via a special encoding attribute (like scale_factor and add_offset, we could imagine encoded_dtype='uint16' and decoded_dtype='float32'? Or via a special kwarg to pass to open_dataset, like open_dataset(decode_cf=True, decoding_mapping={'/path/to/my_variable': 'float32'}?

Thanks!

[kmuehlbauer]

@eschalkargans

The desired target dtype can be specified by setting scale_factor and add_offset to this wanted dtype.

The CF conventions explicitly mention this here http://cfconventions.org/Data/cf-conventions/cf-conventions-1.11/cf-conventions.html#packed-data

_When packed data is written, the scale_factor and add_offset attributes must be of the same type as the unpacked data, which must be either float or double. Data of type float must be packed into one of these types: byte, unsigned byte, short, unsigned short. Data of type double must be packed into one of these types: byte, unsigned byte, short, unsigned short, int, unsigned int.

When packed data is read, it should be unpacked to the type of the scale_factor and add_offset attributes, which must have the same type if both are present. For guidance only, we suggest that packed data which does not conform to the rules of this section regarding the types of the data variable and attributes should be unpacked to double type, in order to minimise the risk of loss of precision._

[eschalkargans]

Excerpts from xr.show_versions():

xarray: 2024.5.0
pandas: 2.2.2
numpy: 1.26.4

[eschalkargans]

@kmuehlbauer

The desired target dtype can be specified by setting scale_factor and add_offset to this wanted dtype.

This is not possible when serializing to Zarr, because such information is stored as JSON, and JSON can represent only integers or floatings, without any consideration for the size in bytes. So we cannot distinguish from floats and doubles

[kmuehlbauer]

@eschalkargans I totally understand the issue with zarr but isn't that an implementation detail of zarr? Could this by chance be handled in the zarr backend code instead of the CF coder?

IMHO, this isn't CF specific and should be handled in the backend code. That's from my perspective, others might have other opinions, though.

[dcherian]

You could disable that part of the decoding pipeline and do it yourself?

[dcherian]

Alternatively, we could start handling valid_min and valid_max and use those to choose the smallest type that can represent data in that range.