Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix md5 hash logic. Make it order agnostic. #640

Merged
merged 2 commits into from
Sep 25, 2024
Merged

Conversation

damjad
Copy link
Contributor

@damjad damjad commented Sep 24, 2024

The order of headers is not consistent in API calls. Sometimes, md5 comes first, and sometimes crc32.
The current code works fine when md5 comes first and fails when crc comes first.

Failure example:

curl -I https://storage.googleapis.com/gcp-public-data-arco-era5/ar/1959-2022-1h-240x121_equiangular_with_poles_conservative.zarr/.zattrs
HTTP/2 200
<redacted>
x-goog-hash: crc32c=KXvQqg==
x-goog-hash: md5=mZFLkyvTelC5g8XnyQrpOw==
<redacted>

In the above example, crc comes first and md5 comes later. The header x-goog-hash has the value crc32c=KXvQqg==, md5=mZFLkyvTelC5g8XnyQrpOw==.

Please take a look at the space after the comma. This extra space was failing the code for md5. This extra space is there because of how requests lib handles duplicate headers.

Order of headers is not consistent in API calls. Sometimes, md5 comes first and sometimes crc32.
@martindurant
Copy link
Member

Just a space :)

@martindurant martindurant merged commit 9f47c80 into fsspec:main Sep 25, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants