Adapt storage tests for changes in fsspec (#1819) (#1679)

[AdamWill]

This is adapted from the fixes that were rolled into #1785 for the v3 branch.

TODO:

• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

Deleted TODO items are not applicable as this changes no code, only tests.

[AdamWill]

On v3 the check in test_hierarchy was commented out entirely, but that seems excessive to me. The comment # no exception raised if path does not exist or is leaf indicates the primary point of these checks is to make sure the listdir calls run without causing an assertion, and we can still do that, and check that the result is one of the two we know are possible.

[dstansby]

Thanks for this! To get the tests passing, I think the version of fsspec in requirements_dev_optional.txt will need increasing to a version that's compatible with the new tests.

[AdamWill]

ugh, it's been a while, but I think the change is supposed to work with both old and new fsspec? I'll try and find a minute to remember it...