Ensure consistent use of versions during pagination

[rlamy]

Fixes #590.

I didn't add a test since I don't know how to write a sensible unit-test for paginated results.

[martindurant]

So that I understand what's going on here: listing of files should either be versioned or not, because it's not anticipated to be able to "list" a specific file generation? We usually do allow list of a specific path, though (would return a list of one item, essentially the same as info() ), so we need to make sure that that still works. Of course, such a listing with a specified generation would never be paginated.

To test, the call accepts max-keys, so that could be set to a small number for the sake of a test. I'm not immediately sure if that would need to be a monkeypatch. (having said that, pushing >1000 files to a local moto server shouldn't take that long at all)

[rlamy]

The only use of generation is to decide whether to ask for versions. I guess it's indeed to support single-file ls. The issue seems to be that the page token may be rejected (depending on bucket settings, auth method, ...) if versions isn't set in the same way as in the first call.

I tried your suggestion for testing, but it looks like gcs-fake-server (this is GCS, not S3 😉) doesn't ever return paginated results.

[martindurant]

Got it :)

[rlamy]

Thank you! Any idea of when the next release will be?

[martindurant]

when the next release will be

As soon as we've sorted out the aiobotocore version pinning issue in s3fs (because they are always released together with fsspec)