array_api usage

[ilan-gold]

Hello all,

I have been looking into the core code here (teehee) and I noticed the recent #1967. It made me wonder why array_api + dlpack is not used (in general, and also specifically in that PR). I am guessing "no one has done it yet" is the answer, which is fair considering we all have other things going on, but then I am curious what #1967 gives that array_api + dlpack wouldn't.

I think cupy, jax (mentioned in the PR), and numpy all support array_api if memory serves correctly.

Sorry for the short question, with probably a long answer, but don't want to presume too much. I also did a quick search of the issues and found nothing relevant to the array api, but might have missed something: https://data-apis.org/array-api/latest/index.html

[d-v-b]

We discussed the array API when we were thinking about the design for the Zarr array class. See #1614. My thoughts today are pretty much the same as they were back then -- the array api describes a lot of transformations of in-memory arrays, but zarr arrays are not generally in-memory, which puts a lot of the array API off limits for us without basically re-inventing dask.

That doesn't mean we should deliberately deviate from the array api of course. But I don't think we are the target for that standard.

[d-v-b]

this clarification is helpful, I think I'm the one who misunderstood things :) maybe @akshaysubr and @madsbk can shed more light on the design of the Buffer class in the context of your question?

[ilan-gold]

Thanks! And just to clarify with @dcherian's point - the array_api requires dlpack. We could get it for free basically, see e.g., https://data-apis.org/array-api/latest/API_specification/generated/array_api.array.__dlpack__.html#dlpack

[ilan-gold]

I think I will convert this into an issue since I am now a bit more convinced of its utility.

[ilan-gold]

Also @d-v-b I am 100% in agreement. It does not make sense for the Array class.

[akshaysubr]

Thanks for bringing this up. This is certainly worth some discussion and in fact, DLPack was in consideration but just not prioritized for the initial implementation. I'll comment a bit more on the issue that's created to avoid fragmentation.