Endorsing Scientific Python SPECs for xarray: Minimal Supported Versions and Nightly Wheels

[andersy005]

@pydata/xarray,

during today's meeting, @jarrodmillman presented a few Scientific Python SPECs that may be of interest to xarray and suggested endorsing them. it's important to note that endorsing these SPECs simply means that we believe they are good ideas and we would aim to implement our own versions accordingly. it is not a binding contract that forces us to follow a specific approach.

for instance, SPEC 0 outlines a policy regarding minimum supported versions. it recommends that all projects within the Scientific Python ecosystem adopt a common time-based policy for dropping support of older Python and core package versions. the policy applies to feature releases (e.g., Python 3.8.0, NumPy 1.19.0) rather than minor releases (e.g., Python 3.8.1, NumPy 1.19.2).

specifically, it suggests dropping support for a given version of Python three years after its initial release, and for other core packages two years after their initial release. it's worth mentioning that some aspects of this policy align with NumPy's NEP-29.

if there are no objections or reservations, we can proceed with endorsing the following SPECs:

• SPEC 0 - Minimal supported versions: NumPy has already endorsed it and implemented it through NEP29
• SPEC 4 - Nightly wheels: We are currently doing this and utilizing the proposed community organization on Anaconda (see here).

assuming we agree to move forward, i will submit two separate pull requests on the SPECs repository (https://github.com/scientific-python/specs) to officially state our endorsement

[max-sixty]

Thanks @andersy005 !

Sounds good on having a convention.

That said, does the convention strike anyone as quite long? This would mean we'd need to support pandas 1.5.0 for another year in our main branch. Our current convention is to support projects such as pandas for one year. As a reminder, this doesn't mean folks have to upgrade pandas, it just means they can't use the newer versions of xarray without upgrading pandas...

FWIW, I think it'd be worth clarifying in the doc that "support" here means "how long new releases of a package should permit older versions of dependencies", not the more conventional meaning of support as "we fix bugs for old versions" — like most medium-sized open-source projects, we only release fixes for the current version of xarray...

[jarrodmillman]

Feel free to make a PR to https://github.com/scientific-python/specs/blob/main/spec-0000/index.md with language about

"how long new releases of a package should permit older versions of dependencies",
not the more conventional meaning of support as "we fix bugs for old versions

That is definitely the intention and would be worth clarifying.

Also endorsing the SPEC doesn't mean you don't drop support earlier. We may want to clarify that too. My main interest with SPEC 0 is to make sure we don't support old versions of our dependencies forever. For instance, in NetworkX we dropped support for scipy 1.8 a few releases ago because we wanted to use the new scipy.sparse array API, which was introduced in 1.9 (see scipy/scipy#14822).

[max-sixty]

Done! scientific-python/specs#269

Also endorsing the SPEC doesn't mean you don't drop support earlier. We may want to clarify that too.

I do think in its current state that doesn't come across. And some sort of coordination would still be helpful — e.g. if pandas decided to have a shorter window, then there's no point xarray having a longer window, given xarray requires pandas.

[andersy005]

The PR containing endorsements resides here:

• spec-0 and spec-4 endorsements by xarray scientific-python/specs#270

[max-sixty]

Does this mean we're changing our support, extending from 12/18 months for deps/numpy, to 24 months?

[keewis]

from what I can tell, the 24 months would only apply to the core packages, of which we only use pandas, matplotlib, and numpy. And because our policy is "12 months from the next version's release", we're pretty close to the policy already. But yeah, maybe we should discuss the version policy in the next meeting.

However, as far as I remember, endorsement does not mean we have to adopt the SPEC entirely, just that we think this is a good idea in general.

[tupui]

Hi, you are correct, endorsement does not mean that you have to strictly follow the SPEC. It means that you agree with the general principles and have something along the line. So it's ok to say that you endorse and that your implementation of the SPEC means that you support for x years.

We are still making this as we go 😃 in the future, we might have "cards" to fill by projects with items to fill. i.e. here there would be an item for the number of years of support and you would put your own value.

[crusaderky]

I am too quite averse to supporting pandas releases that came out 2 years ago.