nox vs tox

[bjlittle]

Just thought I'd start a developer conversation with regards to nox and tox, and which testing management and automation tool iris should adopt moving forwards.

We've had time to understand and leverage nox within iris for a while now, however there are several other packages that are starting to diverge and opt for tox instead.

I think it would be ideal to choose one tool, if possible, to ease the burden on developers as they hop between different SciTools packages (and other infrastructure) that we support. Consistent tooling is a big win, so let's see if we can agree on one option here.

To start the conversation, as I see it...

nox

pros

1. Very flexible
2. Easy customisation
3. Leverage the power of Python to do "anything"

cons

1. Relatively niche
2. Questions regarding support
3. No plugin ecosystem

tox

pros

1. Widely adopted and recognised as a standard
2. Rich plugin ecosystem, including conda
3. Well supported

cons

1. Can be restrictive for certain use cases, due to YAML expressiveness
2. "yet another settings language" to learn
3. need to re-implement the session caching, which we implemented for nox
4. prefer coding over "configuration options" (?@pp-mo personal preference?)
   - i.e. the converse of nox.pros[3] = 'Leverage the power of Python to do "anything"'

---

I tried to think of more cons for tox to make it balanced, but can't think of anything obvious at the moment... perhaps others can fill the gap.

Thoughts?

[pp-mo]

tried to think of more cons for tox ... perhaps others can fill the gap

I added a couple 😉

[jamesp]

I mean, you can still have the power of python with tox:

[testenv:do_complicated_thing]
description = Perform a complicated set of steps in an isolated environment
skip_install = True
deps =
    pyyaml
commands =
    python scripts/complicated_script.py

[jamesp]

+pro for tox is the plugin tox-run-command which is very useful for adhoc dev things locally.

[bjlittle]

I do find the tox plugin ecosystem very rich, and I must confess I've not missed the need to use python akin to nox; so for me it's not that big a pro for nox.

[bjlittle]

@pp-mo Also, when it comes to caching, nox and tox are on a par. Plus both would need caching at a higher CI infrastructure level, so I don't necessarily see that as an explicit con solely against tox.

[bjlittle]

I'd like this discussion to eventually come to a conclusion at some point.

So to those ends, once you make up your mind please cast your vote in the following table:

Adopt tox	Continue using nox
@bjlittle	@wjbenfold
@trexfeathers	@pp-mo
@lbdreyer
@tkknight

[bjlittle]

Looks like we're in favour of tox adoption

[wjbenfold]

My vote is nox because

• it's easier to get started understanding nox because it's mostly python doing python things, rather than having to learn tox to get started working with it
• voting against a unanimous consensus is daunting

[trexfeathers]

Well done for doing it, wouldn't be much of a discussion otherwise! 😍

[pp-mo]

After some thought, I also think we could+should stay with nox.
I'm generally preferring nox as I think it's "nicer" to use with a lower bar to understanding.
So maybe it's worth our support, for a while, while we work out whether it has a bright future or will remain niche.

Interestingly, tox itself does reference nox as a possibly more flexible alternative :
"Nox is a project similar in spirit to tox but different in approach. Nox’s key difference is that it uses Python scripts instead of a configuration file. Nox might be useful if you find tox’s configuration too limiting but aren’t looking to move to something as general-purpose as Invoke or Make."

If nox doesn't really gain traction, and remains a bit niche going forward, then I think we can ~easily transition our existing usage later, if there's a desire to do so.
We should reconsider if we hit something it really can't do, or we find ourselves adding something that's overly clever and nox-specific, but for now, I don't think we're at that point.

[trexfeathers]

From @SciTools/peloton: decision by end of July 2022 ⏲️

[bjlittle]

Hmmm it's now September.

I think we need to make a decision and move this forward or close it down.

[tkknight]

We have a 2:1 in favour of tox including our lead dev @bjlittle, so I think the decision has been made. tox FTW!

[trexfeathers]

Any tox experts out there: I'm gonna need some help working out an alternative to these lines that we use to create benchmarking environments:

iris/benchmarks/asv.conf.json

Lines 22 to 23 in b79c61d

	"sed -i 's/_PY_VERSIONS_ALL/_PY_VERSION_LATEST/g' noxfile.py",
	"nox --envdir={conf_dir}/.asv/env/nox01 --session=tests --install-only --no-error-on-external-run --verbose"

[pp-mo]

I think we can close this as decided.
Nearly there with #5184 (though it has been static for a while)