(#819) Github CI workflow to pin dependencies and populate release notes with dodal/nexgen version

[rtuck99]

Fixes #819

This workflow does the following:

• pins all our direct dependencies in setup.cfg under install_requires by:

  • Removing any explicit pin hashes in setup.cfg
  • Running pip freeze to extract the latest package versions, and attaches these to all the dependencies
  • Pushes a commit with a comment of the format "Pin dependencies prior to release. Dodal , nexgen " which is tagged with the tag name you specify when launching the workflow

• As doing an ordinary GH release requires you to specify a git commit ref beforehand, I've implemented this as a separate workflow which you should run before doing the GH release, that accepts the tagname, then you do a GH release as normal using this tag

• It needs the GH workflow token to have permissions to do a push to main branch

• Indirect dependencies aren't pinned

• I've tested it as far as I can locally - using act as described here https://github.com/DiamondLightSource/hyperion/wiki/Running-github-workflows-locally

[olliesilvester]

Logically I think this looks OK, but I have some thoughts:

• Could we have some unit tests for the functions in pin_versions.py
• I think there are some easier ways to do a lot of the functions in this script. Eg use re.sub to remove git hashes from a line, without worrying about removing the whole line and remembering comments.
• Are we able to get the release notes to say whether or not dodal/nexgen have changed versions?
• Is there a way to combine this with the original workflow, so we can do both at once?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (b16bb47) 92.88% compared to head (f02716a) 92.88%.
Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1097   +/-   ##
=======================================
  Coverage   92.88%   92.88%           
=======================================
  Files          65       65           
  Lines        3316     3316           
=======================================
  Hits         3080     3080           
  Misses        236      236           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[rtuck99]

Logically I think this looks OK, but I have some thoughts:

* Could we have some unit tests for the functions in `pin_versions.py`

I've added some unit tests. They don't run from CI but can be run manually.

* I think there are some easier ways to do a lot of the functions in this script. Eg use re.sub to remove git hashes from a line, without worrying about removing the whole line and remembering comments.

Whilst I probably could have used regular expressions to handle comments, I decided not to. Even with REs there would still be all this "remembering" of comments unless you want them stripped out of the output files. A lot of the parsing in here is fairly context-sensitive as only some of the lines in setup.cfg need to be processed, so the main processing loop doesn't use REs. If the file structure wasn't section-based I probably would have done something in sed rather than python.

Also, it turned out that as it needs to be processed twice, (once to unpin, once to pin) better to factor it out into lots of small methods.

* Are we able to get the release notes to say whether or not dodal/nexgen have changed versions?

Maybe, but suspect that would be more fiddly - as we'd need to check out the setup.cfg from the last tagged version, parse it and extract the versions. Also the workflow would need an extra input or we'd have to assume the version format.

* Is there a way to combine this with the original workflow, so we can do both at once?

Maybe but it isn't clear from the github documentation - the github release workflow trigger
https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#release
has a created activity type, but at that point it's already got a git hash.
My working assumption is that as if you manually create a release at time A and someone later merges a commit at time B before you've published the release at time C, you wouldn't want that random commit to sneak into the release, therefore it's not a thing I would expect it to support.

[olliesilvester]

OK, I think this seems sensible. As we discussed, it would eventually be nice to include this in the same workflow as the hyperion release, but we can leave it like this for now, since then it remains optional + we can make sure it works properly. If adding the tests to CI is too fiddly then I guess it doesn't matter too much, probably not worth spending ages trying to figure that out. I will approve once we've figured out the failing unit tests

[olliesilvester]

Looks good, potential for some follow up issues (tests in CI, combining workflows, linking release notes for other dependencies)