Skip to content

Commit

Permalink
Added whatsnew notes on netcdf delayed saving and Distributed support. (
Browse files Browse the repository at this point in the history
#5322)

* Added whatsnew notes on netcdf delayed saving and Distributed support.

* Fix formatting.

* --amend

Review change.

* Fix formatting.
  • Loading branch information
pp-mo committed May 18, 2023
1 parent aea58cc commit 79bc09e
Showing 1 changed file with 45 additions and 22 deletions.
67 changes: 45 additions & 22 deletions docs/src/whatsnew/3.6.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,28 +8,51 @@ This document explains the changes made to Iris for this release


.. dropdown:: v3.6 Release Highlights
:color: primary
:icon: info
:animate: fade-in
:open:

We're so excited about our recent support for **delayed saving of lazy data
to netCDF** (:pull:`5191`) that we're celebrating this important step change
in behavour with its very own dedicated release 🥳

We're super keen for the community to leverage the benefit of this new
feature within Iris that we've brought this release forward several months.
As a result, this minor release of Iris is intentionally light in content.
However, there are some other goodies available for you to enjoy, such as:

* Performing lazy arithmetic with an Iris :class:`~iris.cube.Cube` and a
:class:`dask.array.Array`, and
* Various improvements to our documentation resulting from adoption of
`sphinx-design`_ and `sphinx-apidoc`_.

As always, get in touch with us on :issue:`GitHub<new/choose>`, particularly
if you have any feedback with regards to delayed saving, or have any issues
or feature requests for improving Iris. Enjoy!
:color: primary
:icon: info
:animate: fade-in
:open:

We're so excited about our recent support for **delayed saving of lazy data
to netCDF** (:pull:`5191`) that we're celebrating this important step change
in behavour with its very own dedicated release 🥳

By using ``iris.save(..., compute=False)`` you can now save to multiple netcdf files
in parallel. See the new ``compute`` keyword in :func:`iris.fileformats.netcdf.save`.
This can share and re-use any common (lazy) result computations, and it makes much
better use of resources during any file-system waiting (i.e. it can use such periods to
progress the *other* saves).

Usage example::

# Create output files with delayed data saving.
delayeds = [
iris.save(cubes, filepath, compute=False)
for cubes, filepath in zip(output_cubesets, output_filepaths)
]
# Complete saves in parallel.
dask.compute(*delayeds)

This advance also includes **another substantial benefit**, because netcdf saves can
now use a
`Dask.distributed scheduler <https://docs.dask.org/en/stable/scheduling.html>`_.
With `Distributed <https://distributed.dask.org/en/stable/>`_ you can parallelise the
saves across a whole cluster. Whereas previously, the netcdf saving *only* worked with
a "threaded" scheduler, limiting it to a single CPU.

We're so super keen for the community to leverage the benefit of this new
feature within Iris that we've brought this release forward several months.
As a result, this minor release of Iris is intentionally light in content.
However, there are some other goodies available for you to enjoy, such as:

* Performing lazy arithmetic with an Iris :class:`~iris.cube.Cube` and a
:class:`dask.array.Array`, and
* Various improvements to our documentation resulting from adoption of
`sphinx-design`_ and `sphinx-apidoc`_.

As always, get in touch with us on :issue:`GitHub<new/choose>`, particularly
if you have any feedback with regards to delayed saving, or have any issues
or feature requests for improving Iris. Enjoy!


📢 Announcements
Expand Down

0 comments on commit 79bc09e

Please sign in to comment.