Memory-friendly parallel Python scripts to manipulate netCDF files.
On NCAR machines, the following commands will load all the necessary libraries and modules:
module load python
ncar_pylib
On other machines, use conda to install mpi4py, netCDF4, xarray, numpy.
The scripts must be run on multiple MPI tasks (using mpiexec or equivalent) via interactive jobs or batch jobs. A typical usage on an interactive session:
module load python
ncar_pylib
mpiexec -n 36 compress.py -f *.nc
To run parallel jobs on cheyenne, see https://www2.cisl.ucar.edu/resources/computational-systems/cheyenne/running-jobs/submitting-jobs-pbs
usage: compress.py [-h] -f [path [path ...]] [-x excl] [-v]
Compresses a given set of netcdf files in parallel. The compressed files will
be saved with a prefix "cmpr_". After having checked the compressed files are
ok, you can run the following linux command to OVERRIDE all of the original
files with their compressed versions: rename "cmpr_" "" cmpr_*
optional arguments:
-h, --help show this help message and exit
-f [path [path ...]] path to hist file(s) to read. Multiple files and wild
characters (*) accepted.
-x excl (optional). File names that have the string provided
after this flag will be discarded.
-v Verbose logging
usage: mavg.py [-h] -f [path [path ...]] [-x excl] [-v]
Generates weighed monthly avg files from given history files with more
frequently recorded data. The monthly files will be saved with a prefix
"mavg_". After having checked the average files are ok, you can remove the
prefix from all files at once by running the following linux command (Warning:
This will override any existing file, averaged or not!):
rename "mavg_" "" mavg_*
optional arguments:
-h, --help show this help message and exit
-f [path [path ...]] path to hist file(s) to read. Multiple files and wild
characters (*) accepted.
-x excl (optional). File names that have the string provided
after this flag will be discarded.
-v Verbose logging