Skip to content

Conference call notes 20240619

Jump to bottom
Kenneth Hoste edited this page Jun 19, 2024 · 2 revisions

(back to Conference calls)

Notes on the 248th EasyBuild conference call, Wednesday 19 June 2024 (08:00 UTC)

Attendees

List of attendees (15):

  • Bibek Chapagain (Barcelona Supercomputing Centre, Spain)
  • Alex Domingo (Vrije Universiteit Brussel, Belgium)
  • Jasper Grimm (University of York, UK)
  • Alexander Grund (TU Dresden, Germany)
  • Kenneth Hoste (HPC-UGent, Belgium)
  • Chia-Jung Hsu (Chalmers University of Technology, Sweden)
  • Adam Huffman (Big Data Institute, Oxford, UK)
  • Kurt Lust (UAntwerpen, Belgium + LUMI User Support Team)
  • Sébastien Moretti (SIB, Switzerland)
  • Mikael Öhman (Chalmers University of Technology, Sweden)
  • Jörg Saßmannshausen (Imperial College London, UK)
  • Åke Sandgren (Umeå University, Sweden)
  • Yunqi Shao (Chalmers University of Technology, Sweden)
  • Alain van Hoof (TU Eindhoven, Netherlands)
  • Cintia Willemyns (Vrije Universiteit Brussel, Belgium)

Agenda

  • overview of recent developments
  • outlook to EasyBuild 5.0 release
  • 2024a update of common toolchains
  • Q&A

Recent developments

  • latest EasyBuild release: 4.9.2 (12 June 2024)
  • next EasyBuild release: hoping to make that EasyBuild 5.0.0 (but not before Aug'24)
  • EasyBuild v5.0.0
    • detailed notes on latest developments in https://hackmd.io/tZtz6vMmRfGL6N9Rf__OEA
    • to test development version of EasyBuild 5.0: 
      # set up Python virtual environment, and jump into it
python3 -m venv eb5
source eb5/bin/activate

# install EasyBuild 5.0 development version into it
pip install https://github.com/easybuilders/easybuild-framework/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyblocks/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyconfigs/archive/5.0.x.tar.gz

# go!
eb --version

Merged PRs

  • docs (merged PRs)

    • ...

  • framework (merged PRs)

    • bug fixes
      • symlink downloaded repo at specified commit when using --from-commit so easyconfigs for dependencies are found (PR #4552)
      • don't try to determine repo version when file that contains version doesn't exist (PR #4553)
    • enhancements
      • improve behavior when using extension which has nosource enabled (PR #4506)
      • enhance get_software_libdir to return lib or lib64 if only one of them contains library files (PR #4513)
    • changes
      • ...
    • code cleanup
      • ...
    • EasyBuild 5.0 (5.0.x branch)
      • trigger pre-hook in run_shell_cmd as early as possible so dry-run + trace output is correct (PR #4544)
      • add renamed alternative constants/templates/parameters (PR #4514)
      • include alternative names of easyconfig parameters in output of --avail-easyconfig-params (PR #4549)

  • easyblocks (merged PRs)

    • bug fixes
      • don't manipulate setup.py for Python 3.12+ when using alternate sysroot (PR #3313)
      • avoid sanity check on the software itself for CargoPythonBundle easyblock (PR #3341)
      • fix malformed $CPATH for intel-compilers on Debian/Ubuntu (PR #3346)
      • use PRRTE MCA environment variable for oversubscription in OpenMPI easyblock (PR #3360)
      • fix import of parse in openssl_wrapper.py for Python 2.7 (PR #3364)
      • stick to flake8 7.0.0 for linting easyblocks (for now) (PR #3365)
        • to fix broken code style checks in CI for develop branch
      • make sure that OpenFOAM's wmake can find MPFR and GMP if CGAL >= 5.0 (header-only) is used (PR #3366)
      • also consider 'normalised' package name with underscore rather than dash in EasyBuild easyblock (PR #3358)
        • easybuild_framework vs easybuild-framework, enforced by recent versions of setuptools when publishing source tarballs to PyPI
        • workaround for EasyBuild v4.9.2 implemented in easyconfig to ensure that eb --install-latest-eb-release still works (easyconfigs PR #20818)
    • enhancements
      • auto-disable FFTW quad precision on RISC-V (PR #3314)
      • add libjpeg-turbo to include dir for torchvision easyblock (PR #3322 + PR #3353]
      • Add a sanity check for OpenFOAM's wmake to make sure it finds the compiler commands (PR #3328)
      • Always override the compiler version check for CUDA installations (PR #3329)
    • updates
      • ...
    • changes
      • ...
    • new easyblocks
      • ...
    • EasyBuild 5.0
      • ...

  • easyconfigs (merged PRs)

    • over 125 easyconfig PRs were merged since last conf call
    • bug fixes/reports
      • add missing dependencies for funannotate (PR #17690)
      • compile LERC test binary in postinstallcmds to fix sanity check error when RPATH linking is enabled (PR #19386)
      • Fix Highway 1.0.3 on some systems by disabling AVX3_DL (PR #20298)
        • initial fix was for build problem, but then tests failed, so disabled AVX3_DL for Highway 1.0.3
        • similar problems with Highway on AMD Genoa
        • VUB has adopted Highway version 1.0.5 on Genoa + custom modulerc to provide a fake module for older Highway version (1.0.4)
      • add patch to increase compatibility with AVX512 platforms for bwa-mem2 v2.2.1 (PR #20434)
      • Fix detection of CC/CXX compilers for wmake in OpenFOAM v2306 + v2312 (PR #20517)
      • add missing HDF5 dependency in recent Armadillo easyconfigs (>= 11.4.3) (PR #20710)
      • add patches for failing LAPACK tests and RISC-V test segfaults to OpenBLAS 0.3.27 (PR #20745)
      • make sure mummerplot can use gnuplot if available for recent MUMmer (PR #20749)
      • prevent configure script of recent BLAST+ versions from prepending system paths to $PATH (PR #20751)
      • Fix fastparquet v2023.4.0 using CargoPythonBundle easyblock (PR #20775)
        • Mikael is trying to make sure there's a working easyconfig for things that should be using CargoPythonBundle to control which Rust crates are being installed
        • there's other cases (librsvg, libavif) that use cargo behind our backs, which will be trickier to control...
      • add 'pic' flag to IML (PR #20789)
      • add patch to fix build of pdsh 2.34 with Slurm 23.x (PR #20795)
      • add patch to fix build error with numpy with some compilers (PR #20817)
        • patch was added to SciPy-bundle easyconfigs for consistency
        • Alexander's upstream PR was not accepted because numpy.distutils is not actually used anymore in recent numpy versions (>=1.24)
        • numpy 2.0 is out, should be fun...
      • rename unpacked sources for components of EasyBuild v4.9.2, to ensure that '--install-latest-eb-release' works with older EasyBuild versions (PR #20818)
      • fix build of OpenBLAS 0.3.24 on A64FX (PR #20820)
      • add GMP and MPFR as dependencies to OpenFOAM v2306 and v2312 (PR #20841)
      • ...
    • enhancements
      • ...
    • (noteworthy) new software
      • ...
    • noteworthy software updates
      • Python-bundle-PyPI v2024.06 w/ GCCcore/13.3.0 (for upcoming foss/2024a) (PR #20827)
      • SciPy-bundle v2024.05, w/ foss/2024.05 (candidate for foss/2024a) (PR #20830)
    • changes
      • bump Cython to v3.0.8 in Cartopy v0.22.0 easyconfig for foss/2023a toolchain, to avoid dependency version conflict with sckit-learn v1.4.2, which requires Cython >= v3.0.8 (PR #20525)
      • stick to gfbf/2023a as toolchain for ipympl v0.9.3 (PR #20586)
      • Use external PRRTE 3.1.5 in OpenMPI 5.0.3 (will be used in foss/2024a) (PR #20698)
      • remove easyconfigs for CellBender v0.3.1, since this version has been redacted due to a serious bug (PR #20722)
      • move all easyconfigs for libavif to GCCcore toolchain + fix dependencies (PR #20747)
      • comment out PSM2 dependency in recent libfabric easyconfigs, since it pulls in CUDA as dependency (PR #20794)

Open (active) PRs

  • docs (open PRs + issues)

    • ...

  • framework (open PRs + issues)

    • bug fixes
      • Correctly evaluate result for --dep-graph (PR #4554)
    • enhancements
      • ...
    • changes
      • ...
    • EasyBuild 5.0

  • easyblocks (open PRs + issues)

    • bug fixes
      • Patch ctypes for Python installations when filtering LD_LIBRARY_PATH (PR #3352)
    • enhancements
      • enhance CMakeMake easyblock to check whether correct Python installation was picked up by cmake (PR #3233)
      • Set build type for PyTorch explicitely (PR #3332)
      • add testing by default for recent Python versions (3.10+) (PR #3355)
    • updates
      • update CP2K easyblock to use do_regtest.py script for recent versions (WIP) (PR #2984)
    • changes
      • Merge of the ConfigureMake and CMakeMake versions of the easyblock for QuantumESPRESSO (PR #3338)
        • test suite fails because of multiple class definitions in same easyblock file...
    • new
      • ...
    • EasyBuild 5.0
      • rename custom easyblock parameters to adher to naming policy and make things more consistent (PR #3361)

  • easyconfigs (open PRs + issues)

    • bug fixes/reports
      • Checksums have changed for all libxc versions (issue #20417)
        • We'll be able to properly fix this in EasyBuild 5.0 by creating our own tarballs via git clone
      • disable more torchtext tests that fetch remote data in PyTorch-bundle v2.1.2 (PR #20713)
      • add patch to SciPy-bundle 2024.05 that fixes numpy test failures on RISC-V (PR #20847)
    • enhancements
      • ...
    • (noteworthy) new software
      • jax 0.4.25 w/ CUDA (PR #20119)
        • failing tests on V100 (lack of sufficient VRAM) + H100 (too new platform?)
        • we may need to proceed with ignoring failing tests...
      • TensorFlow v2.15.1 w/ CUDA 12.1.1 (PR #20358)
        • Alexander says "should be fine"
      • R-bundle-CRAN w/ 2023b (PR #20756)
      • GROMACS v2024.2 w/ CUDA 12.5.0 (PR #20809)
        • TimeOut test on HPC-UGent systems, or test suite takes forever, not clear why
        • more test reports from GPU systems welcome here...
        • using CUDA 12.5.0 because building with older CUDA version didn't work (due to header file in GCC 13.2)
    • changes
      • ...

2024a common toolchains

  • GCC 13.3 as a base
  • latest version of all components (except GCC, not using 14.1):
    • foss/2024a (candidate toolchain foss/2024.05 merged: PR #20646)
      • GCC 13.3 (20240521) + binutils 2.42 (20240129)
      • OpenMPI 5.0.3 (20240408)
        • PMIx 5.0.2 (20240321), UCX 1.16.0 (20240415), libfabric 1.21.0 (20240329)
      • FlexiBLAS 3.4.3 (20240521) + OpenBLAS 0.3.27 (20240404) + BLIS 1.0 (20240506)
      • FFTW 3.3.10 (20210914)
      • ScaLAPACK 2.2.0 (20220202)
    • intel/2024a (TODO: set up candidate toolchain intel/2024.04?)
      • GCCcore 13.3 + binutils 2.42
      • Intel C/C++/Fortran compilers 2024.1.0 (20240327)
        • some trouble arose when combining C/C++ 2024.1.2 with Fortran 2024.1.0
        • bundle provided by Intel sticks to 2024.1.0 for both
        • we should take a look at the release notes
      • Intel MPI 2021.12.1 (20240424)
      • Intel MKL 2024.1.0 (20240327)

Q&A / others

  • (Jörg) problems with ELPA with Intel compilers on AMD systems
    • see easyblocks issue #3363
    • use of -xHost causes trouble here
    • is get_cpu_features claiming that AMD Rome supports AVX512?!
  • (Adam) --module-only question: 
    Not sure my mic is working - one of my colleagues wants to try out different MNSs. Is using --module-only with --installpath-modules the advisable way to do that?
    • need to make sure that current modules are not in view ($MODULEPATH)
    • according to Åke/Mikael using --module-only works quite well
      • only handful of problems arose
    • we could be doing a better job at testing whether --module-only works (and produces the same module file)
  • (Jörg) can we grab Rust crates for a particular commit
    • see polars easyonfig
Clone this wiki locally