Use wider-compatible LLVM-OpenMP, bundle libgfortran dylibs for mac…

…OS wheels (#4092)

* Set `MACOSX_DEPLOYMENT_TARGET` for prerequisites

* Remove previously set environment variables

* Remove Homebrew libomp installations

* Ignore `pybamm-requires` OpenMP downloads

* Rework docs to add OpenMP instructions

* Download LLVM-OpenMP on macOS

* Remove unused compiler argument

* Fix rpath for libomp dylib at wheel repair

* Fix a failing link

* Revert "Fix rpath for libomp dylib at wheel repair"

This reverts commit 6fb8ef0.

* Fix RPATHs for SuiteSparse and libomp

* 11.1 for arm64 macOS and 11.0 for amd64 macOS

* Set up Fortran compiler via GHA

* Downgrade to gcc 12

* Try fix for macOS amd64 wheels first

* Build on macos-13 image instead

* Debug further, try building against 11.0

* And now build for macOS arm64

* Try building arm64 against 11.0

* Rename wheel from 11_1 to 11_0

* Cleanup and fix up commands

* Rename wheel in dest dir instead

* Cleanup plus references to SciPy license

.github/workflows/publish_pypi.yml

@@ -99,7 +99,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        os: [ubuntu-latest, macos-12]
+        os: [ubuntu-latest, macos-13]
     steps:
       - uses: actions/checkout@v4
         name: Check out PyBaMM repository
@@ -112,15 +112,6 @@ jobs:
       - name: Clone pybind11 repo (no history)
         run: git clone --depth 1 --branch v2.11.1 https://github.com/pybind/pybind11.git
 
-      # sometimes gfortran cannot be found, so reinstall gcc just to be sure
-      - name: Install SuiteSparse and SUNDIALS on macOS
-        if: matrix.os == 'macos-12'
-        run: |
-          brew install graphviz libomp
-          brew reinstall gcc
-          python -m pip install cmake wget
-          python scripts/install_KLU_Sundials.py
-
       - name: Build wheels on Linux
         run: pipx run cibuildwheel --output-dir wheelhouse
         if: matrix.os == 'ubuntu-latest'
@@ -134,11 +125,50 @@ jobs:
           CIBW_TEST_COMMAND: python -c "import pybamm; pybamm.IDAKLUSolver()"
 
       - name: Build wheels on macOS amd64
-        if: matrix.os == 'macos-12'
+        if: matrix.os == 'macos-13'
         run: pipx run cibuildwheel --output-dir wheelhouse
         env:
-          CIBW_BEFORE_BUILD_MACOS: >
-            python -m pip install --upgrade cmake casadi setuptools wheel
+          MACOSX_DEPLOYMENT_TARGET: 11.0
+          # Sourced from
+          # License: BSD-3-Clause
+          # https://github.com/scipy/scipy/blob/f2d4775e7762fad984f8f0acd8227c725ff21630/tools/wheels/cibw_before_build_macos.sh#L23-L49
+          CIBW_BEFORE_ALL_MACOS: |
+            set -e -x
+
+            # download gfortran with proper macos minimum version (11.0)
+            curl -L https://github.com/isuruf/gcc/releases/download/gcc-11.3.0-2/gfortran-darwin-x86_64-native.tar.gz -o gfortran.tar.gz
+
+            GFORTRAN_SHA256=$(shasum --algorithm 256 gfortran.tar.gz)
+            KNOWN_SHA256="981367dd0ad4335613e91bbee453d60b6669f5d7e976d18c7bdb7f1966f26ae4  gfortran.tar.gz"
+            if [ "$GFORTRAN_SHA256" != "$KNOWN_SHA256" ]; then
+              echo "SHA256 mismatch for gfortran.tar.gz"
+              echo "expected: $KNOWN_SHA256"
+              echo "got: $GFORTRAN_SHA256"
+              exit 1
+            fi
+
+            mkdir -p gfortran_installed/
+            tar -xv -C gfortran_installed/ -f gfortran.tar.gz
+
+            export FC=$(pwd)/gfortran_installed/gfortran-darwin-x86_64-native/bin/gfortran
+            export PATH=$(pwd)/gfortran_installed/gfortran-darwin-x86_64-native/bin:$PATH
+
+            # link libgfortran.5.dylib, libgfortran.dylib, libquadmath.0.dylib, libquadmath.dylib, libgcc_s.1.dylib, libgcc_s.1.1.dylib
+            # and place them in $HOME/.local/lib, and then change rpath
+            # to $HOME/.local/lib
+
+            mkdir -p $HOME/.local/lib
+            lib_dir=$(pwd)/gfortran_installed/gfortran-darwin-x86_64-native/lib
+            for lib in libgfortran.5.dylib libgfortran.dylib libquadmath.0.dylib libquadmath.dylib libgcc_s.1.dylib libgcc_s.1.1.dylib; do
+              cp $lib_dir/$lib $HOME/.local/lib/
+              install_name_tool -id $HOME/.local/lib/$lib $HOME/.local/lib/$lib
+            done
+
+            export SDKROOT=${SDKROOT:-$(xcrun --show-sdk-path)}
+
+            python -m pip install cmake wget
+            python scripts/install_KLU_Sundials.py
+          CIBW_BEFORE_BUILD_MACOS: python -m pip install --upgrade pip cmake casadi setuptools wheel delocate
           CIBW_REPAIR_WHEEL_COMMAND_MACOS: delocate-listdeps {wheel} && delocate-wheel -v -w {dest_dir} {wheel}
           CIBW_TEST_COMMAND: python -c "import pybamm; pybamm.IDAKLUSolver()"
 
@@ -162,18 +192,57 @@ jobs:
       - name: Clone pybind11 repo (no history)
         run: git clone --depth 1 --branch v2.11.1 https://github.com/pybind/pybind11.git
 
-      - name: Install SuiteSparse and SUNDIALS on macOS
-        run: |
-          brew install graphviz libomp
-          brew reinstall gcc
-          python -m pip install cmake pipx
-          python scripts/install_KLU_Sundials.py
-
       - name: Build wheels on macOS arm64
-        run: python -m pipx run cibuildwheel --output-dir wheelhouse
+        run: |
+          python -m pip install cibuildwheel
+          python -m cibuildwheel --output-dir wheelhouse
         env:
-          CIBW_BEFORE_BUILD: python -m pip install cmake casadi setuptools wheel delocate
-          CIBW_REPAIR_WHEEL_COMMAND: delocate-listdeps {wheel} && delocate-wheel -v -w {dest_dir} {wheel}
+          MACOSX_DEPLOYMENT_TARGET: 11.0
+          # Sourced from
+          # License: BSD-3-Clause
+          # https://github.com/scipy/scipy/blob/f2d4775e7762fad984f8f0acd8227c725ff21630/tools/wheels/cibw_before_build_macos.sh#L23-L49
+          CIBW_BEFORE_ALL_MACOS: |
+            set -e -x
+
+            # download gfortran with proper macos minimum version (11.0)
+            curl -L https://github.com/isuruf/gcc/releases/download/gcc-11.3.0-2/gfortran-darwin-arm64-native.tar.gz -o gfortran.tar.gz
+
+            GFORTRAN_SHA256=$(shasum --algorithm 256 gfortran.tar.gz)
+            KNOWN_SHA256="84364eee32ba843d883fb8124867e2bf61a0cd73b6416d9897ceff7b85a24604  gfortran.tar.gz"
+            if [ "$GFORTRAN_SHA256" != "$KNOWN_SHA256" ]; then
+              echo "SHA256 mismatch for gfortran.tar.gz"
+              echo "expected: $KNOWN_SHA256"
+              echo "got: $GFORTRAN_SHA256"
+              exit 1
+            fi
+
+            mkdir -p gfortran_installed/
+            tar -xv -C gfortran_installed/ -f gfortran.tar.gz
+
+            export FC=$(pwd)/gfortran_installed/gfortran-darwin-arm64-native/bin/gfortran
+            export PATH=$(pwd)/gfortran_installed/gfortran-darwin-arm64-native/bin:$PATH
+
+            # link libgfortran.5.dylib, libgfortran.dylib, libquadmath.0.dylib, libquadmath.dylib, libgcc_s.1.1.dylib
+            # and place them in $HOME/.local/lib, and then change rpath
+            # note: libgcc_s.1.dylib not present for arm64, skip for now
+            # to $HOME/.local/lib
+
+            mkdir -p $HOME/.local/lib
+            lib_dir=$(pwd)/gfortran_installed/gfortran-darwin-arm64-native/lib
+            for lib in libgfortran.5.dylib libgfortran.dylib libquadmath.0.dylib libquadmath.dylib libgcc_s.1.1.dylib; do
+              cp $lib_dir/$lib $HOME/.local/lib/
+              install_name_tool -id $HOME/.local/lib/$lib $HOME/.local/lib/$lib
+            done
+
+            export SDKROOT=${SDKROOT:-$(xcrun --show-sdk-path)}
+
+            python -m pip install cmake wget
+            python scripts/install_KLU_Sundials.py
+          CIBW_BEFORE_BUILD_MACOS: python -m pip install --upgrade pip cmake casadi setuptools wheel delocate
+          # Use higher macOS target for now: https://github.com/casadi/casadi/issues/3698
+          CIBW_REPAIR_WHEEL_COMMAND_MACOS: |
+            delocate-listdeps {wheel} && delocate-wheel -v -w {dest_dir} {wheel} --require-target-macos-version 11.1
+            for file in {dest_dir}/*.whl; do mv "$file" "${file//macosx_11_1/macosx_11_0}"; done
           CIBW_TEST_COMMAND: python -c "import pybamm; pybamm.IDAKLUSolver()"
 
       - name: Upload wheels for macOS arm64

.github/workflows/run_periodic_tests.yml

@@ -68,7 +68,7 @@ jobs:
         if: matrix.os == 'macos-12' || matrix.os == 'macos-14'
         run: |
           brew analytics off
-          brew install graphviz libomp
+          brew install graphviz
           brew reinstall gcc
 
       - name: Install Windows system dependencies

.github/workflows/test_on_push.yml

@@ -74,7 +74,7 @@ jobs:
         # sometimes gfortran cannot be found, so reinstall gcc just to be sure
         run: |
           brew analytics off
-          brew install graphviz libomp
+          brew install graphviz
           brew reinstall gcc
 
       - name: Install Windows system dependencies
@@ -209,7 +209,7 @@ jobs:
         # sometimes gfortran cannot be found, so reinstall gcc just to be sure
         run: |
           brew analytics off
-          brew install graphviz libomp
+          brew install graphviz
           brew reinstall gcc
 
       - name: Install Windows system dependencies

.gitignore

@@ -137,3 +137,7 @@ results/
 
 # tests
 test_callback.log
+
+# openmp downloads
+install_KLU_Sundials/openmp-*
+install_KLU_Sundials/usr/

.lycheeignore

@@ -1,6 +1,7 @@
 # a list of links/files to be ignored by lychee link checker (see workflow file)
 https://github.com/DrTimothyAldenDavis/SuiteSparse/archive/v%7BSUITESPARSE_VERSION%7D.tar.gz
 https://github.com/LLNL/sundials/releases/download/v%7BSUNDIALS_VERSION%7D/sundials-%7BSUNDIALS_VERSION%7D.tar.gz
+https://mac.r-project.org/openmp/openmp-%7BOPENMP_VERSION%7D-darwin20-Release.tar.gz
 
 # Errors in docs/source/user_guide/getting_started.md
 file:///home/runner/work/PyBaMM/PyBaMM/docs/source/user_guide/api_docs

docs/source/user_guide/installation/install-from-source.rst

@@ -43,11 +43,11 @@ You can install the above with
 
 	Where ``X`` is the version sub-number.
 
-.. tab:: MacOS
+.. tab:: macOS
 
 	.. code:: bash
 
-		brew install python openblas gcc gfortran graphviz libomp cmake pandoc
+		brew install python openblas gcc gfortran graphviz cmake pandoc
 
 .. note::
 
@@ -82,8 +82,9 @@ If you are running windows, you can simply skip this section and jump to :ref:`p
 	  # in the PyBaMM/ directory
 	  nox -s pybamm-requires
 
-This will download, compile and install the SuiteSparse and SUNDIALS libraries.
-Both libraries are installed in ``~/.local``.
+This will download, compile and install the SuiteSparse and SUNDIALS (with OpenMP) libraries
+and the ``pybind11`` headers.
+SuiteSparse and SUNDIALS are installed in ``~/.local`` by default.
 
 For users requiring more control over the installation process, the ``pybamm-requires`` session supports additional command-line arguments:
 
@@ -110,7 +111,7 @@ If you'd rather do things yourself,
 
 1. Make sure you have CMake installed
 2. Compile and install SuiteSparse (PyBaMM only requires the ``KLU`` component).
-3. Compile and install SUNDIALS.
+3. Compile and install SUNDIALS with `OpenMP support <https://mac.r-project.org/openmp/>`_.
 4. Clone the pybind11 repository in the ``PyBaMM/`` directory (make sure the directory is named ``pybind11``).
 
 
@@ -315,12 +316,12 @@ source files to your current directory.
 ``ValueError: Integrator name ida does not exist``, or
 ``ValueError: Integrator name cvode does not exist``.
 
-**Solution:** This could mean that you have not installed
-``scikits.odes`` correctly, check the instructions given above and make
+**Solution:** This could mean that you have not linked to
+SUNDIALS correctly, check the instructions given above and make
 sure each command was successful.
 
 One possibility is that you have not set your ``LD_LIBRARY_PATH`` to
-point to the sundials library, type ``echo $LD_LIBRARY_PATH`` and make
+point to the SUNDIALS library, type ``echo $LD_LIBRARY_PATH`` and make
 sure one of the directories printed out corresponds to where the
 SUNDIALS libraries are located.