Skip to content
CI

Fix gcc 7 optimization bug yet more #201

Sign in to view logs

Fix gcc 7 optimization bug yet more

Fix gcc 7 optimization bug yet more #201

Workflow file for this run

.github/workflows/ci.yaml at b236fff
# ==================================================================================================
# Header
# ==================================================================================================
name: CI
on: [push, pull_request]
# on:
# pull_request:
# push:
# branches:
# - master
# - misc
# ==================================================================================================
# Jobs
# ==================================================================================================
jobs:
ci:
runs-on: ${{ matrix.os }}
# We have issues with MacOS... because of reasons...
# So we have tests that are currently failing, but still want to keep them around,
# as some of them seem to be issues with Clang itself, and not with our code,
# and are hence expected to be solved with some update on their end at a later point.
# We try to use this here to have the whole workflow succeed even with these jobs failing,
# see https://github.com/actions/toolkit/issues/399
# --> We have deactivated the failing macos jobs for now, so let's deactivate this switch here,
# so that for now, we fail the workflow when a single job fails. That makes it easier to see
# problems down the line.
# continue-on-error: true
# ==========================================================================
# Matrix
# ==========================================================================
strategy:
fail-fast: false
# -------------------------------------------------------
# matrix
# -------------------------------------------------------
# https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#choosing-github-hosted-runners
# https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners
matrix:
# Need dummy entries for the matrix to pick up the variables.
# We later set all other combinations that we want explicitly.
os:
- ubuntu-latest
- macos-latest
compiler:
- gcc
- llvm
# Default to debug build
build_type:
- DEBUG
# - RELEASE
# Default to building with htslib
htslib:
- ON
# - OFF
# -------------------------------------------------------
# include
# -------------------------------------------------------
# Instead of a matrix where we only have some combinations that work, and others that do not,
# we just directly opt for a list of all combinations that we actually want to test.
# Easier to reason about, easier to maintain, easier to extend.
# Newer compiler versions are not available on all Ubuntu, so some are excluded.
# Older GCC versions are also not supported by MacOS 12 any more.
# Currently, llvm-9 causes weird segfauls on Ubuntu, which do not seem to be our fault,
# but rather a bug in their thread forking, according to valgrind.
# Need to investiage further at some point. We used setup-cpp locally to install
# the very same version of llvm-9, and got tons of segfauls for all kind of functions.
# We also currently exclude clang from macos, as it fails with different types of
# linker errors. As this is merely a problem in the setup, we defer solving this for now.
include:
# -------------------------------
# Ubuntu 20.04
# -------------------------------
- os: ubuntu-20.04
compiler: gcc-7
- os: ubuntu-20.04
compiler: gcc-8
- os: ubuntu-20.04
compiler: gcc-9
- os: ubuntu-20.04
compiler: gcc-10
# Setup fails
# - os: ubuntu-20.04
# compiler: gcc-11
# - os: ubuntu-20.04
# compiler: gcc-12
- os: ubuntu-20.04
compiler: gcc-13
# Too new
# - os: ubuntu-20.04
# compiler: gcc-14
- os: ubuntu-20.04
compiler: llvm-6
- os: ubuntu-20.04
compiler: llvm-7
- os: ubuntu-20.04
compiler: llvm-8
# Weird segfault
# - os: ubuntu-20.04
# compiler: llvm-9
- os: ubuntu-20.04
compiler: llvm-10
- os: ubuntu-20.04
compiler: llvm-11
- os: ubuntu-20.04
compiler: llvm-12
- os: ubuntu-20.04
compiler: llvm-13
- os: ubuntu-20.04
compiler: llvm-14
- os: ubuntu-20.04
compiler: llvm-15
# Setup fails
# - os: ubuntu-20.04
# compiler: llvm-16
- os: ubuntu-20.04
compiler: llvm-17
# Setup fails
# - os: ubuntu-20.04
# compiler: llvm-18
# -------------------------------
# Ubuntu 22.04
# -------------------------------
# Too old
# - os: ubuntu-22.04
# compiler: gcc-7
# - os: ubuntu-22.04
# compiler: gcc-8
- os: ubuntu-22.04
compiler: gcc-9
- os: ubuntu-22.04
compiler: gcc-10
- os: ubuntu-22.04
compiler: gcc-11
- os: ubuntu-22.04
compiler: gcc-12
- os: ubuntu-22.04
compiler: gcc-13
# Setup fails
# - os: ubuntu-22.04
# compiler: gcc-14
- os: ubuntu-22.04
compiler: llvm-6
- os: ubuntu-22.04
compiler: llvm-7
- os: ubuntu-22.04
compiler: llvm-8
# Weird segfault
# - os: ubuntu-22.04
# compiler: llvm-9
- os: ubuntu-22.04
compiler: llvm-10
- os: ubuntu-22.04
compiler: llvm-11
- os: ubuntu-22.04
compiler: llvm-12
- os: ubuntu-22.04
compiler: llvm-13
- os: ubuntu-22.04
compiler: llvm-14
- os: ubuntu-22.04
compiler: llvm-15
- os: ubuntu-22.04
compiler: llvm-16
- os: ubuntu-22.04
compiler: llvm-17
# Setup fails
# - os: ubuntu-22.04
# compiler: llvm-18
# -------------------------------
# Ubuntu 24.04
# -------------------------------
# Too old
# - os: ubuntu-24.04
# compiler: gcc-7
# - os: ubuntu-24.04
# compiler: gcc-8
- os: ubuntu-24.04
compiler: gcc-9
- os: ubuntu-24.04
compiler: gcc-10
- os: ubuntu-24.04
compiler: gcc-11
- os: ubuntu-24.04
compiler: gcc-12
- os: ubuntu-24.04
compiler: gcc-13
- os: ubuntu-24.04
compiler: gcc-14
# Setup fails
# - os: ubuntu-24.04
# compiler: llvm-6
# - os: ubuntu-24.04
# compiler: llvm-7
# - os: ubuntu-24.04
# compiler: llvm-8
# - os: ubuntu-24.04
# compiler: llvm-9
# - os: ubuntu-24.04
# compiler: llvm-10
# - os: ubuntu-24.04
# compiler: llvm-11
# - os: ubuntu-24.04
# compiler: llvm-12
# - os: ubuntu-24.04
# compiler: llvm-13
# - os: ubuntu-24.04
# compiler: llvm-14
# - os: ubuntu-24.04
# compiler: llvm-15
# - os: ubuntu-24.04
# compiler: llvm-16
# - os: ubuntu-24.04
# compiler: llvm-17
- os: ubuntu-24.04
compiler: llvm-18
# -------------------------------
# MacOS 12
# -------------------------------
# Setup fails (too old?)
# - os: macos-12
# compiler: gcc-7
# - os: macos-12
# compiler: gcc-8
- os: macos-12
compiler: gcc-9
- os: macos-12
compiler: gcc-10
- os: macos-12
compiler: gcc-11
- os: macos-12
compiler: gcc-12
# Weird compiler errors
# - os: macos-12
# compiler: gcc-13
- os: macos-12
compiler: gcc-14
# Setup fails
# - os: macos-12
# compiler: llvm-6
# - os: macos-12
# compiler: llvm-7
# - os: macos-12
# compiler: llvm-8
# - os: macos-12
# compiler: llvm-9
# - os: macos-12
# compiler: llvm-10
# - os: macos-12
# compiler: llvm-11
- os: macos-12
compiler: llvm-12
- os: macos-12
compiler: llvm-13
- os: macos-12
compiler: llvm-14
- os: macos-12
compiler: llvm-15
# Setup fails
# - os: macos-12
# compiler: llvm-16
# - os: macos-12
# compiler: llvm-17
# - os: macos-12
# compiler: llvm-18
# -------------------------------
# MacOS 13
# -------------------------------
# Setup fails
# - os: macos-13
# compiler: gcc-7
# - os: macos-13
# compiler: gcc-8
- os: macos-13
compiler: gcc-9
- os: macos-13
compiler: gcc-10
- os: macos-13
compiler: gcc-11
- os: macos-13
compiler: gcc-12
# Weird compiler errors
# - os: macos-13
# compiler: gcc-13
- os: macos-13
compiler: gcc-14
# Setup fails
# - os: macos-13
# compiler: llvm-6
# - os: macos-13
# compiler: llvm-7
# - os: macos-13
# compiler: llvm-8
# - os: macos-13
# compiler: llvm-9
# Weird compiler errors
# - os: macos-13
# compiler: llvm-10
# - os: macos-13
# compiler: llvm-11
- os: macos-13
compiler: llvm-12
- os: macos-13
compiler: llvm-13
- os: macos-13
compiler: llvm-14
- os: macos-13
compiler: llvm-15
# Setup fails
# - os: macos-13
# compiler: llvm-16
# - os: macos-13
# compiler: llvm-17
# - os: macos-13
# compiler: llvm-18
# -------------------------------
# MacOS 14
# -------------------------------
# Setup fails for all
# - os: macos-14
# compiler: gcc-7
# - os: macos-14
# compiler: gcc-8
# - os: macos-14
# compiler: gcc-9
# - os: macos-14
# compiler: gcc-10
# - os: macos-14
# compiler: gcc-11
# - os: macos-14
# compiler: gcc-12
# - os: macos-14
# compiler: gcc-13
# - os: macos-14
# compiler: gcc-14
# - os: macos-14
# compiler: llvm-6
# - os: macos-14
# compiler: llvm-7
# - os: macos-14
# compiler: llvm-8
# - os: macos-14
# compiler: llvm-9
# - os: macos-14
# compiler: llvm-10
# - os: macos-14
# compiler: llvm-11
# - os: macos-14
# compiler: llvm-12
# - os: macos-14
# compiler: llvm-13
# - os: macos-14
# compiler: llvm-14
# - os: macos-14
# compiler: llvm-15
# - os: macos-14
# compiler: llvm-16
# - os: macos-14
# compiler: llvm-17
# - os: macos-14
# compiler: llvm-18
# -------------------------------
# Release
# -------------------------------
- os: ubuntu-latest
compiler: gcc
build_type: RELEASE
- os: ubuntu-latest
compiler: llvm
build_type: RELEASE
- os: macos-latest
compiler: gcc
build_type: RELEASE
- os: macos-latest
compiler: llvm
build_type: RELEASE
- os: macos-latest
compiler: apple
build_type: RELEASE
# -------------------------------
# no htslib
# -------------------------------
- os: ubuntu-latest
compiler: gcc
htslib: OFF
- os: ubuntu-latest
compiler: llvm
htslib: OFF
- os: macos-latest
compiler: gcc
htslib: OFF
- os: macos-latest
compiler: llvm
htslib: OFF
- os: macos-latest
compiler: apple
htslib: OFF
# -------------------------------
# AppleClang
# -------------------------------
- os: macos-12
compiler: apple
- os: macos-13
compiler: apple
- os: macos-14
compiler: apple
# ==========================================================================
# Steps
# ==========================================================================
steps:
- name: Checkout
uses: actions/checkout@v4.0.0
# -------------------------------------------------------
# Setup
# -------------------------------------------------------
- name: Setup C++
# Reference: https://github.com/marketplace/actions/setup-cpp-c-c
uses: aminya/setup-cpp@v0.37.0
if: ${{ matrix.compiler != 'apple' && matrix.os != 'macos-latest' }}
with:
compiler: ${{ matrix.compiler }}
cmake: true
make: true
- name: Setup C++ - AppleClang
if: matrix.compiler == 'apple'
run: |
brew install cmake
# -------------------------------------------------------
# Dependencies
# -------------------------------------------------------
- name: Install Dependencies - Ubuntu
if: runner.os == 'linux'
# 1: htslib dependencies, see https://github.com/samtools/htslib/blob/develop/INSTALL
# 2: clang on ubuntu 22 is not properly set up, we need libtinfo5,
# see https://github.com/aminya/setup-cpp/issues/149
# 3: Install gdb for debugging output.
run: |
sudo apt-get install autoconf automake zlib1g-dev libbz2-dev liblzma-dev
sudo apt-get install gdb valgrind
if [[ ${{ matrix.os }} != ubuntu-24.04 ]] ; then
sudo apt-get install libtinfo5
fi
# sudo apt-get remove autoconf
# sudo apt-get install autoconf2.64 liblzma-dev libbz2-dev
- name: Install Dependencies - MacOS
if: runner.os == 'macos'
# The commands below were try and error with the CI...
# - install simple dependencies of htslib, and for (trying to get) OpenMP
# - switch xcode mode, see https://stackoverflow.com/a/67654877
# - output xcode info for debugging, see https://stackoverflow.com/a/73765819
# - we also tried xocde-select from https://mac.r-project.org/openmp/ and
# https://open-box.readthedocs.io/en/latest/installation/openmp_macos.html
# but that did not work
# - previously, we used an htslib version that required autoconf 2.69.
# we switched now, but keep this here for reference if needed later
run: |
brew install autoconf automake libdeflate libomp
if [[ ${{ matrix.os }} != macos-14 ]] && [[ ${{ matrix.os }} != macos-latest ]] ; then
echo "Installing debuggers"
brew install gdb
# brew install valgrind
fi
if [[ "${{ matrix.compiler }}" == gcc* ]]; then
echo "Patching xcode gcc issues"
sudo xcode-select --switch /Library/Developer/CommandLineTools
fi
# xcode-select -p
# xcode-select -p
# pkgutil --pkg-info=com.apple.pkg.CLTools_Executables
# /usr/bin/xcodebuild -version
# xcode-select --install
# brew uninstall --ignore-dependencies autoconf
# brew install autoconf@2.69
# echo 'export PATH="/usr/local/opt/autoconf@2.69/bin:$PATH"' >> /Users/runner/.bash_profile
# -------------------------------------------------------
# Configure
# -------------------------------------------------------
- name: Configure
run: |
export GENESIS_DEBUG=ON
export GENESIS_USE_HTSLIB=ON
export GENESIS_USE_OPENMP=ON
# Take matrix includes into account
if [[ "${{ matrix.build_type }}" == "RELEASE" ]]; then export GENESIS_DEBUG=OFF; fi
if [[ "${{ matrix.htslib }}" == "OFF" ]]; then export GENESIS_USE_HTSLIB=OFF; fi
# Deactivate OpenMP on MacOS for now. Causes way too much trouble.
if [[ "${RUNNER_OS}" == "macOS" ]]; then export GENESIS_USE_OPENMP=OFF; fi
# Need to set the xcode patch here as well, as job steps don't keep status.
if [[ "${RUNNER_OS}" == "macOS" ]] && [[ "${{ matrix.compiler }}" == gcc* ]]; then
echo "Patching xcode gcc issues"
sudo xcode-select --switch /Library/Developer/CommandLineTools
fi
cmake -S . -B ./build -DGENESIS_USE_HTSLIB=${GENESIS_USE_HTSLIB} -DGENESIS_USE_OPENMP=${GENESIS_USE_OPENMP}
# - name: MacOS CMake debug
# if: always() && runner.os == 'macos'
# run: |
# CMAKE_OUT="/Users/runner/work/genesis/genesis/build/CMakeFiles/CMakeOutput.log"
# CMAKE_ERR="/Users/runner/work/genesis/genesis/build/CMakeFiles/CMakeError.log"
# if [[ -f ${CMAKE_OUT} ]]; then
# cat ${CMAKE_OUT}
# fi
# if [[ -f ${CMAKE_ERR} ]]; then
# cat ${CMAKE_ERR}
# fi
# -------------------------------------------------------
# Build
# -------------------------------------------------------
- name: Build
run: |
# Weird workarounds. On some CI builds, this is needed somehow for LLVM under Ubuntu.
if [[ "${RUNNER_OS}" != "macOS" ]] && [[ ! -f /usr/bin/gmake ]]; then
sudo ln -s /usr/bin/make /usr/bin/gmake
fi
# Need to set the xcode patch here as well, as job steps don't keep status.
if [[ "${RUNNER_OS}" == "macOS" ]] && [[ "${{ matrix.compiler }}" == gcc* ]]; then
echo "Patching xcode gcc issues"
sudo xcode-select --switch /Library/Developer/CommandLineTools
fi
cmake --build ./build -- -j 4
# -------------------------------------------------------
# Test
# -------------------------------------------------------
- name: Tests
id: test
if: >
success() &&
runner.os != 'windows' &&
matrix.build_type != 'RELEASE'
run: |
./bin/test/genesis_tests
- name: Test - gdb
# If the regular test run failed, we repeat with gdb, to backtrace the stack.
if: ${{ failure() && steps.test.conclusion == 'failure' }}
run: |
# We are not using the `./test/run.sh gdb` here, to make it easier for MacOS.
# gdb on Mac needs to be code signed for executing without admin rights...
# See https://sourceware.org/gdb/wiki/PermissionsDarwin for the details.
if [[ ${{ matrix.os }} == macos* ]] ; then
lldb --batch -o "run" -o "thread backtrace all" -o "quit" -- ./bin/test/genesis_tests
else
gdb -ex "set confirm off" -iex "set pagination off" -ex "run" -ex "thread apply all bt" -ex "quit" --args ./bin/test/genesis_tests
fi
- name: Test - valgrind
# If the regular test run failed, we repeat with valgrind, to find seg fauls and mem issues.
if: ${{ failure() && steps.test.conclusion == 'failure' }}
run: |
valgrind ./bin/test/genesis_tests
# -------------------------------------------------------
# Logs
# -------------------------------------------------------
- name: Upload Logs
if: always()
uses: actions/upload-artifact@v4
with:
name: logs-${{ matrix.os }}-${{ matrix.compiler }}-${{ matrix.build_type }}-htslib-${{ matrix.htslib }}
path: |
./build/**/*.log
./test/out