Skip to content

hpsim/OGL

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Requirements | Compilation | Usage | Known Limitations | Citing | Example |


OGL is a wrapper for ginkgo solvers and preconditioners to provide GPGPU capabilities to OpenFOAM.

Requirements

OGL has the following requirements

  • cmake 3.13+
  • OpenFOAM 6+ or v2106
  • Ginkgo 1.5.0+ (It is recommended to install via OGL)
  • C++17 compliant compiler (gcc or clang)

See also ginkgo's documentation for additional requirements.

build OF versions Documentation

For cuda builds cuda version 12 is recommended. For older cuda versions automatic device detection might fail, in this case please set the cuda architecture manually via -DOGL_CUDA_ARCHITECTURES.

Compilation

OGL can be build using cmake following the standard cmake procedure.

mkdir build && cd build && ccmake ..

By default OGL will fetch and build ginkgo, to specify which backend should be build you can use the following cmake flags -DGINKGO_BUILD_CUDA, -DGINKGO_BUILD_OMP, or -DGINKGO_BUILD_HIP. For example to build OGL with CUDA and OMP support use

cmake -DGINKGO_BUILD_CUDA=ON -DGINKGO_BUILD_OMP=ON ..

Then, compile and install by

make -j && make install

CMakePresets and Ninja builds

If you have Ninja installed on your system we recommend to use ninja over gnu make for better compilation times. We also provide a list of Cmake presets which can be used a recent version of Cmake (>3.20). To display available presets use:

cmake --list-preset

The following example shows how to execute a build and install on a cuda system.

cmake --preset ninja-cuda-release
cmake --build --preset ninja-cpuonly-release  --target install

After a successful build install make sure that the system/controlDict includes the libOGL.so or libOGL.dyLib file:

libs ("libOGL.so");

Usage

OGL solver support the same syntax as the default OpenFOAM solver. Thus, to use Ginkgo's CG solver you can simply replace PCG by GKOCG. In order to run either with CUDA, HIP, or OMP support set the executor keyword to cuda, hip, or omp in the system/fvSolution dictionary.

Argument Default Description
updateRHS true whether to copy the system matrix to device on every solver call
updateInitGuess false whether to copy the initial guess to device on every solver call
export false write the complete system to disk
verbose 0 print out extra info
executor reference the executor where to solve the system matrix, other options are omp, cuda
adaptMinIter true based on the previous solution set minIter to be relaxationFactor*previousIters
relaxationFactor 0.8 use relaxationFactor*previousIters as new minIters
scaling 1.0 Scale the complete system by the scaling factor
forceHostBuffer false whether to copy to host before MPI calls

Supported Solver

Currently, the following solver are supported

  • CG
  • BiCGStab
  • GMRES
  • IR (experimental)

additionally, the following preconditioners are available

Supported Preconditioner

  • BJ, block Jacobi
  • ISAI, Incomplete Sparse Approximate Inverses,
  • ILU, incomplete LU (experimental)
  • IC, incomplete Cholesky (experimental)
  • Multigrid, algebraic multigrid (experimental)

The following optional arguments are supported to modify the preconditioner. Note some preconditioners like IC or (SPD) ISAI require positive values on the system matrix diagonal, thus in case of the pressure equation the complete system needs to be scaled by a factor of -1.0.

Argument Default Preconditioner
SkipSorting True all
Caching 1 all
MaxBlockSize 1 block Jacobi
SparsityPower 1 ISAI
MaxLevels 9 Multigrid
MinCoarseRows 10 Multigrid
ZeroGuess True Multigrid

Supported Matrix Formats (Experimental)

Currently, the following matrix formats can be set by matrixFormat

  • Coo
  • Csr
  • Ell (experimental)
  • Hybrid (experimental)

Known Limitations and Troubleshooting

  • Currently, only basic cyclic boundary conditions are supported, no AMI boundary conditions are supported. Block-coupled matrices are not supported.

  • If you are compiling against a double precision label version of OpenFOAM make sure to set -DOGL_DP_LABELS=ON otherwise errors of the following type can occur undefined symbol: _ZN4Foam10dictionary3addERKNS_7keyTypeEib

Citing

When using OGL please cite the main Ginkgo paper describing Ginkgo's purpose, design and interface, which is available through the following reference:

@Article{Olenik2024,
author={Olenik, Gregor and Koch, Marcel and Boutanios, Ziad and Anzt, Hartwig},
title={Towards a platform-portable linear algebra backend for OpenFOAM},
journal={Meccanica},
year={2024},
month={Jun},
day={12},
issn={1572-9648},
doi={10.1007/s11012-024-01806-1},
url={https://doi.org/10.1007/s11012-024-01806-1}
}

@article{Anzt_Ginkgo_A_Modern_2022,
author = {Anzt, Hartwig and Cojean, Terry and Flegar, Goran and Göbel, Fritz and Grützmacher, Thomas and Nayak, Pratik and Ribizel, Tobias and Tsai, Yuhsiang and Quintana-Ortí, Enrique S.},
doi = {10.1145/3480935},
journal = {ACM Transactions on Mathematical Software},
month = mar,
number = {1},
pages = {1--33},
title = {{Ginkgo: A Modern Linear Operator Algebra Framework for High Performance Computing}},
volume = {48},
year = {2022}
}

Example

Below an animation of a coarse 2D simulation of a karman vortex street performed on a MI100 can be seen. Here both the momentum and Poisson equation are offloaded to the GPU. karman