FCLC

I'll be speaking during the OneAPI dev summit tomorrow, specifically the panel discussion on accelerated computing.

Partially as a meme/Sanity check the challenge of this afternoon:

Can I got from no installs to a GPU accelerated simulation of OpenFOAM using an Arc a770 on Linux?

Plan is #OpenFoam 2212, #petsc 3.19, #OneAPI 2023.1 and #mesa 23 (in case I need to fall back to #OpenCL)

Jun 12, 2023 at 4:37pmWeb
Show threadFCLCJun 12, 2023

Bit of a meme, but this is a little like "Aurora/Sunspot" at home.

Single node with the host CPU being Goldencove #AVX512 (same as SPR) and using an Intel GPU (specifically Arc A770) and reading off of flash.

Show threadKenneth HosteJun 12, 2023
@fclc Keen to hear if you manage!
Show threadFCLCJun 12, 2023
@boegel so far already pivoted to OpenCL (PETSC does have configure flags for SYCL, but as of now is marked as unsupported. Also Intels MPI seem to target iFort and ICC, not the newer IFX/ICX compilers)
Show threadTom HaywardJun 12, 2023
@fclc @boegel if memory serves, it's "mpiifort -fc=" or mpiifort + env variables.
I looked into it when I wanted to check if EasyBuild was going to have a bug when enabling ifx for MPI codes (all was fine, since it was already setting the appropriate env variable).
My makefile, on the other hand, needed patching for a multi-word $FC when using the first approach listed above.
Show threadFCLCJun 12, 2023
@hattom @boegel Yeah, also hit that roadblock and unfortunately the typical work around of passing the MPI compiler followed by the LLVM compiler as a CC argument didn't work out.
specifically
--with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-debugging=0 COPTFLAGS='-cc=icc' CXXOPTFLAGS='-O3 -march=sapphirerapids -mno-amx-tile -mno-amx-int8 -mno-amx-bf16 -cxx=icpx' FOPTFLAGS='-O3 -march=sapphirerapids -mno-amx-tile -mno-amx-int8 -mno-amx-bf16 -fc=ifx ' --with-blaslapack-dir=$MKLROOT
Show threadFCLCJun 12, 2023

@hattom @boegel Did get petsc going and now doing FOAM (standard WMAKE induced headaches beginning)

I "settled" on the classic compilers and AVX2 since fighting for LLVM and AVX512 wasn't the main goal 😅

Show threadFCLCJun 12, 2023
@boegel @hattom I’ll try after foam builds (already 30 minutes into compiling) but I wonder if passing —with-cc=“mpiicc -cc=‘icx’” (and relevant cpp and fort) would do the trick
Show threadFCLCJun 12, 2023
@boegel @hattom Foam finished but it doesn't seem like it worked, will have to look at this again at some point
Show threadJed BrownJun 12, 2023
@fclc @boegel use --download-kokkos --download-kokkos-kernels. Pretty much everything works, though some KK implementations are slower than they should be. Feel free to ping me or Junchao on CNDA slack.
Show threadFCLCJun 13, 2023

@jedbrown @boegel Moist of the way through getting OpenCL working already, just need to get Petsc4foam to load correctly 😅

After the dev summit will probably take a look