Install OpenFOAM 2.4.0 on TACC Lonestar 5 Supercomputer System

OpenFoam TACC Lonestar LS5

OpenFOAM

OpenFOAM is one of the most widely used CFD open source toolbox both in academic and industrial applications at this time. The brief introduction of OpenFOAM can be found in OpenFOAM - Wikipedia and OpenFOAM®.

Build Steps

Step 1

Download and unzip the OpenFOAM source code. TACC's $WORK filesystem is used here.

$ cdw
$ mkdir OpenFOAM
$ cd OpenFOAM
$ wget "http://downloads.sourceforge.net/foam/OpenFOAM-2.4.0.tgz?use_mirror=mesh" -O OpenFOAM-2.4.0.tgz
$ wget "http://downloads.sourceforge.net/foam/ThirdParty-2.4.0.tgz?use_mirror=mesh" -O ThirdParty-2.4.0.tgz
$ tar -xzf OpenFOAM-2.4.0.tgz
$ tar -xzf ThirdParty-2.4.0.tgz

Step 2

Build CGAL-4.6 that is needed in our analytical PLIC method.

• Edit ThirdParty-2.4.0/makeCGAL at line 131:

from

BOOST_ARCH_PATH=$installBASE/$boostPACKAGE

BOOST_ARCH_PATH=$TACC_BOOST_DIR

• Edit OpenFOAM-2.4.0/etc/config/CGAL.sh at line 35:

from

export BOOST_ARCH_PATH=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER/$boost_version

to

export BOOST_ARCH_PATH=$TACC_BOOST_DIR

• Go to ThirdParty-2.4.0 directory and load the boost install in LS5:

$ cd ThirdParty-2.4.0
$ module load boost

• Build cmake-2.8.12.1:

$ ./makeCmake &> build_log_cmake

• Build CGAL-4.6:

$ PATH="$WORK/OpenFOAM/ThirdParty-2.4.0/platforms/linux64Icc/cmake-2.8.12.1/bin/:${PATH}" ./makeCGAL &> build_log_cgal

Step 3

Edit the OpenFOAM-2.4.0/etc/bashrc file and change the following environment variables defined in this file:

• line 45

from

foamInstall=$HOME/$WM_PROJECT

to

foamInstall=$WORK/$WM_PROJECT

• line 66

from

export WM_COMPILER=Gcc

to

export WM_COMPILER=Icc

• line 84

from

export WM_MPLIB=OPENMPI

to

export WM_MPLIB=MV2MPI

• line 104

from

foamOldDirs="$FOAM_INST_DIR $WM_PROJECT_SITE $HOME/$WM_PROJECT/$USER"

to

foamOldDirs="$FOAM_INST_DIR $WM_PROJECT_SITE $WORK/$WM_PROJECT/$USER"

• line 132

from

export WM_PROJECT_USER_DIR=$HOME/$WM_PROJECT/$USER-$WM_PROJECT_VERSION

to

export WM_PROJECT_USER_DIR=$WORK/$WM_PROJECT/$USER-$WM_PROJECT_VERSION

Step 4

Edit the OpenFOAM-2.4.0/etc/config/settings.sh file and change the following environment variables defined in this file:

• Change the default compilers on lines 89-90:

from

export WM_CC='gcc'
export WM_CXX='g++'

to

export WM_CC='mpicc'
export WM_CXX='mpicxx'

• Change the compiler and linker flags on lines 91-93:

from

export WM_CFLAGS='-m64 -fPIC'
export WM_CXXFLAGS='-m64 -fPIC'
export WM_LDFLAGS='-m64'

to

export WM_CFLAGS='-O3 -fPIC'
export WM_CXXFLAGS='-O3 -fPIC'
export WM_LDFLAGS='-O3'

• Add the MV2MPI case we defined in Step 2 after line 364:

MV2MPI)
    export FOAM_MPI=mpi
    export MPI_ARCH_PATH="${MPICH_HOME}"
    ;;

Step 5

• Edit the OpenFOAM-2.4.0/wmake/rules/linux64Icc/c file and change line 5:

from

cc = icc

to

cc = mpicc

Step 6

• Edit the OpenFOAM-2.4.0/wmake/rules/linux64Icc/c++ file and change line 5:

from

CC = icpc -std=c++0x -fp-trap=common -fp-model precise

to

CC = mpicxx -std=c++0x -fp-trap=common -fp-model precise

Step 7

Set and export the following environment variables:

$ export WM_CC=mpicc
$ export WM_CXX=mpicxx

Step 8

Source the etc/bashrc file edited in Step 2 and update the shell environment:

$ cdw
$ cd OpenFOAM/OpenFOAM-2.4.0
$ . etc/bashrc
$ wmSET $FOAM_SETTINGS

Step 9

Start the build and take a 4-5 hour break:

$ ./Allwmake &> build_log
$ ./Allwmake &> build_log_summary

Step 10

To check if everything went well:

$ icoFoam -help

Step 11

Add the scripts below to $HOME/.bashrc:

source $WORK/OpenFOAM/OpenFOAM-2.4.0/etc/bashrc
export LD_LIBRARY_PATH=/work/04032/ddai/lonestar/OpenFOAM/ThirdParty-2.4.0/platforms/linux64Icc/CGAL-4.6/lib:$LD_LIBRARY_PATH
module load boost

the dir /work/04032/ddai/ should be replaced by your own work one in LS5.
Then reload the bash file:

$ . ~/.bashrc

Dezhi, Dai [Ph.D. student]
Mechanical & Aerospace Engineering Department
UT Arlington
Email: dezhi.dai@mavs.uta.edu