Skip to content

Tips on setting up deal.ii and petsc

Tips on setting up deal.ii and petsc published on No Comments on Tips on setting up deal.ii and petsc

Install deal.ii along is quite strait forward. We just need to download the latest package from http://dealii.org/download.html, and compile it according to the documents https://www.dealii.org/developer/readme.html

Generally, this can be down without any problem, and most of the cases in the example folder can be run correctly. However, when comes to the MPI, problem will occur. deall.ii requires to work together with petsc, but there are always some problems. Now I just record some tips which may help the successfully setup of both softwares. These procedures are just concepts and can only work for my case. The directory need to be modified in other system.

1. Deactive the environment setup of intel compiler. This will be investigated in the future.

2. Compile and install gcc-5.2.0, the version 4.8.4 used in ubuntu seems cannot work well with deal.ii+petsc+MPI+cxx11. Besides, in the company's computer, the gcc version is quite old, and we need to build ourselves gcc. Ref: https://gcc.gnu.org/wiki/InstallingGCC

3. Set environment variable,
export PATH=$HOME/gcc-5.2.0/bin:$PATH
export LD_LIBRARY_PATH=$HOME/gcc-5.2.0/lib64

4. In the company's computer, there is no cmake. So we need to compile it, and add it to the $PATH. Also download the boost if required. Boost do not need to install, just set env.

5. Download and install petsc. The most easy way to configure it is
./configure --download-fblaslapack --download-mpich
make all test

6. So we just use this mpich as our default mpi, set env
export PETSC_DIR=$HOME/petsc-3.6.1
export PETSC_ARCH=arch-linux2-c-debug
export PATH=$HOME/petsc-3.6.1/arch-linux2-c-debug/bin:$PATH   (this is for downloaded mpich)

7. In order to run MPI program in deal.ii, we also need a library call METIS. Download and compile it.

8. Download and configure deal.ii
cmake -DCMAKE_INSTALL_PREFIX=$HOME/dealii-8.3.0 -DDEAL_II_WITH_METIS=ON -DMETIS_DIR=$HOME/metis-5.1.0 ../
need to be notice whether the following terms on turn to ON
i.e. C++11, MPI, METIS, PETSC
then, make install(If problems occurs or MPI and PETSC cannot be configured correctly, try "export CC=mpicc CXX=mpicxx"

9. If deal.ii is compiled in the Virtualbox, DON'T compile in parallel

10. If ubuntu 14.04 (Trusty) is used, strange errors may occur during compiling. Ref: https://groups.google.com/forum/#!msg/dealii/1oM3cqR0pWU/zOlMHspD9n8J
The method to fix:
sudo add-apt-repository ppa:bzindovic/suitesparse-bugfix-1319687
sudo apt-get update
sudo apt-get upgrade

--------------------------------------------------------------------------------

2015.10.07

If p4est and trilinos are required, refer to the following steps

11. Download the source code of p4est from: http://www.p4est.org/. Download the source package, and don't clone the git source.

12. Install p4est
$ ./configure --prefix=$HOME/p4est-1.1 --enable-mpi BLAS_LIBS=$MYHOME/petsc-3.6.1/arch-linux2-c-debug/lib/libfblas.a LAPACK_LIBS=$MYHOME/petsc-3.6.1/arch-linux2-c-debug/lib/libflapack.a
$ make -j2 && make install

13. Install netcdf, matio, glm(what's this? Required by trilinos)
$ sudo apt-get install libnetcdf-dev libmatio-dev libglm-dev

14. Unzip the trilinon source package, refer to the INSTALL file
$ mkdir build && cd build
$ cmake -DTPL_ENABLE_MPI=ON -DTrilinos_ENABLE_ALL_PACKAGES=ON \
-DCMAKE_INSTALL_PREFIX=<path to install Trilinos into> ../
$ make -j<n> install

15. Alternatively, maybe it is a better ideal to follow the guild on dealii website as https://www.dealii.org/developer/external-libs/trilinos.html. There are also many other guild regarding to the lib installation and worth reading.
cmake \
-D Trilinos_ENABLE_Sacado=ON \
-D Trilinos_ENABLE_MueLu:BOOL=ON \
-D Trilinos_ENABLE_Stratimikos=ON \
-D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_CXX_FLAGS="-g -O3" \
-D CMAKE_C_FLAGS="-g -O3" \
-D CMAKE_FORTRAN_FLAGS="-g -O5" \
-D Trilinos_EXTRA_LINK_FLAGS="-lgfortran" \
-D CMAKE_VERBOSE_MAKEFILE=FALSE \
-D Trilinos_VERBOSE_CONFIGURE=FALSE \
-D TPL_ENABLE_MPI=ON \
-D BUILD_SHARED_LIBS=ON \
-D CMAKE_INSTALL_PREFIX:PATH=$HOME/share/trilinos \

if blas/lapack path need to be specified, we need

cmake \
-D Trilinos_ENABLE_Sacado=ON \
-D Trilinos_ENABLE_MueLu:BOOL=ON \
-D Trilinos_ENABLE_Stratimikos=ON \
-D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_CXX_FLAGS="-g -O3" \
-D CMAKE_C_FLAGS="-g -O3" \
-D CMAKE_FORTRAN_FLAGS="-g -O5" \
-D Trilinos_EXTRA_LINK_FLAGS="-lgfortran -pthread" \
-D CMAKE_VERBOSE_MAKEFILE=FALSE \
-D Trilinos_VERBOSE_CONFIGURE=FALSE \
-D TPL_ENABLE_MPI=ON \
-D BUILD_SHARED_LIBS=ON \
\
-D TPL_ENABLE_BLAS:BOOL=ON \
-D BLAS_INCLUDE_DIRS=$MYHOME/OpenBLAS-0.2.14/include \
-D BLAS_LIBRARY_DIRS=$MYHOME/OpenBLAS-0.2.14/lib \
-D BLAS_LIBRARY_NAMES:STRING="libopenblas.a" \
-D TPL_ENABLE_LAPACK=ON \
-D LAPACK_INCLUDE_DIRS=$MYHOME/OpenBLAS-0.2.14/include \
-D LAPACK_LIBRARY_DIRS=$MYHOME/OpenBLAS-0.2.14/lib \
-D LAPACK_LIBRARY_NAMES:STRING="libopenblas.a" \
\
-D CMAKE_INSTALL_PREFIX:PATH=$MYHOME/trilinos-11.14.3 \
../

Currently, I found gcc-5.2.0 + OpenBLAS-0.2.41 + trilinos-11.14.3 can work. -pthread is very important here. otherwise errors occurs. Using other Blas semms problem

In order to instal OpenBLAS in virtualbox, use
make TARGET=NEHALEM

16. Then finally we can configure deal.ii by

cmake -DCMAKE_INSTALL_PREFIX=$HOME/dealii-8.3.0 -DDEAL_II_WITH_METIS=ON -DMETIS_DIR=$HOME/metis-5.1.0 -DDEAL_II_WITH_P4EST=ON -DP4EST_DIR=$HOME/p4est-1.1 -DDEAL_II_WITH_TRILINOS=ON -DTRILINOS_DIR=$HOME/trilinos-12.2.1 ../

17. We need also add -pthread in the CMakeLists.txt as
set (CMAKE_CXX_FLAGS "-pthread")
SET(CMAKE_CXX_FLAGS "-pthread")
what is the differences?

--------------------------------------------------------------------------------

2015.10.07

这两天在家调程序,发现如果使用了trilinos库,在家里的系统里编译的就一直有问题,总是出现错误。先是不认相关的库,比如对于

TrilinosWrappers::SparseMatrix tm;
TrilinosWrappers::MPI::Vector rhs;
总是说sparseMatrix不是TrilinosWrappers中的类型,或者MPI不是TrilinosWrappers中的类型。后来查了/include/deal.II/lac/trilinos_vector.h ,看起来应该是#ifdef DEAL_II_WITH_TRILINOS这个判断并没有支持trilinos,所以后面就不继续声明了。干脆重装trilinos并重新编译dealii,所有都通过了,程序编译也都成功了,结果总是出现错误

/usr/bin/ld: warning: libmpi.so.1, needed by /home/nicklj/trilinos-11.14.3/lib/libmuelu-adapters.so, may conflict with libmpi.so.12
/usr/bin/ld: warning: libmpi.so.1, needed by /home/nicklj/trilinos-11.14.3/lib/libmuelu-adapters.so, may conflict with libmpi.so.12
CMakeFiles/main.dir/src/DE_implicit.cc.o: In function `dealii::TrilinosWrappers::MPI::Vector& dealii::TrilinosWrappers::MPI::Vector::operator=<double>(dealii::Vector<double> const&)':
/home/nicklj/dealii-8.3.0/include/deal.II/lac/trilinos_vector.h:695: undefined reference to `Epetra_MpiComm::Epetra_MpiComm(int)'
/home/nicklj/dealii-8.3.0/lib/libdeal_II.g.so.8.3.0: undefined reference to `Teuchos::details::safeCommFree(int*)'
collect2: error: ld returned 1 exit status
make[2]: *** [main] Error 1
make[1]: *** [CMakeFiles/main.dir/all] Error 2

在网上找了好半天,发现这个帖子:https://groups.google.com/forum/#!topic/dealii/dX5VZ61JiZE

说应该主要原因是trilinos和dealii采用了不同的编译器,前面已经说了,编译dealii的时候,我们会声明export CC=mpicc, export CXX=mpicxx,因为我们设置过路径让petsc帮我们装过mpich为默认mpi,所以dealii编译应该是用的这个。检查trilinos编译目录下的Makefile.export.Trilinos,发现

Trilinos_CXX_COMPILER=/usr/bin/mpicxx
Trilinos_C_COMPILER=/usr/bin/mpicc
Trilinos_Fortran_COMPILER=/usr/bin/mpif90

说明trilinos选择的mpi是系统默认的,难怪会出现总题,现在我们要改trilinos的编译配置,加一句-D MPI_BASE_DIR=$HOME/petsc-3.6.1/arch-linux2-c-debug/bin,最终配置为

cmake \
-D Trilinos_ENABLE_Sacado=ON \
-D Trilinos_ENABLE_MueLu:BOOL=ON \
-D Trilinos_ENABLE_Anasazi:BOOL=ON \
-D Trilinos_ENABLE_Epetra:BOOL=ON \
-D Trilinos_ENABLE_EpetraExt:BOOL=ON \
-D Trilinos_ENABLE_Triutils:BOOL=ON \
-D Trilinos_ENABLE_Belos:BOOL=ON \
-D Trilinos_ENABLE_Ifpack:BOOL=ON \
-D Trilinos_ENABLE_TESTS:BOOL=ON \
-D Trilinos_ENABLE_Stratimikos=ON \
-D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_CXX_FLAGS="-g -O3" \
-D CMAKE_C_FLAGS="-g -O3" \
-D CMAKE_FORTRAN_FLAGS="-g -O5" \
-D Trilinos_EXTRA_LINK_FLAGS="-lgfortran -pthread" \
-D CMAKE_VERBOSE_MAKEFILE=FALSE \
-D Trilinos_VERBOSE_CONFIGURE=FALSE \
-D TPL_ENABLE_MPI=ON \
-D MPI_BASE_DIR=$HOME/petsc-3.6.1/arch-linux2-c-debug/bin \
-D BUILD_SHARED_LIBS=ON \
\
-D TPL_ENABLE_BLAS:BOOL=ON \
-D BLAS_INCLUDE_DIRS=$MYHOME/OpenBLAS-0.2.14/include \
-D BLAS_LIBRARY_DIRS=$MYHOME/OpenBLAS-0.2.14/lib \
-D BLAS_LIBRARY_NAMES:STRING="libopenblas.a" \
-D TPL_ENABLE_LAPACK=ON \
-D LAPACK_INCLUDE_DIRS=$MYHOME/OpenBLAS-0.2.14/include \
-D LAPACK_LIBRARY_DIRS=$MYHOME/OpenBLAS-0.2.14/lib \
-D LAPACK_LIBRARY_NAMES:STRING="libopenblas.a" \
\
-D CMAKE_INSTALL_PREFIX:PATH=$MYHOME/trilinos-11.14.3 \
../

最后补充一下,编译trilinos的时候似乎用单核编成功率更高?

-----------------------------------------------------------------------------------------------

2015.12.17

安装slepc库,用于eigenvalue求解

下载slepc,http://slepc.upv.es/download/download.php?filename=slepc-3.6.1.tar.gz,把最后的版本号改为和petsc相同的版本

在$HOME下解压

export SLEPC_DIR=$HOME/slepc-3.6.1
cd $HOME/slepc-3.6.1
./configure
make

配置dealii编译参数

cmake -DCMAKE_INSTALL_PREFIX=$HOME/dealii-8.3.0 -DDEAL_II_WITH_METIS=ON -DMETIS_DIR=$HOME/metis-5.1.0 -DDEAL_II_WITH_P4EST=ON -DP4EST_DIR=$HOME/p4est-1.1 -DDEAL_II_WITH_TRILINOS=ON -DTRILINOS_DIR=$HOME/trilinos-11.14.3 -DDEAL_II_WITH_SLEPC=ON -DSLEPC_DIR=$HOME/slepc-3.6.1 ../