How do I compile VASP 5.3.5 on Cray XC? - compilation

Question

I need to compile VASP 5.3.5 on a Cray XC MPP machine. The system has the Gnu, Intel and Cray compiling environments available. There is also a choice of Intel MKL or Cray LibSci for BLAS, LAPACK, ScaLAPCK.
Which is the best compiler to use, the best numerical library to use and the Makefile options to use?

Answer 1

Tests on ARCHER (http://www.archer.ac.uk) have shown that using the Intel compiler with Intel MKL and FFTW produce the best performance and the most stable build of VASP 5.3.5 on the Cray XC30 system.
Full compilation instructions can be found at:
http://www.archer.ac.uk/documentation/software/vasp/compiling_5-3-5-phase2.php
Briefly, the procedure is:
module swap PrgEnv-cray PrgEnv-intel
module load fftw
module load cray-pe-hugepages2M
Modify the library makefile to have the following options:
CPP = gcc -E -P -C $*.F >$*.f
FC=ftn
CFLAGS = -O3
FFLAGS = -O3 -unroll -ip -no-prec-div -xAVX
FREE = -free
Build the library (assuming makefile is called "makefile.cray_xc_intel.lib"):
cd vasp.5.lib
make -f makefile.cray_xc_intel.lib
Move to the main source code directory:
cd ../vasp.5.3
Setup the preprocessor options in the Makefile (this is for the multiple K-points version):
CPP = $(CPP_) -DMPI -DHOST=\"CrayXC-Intel\" \
-DNGZhalf \
-DLONGCHAR \
-Dkind8 \
-DCACHE_SIZE=2000 \
-Davoidalloc \
-DRPROMU_DGEMV \
-DMPI_BLOCK=100000 \
-Duse_collective \
-Drandom_array \
-DscaLAPACK
Set the makefile compilation options:
FC=ftn
FCL=$(FC)
CPP_ = ./preprocess <$*.F | cpp -P -C -traditional >$*$(SUFFIX)
FFLAGS = -free -march=corei7-avx -assume byterecl -m64
OFLAG = -O3 -ip -fno-alias -unroll-aggressive -opt-prefetch -use-intel-optimized-headers -no-prec-div
OFLAG_LOW = -O1 -g -ftz
OBJ_LOW = broydon.o
Set the makefile linear algebra library options for Intel MKL:
MKL_PATH = $(MKLROOT)/lib/intel64
BLAS=
LAPACK=
BLACS=
SCA=
LIB = ../vasp.5.lib/linpack_double.o -L../vasp.5.lib -ldmy \
${MKL_PATH}/libmkl_blas95_lp64.a ${MKL_PATH}/libmkl_lapack95_lp64.a \
${MKL_PATH}/libmkl_scalapack_lp64.a \
-Wl,--start-group ${MKL_PATH}/libmkl_intel_lp64.a \
${MKL_PATH}/libmkl_sequential.a ${MKL_PATH}/libmkl_core.a \
${MKL_PATH}/libmkl_blacs_intelmpi_lp64.a -Wl,--end-group -lpthread -lm
Finally, set the makefile options for linking FFTW:
FFT3D = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o
Now build the code (assuming makefile is called "makefile.cray_xc_intel"):
make -f makefile.cray_xc_intel