WRF on ORR2 with Intel compilers

Orr2 is the HPC cluster of the UCD School of Mathematics and Statistics.

ORR2 architecture

Orr2 consists of 20 compute nodes on the following queues (‘qstat -f’ will show all queues):

Remove modules:

ORR2 loads an openmpi/gnu module by default. We want to use intel compilers, so remove all loaded modules:

module purge

Set up the Intel compiler environment:

Note the dot at the start of the command – this is important. You need to “source” the file (so it has an affect in your current shell), not just run it.

. /share/apps/intelPS/composer_xe_2015.3.187/bin/compilervars.sh intel64
export PATH=/share/apps/mvapich2intel/bin:$PATH

Check the compilers:

[conor@orr2 LIBRARIES]$ mpif90 --version
ifort (IFORT) 15.0.3 20150407
Copyright (C) 1985-2015 Intel Corporation. All rights reserved.

[conor@orr2 LIBRARIES]$ mpicc --version
icc (ICC) 15.0.3 20150407
Copyright (C) 1985-2015 Intel Corporation. All rights reserved.

Build LIBRARIES:

Although there are newer versions of these libraries available, I’m using the ones linked to from the WRF Download page: http://www2.mmm.ucar.edu/wrf/users/download/get_sources.html Note that log files may contain some errors, but they may not be important for the overall WRF installation. Change into your home directory and create a folder for these libraries:

cd
mkdir LIBRARIES

Each of the libraries below should be built in the LIBRARIES directory

NetCDF

Download: http://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/netcdf-4.1.3.tar.gz and put it in your LIBRARIES folder.

cd
cd LIBRARIES
tar -xzf netcdf-4.1.3.tar.gz
cd netcdf-4.1.3
export DIR=/home/conor/LIBRARIES
export CC=icc
export CXX=icpc
export CFLAGS='-O2 -fpic'
export CXXFLAGS='-O2 -fpic'
export F77=ifort
export FC=ifort
export F90=ifort
export FFLAGS='-O2 -fpic'
export CPP='icc -E'
export CXXCPP='icpc -E'

note: Intel compiler option ‘-fpic’ generates position-independent code, and must be used when building shared objects. Configure, pointing to required install directory:

./configure --prefix=$DIR/netcdf --disable-dap --disable-netcdf-4 --disable-shared

make and check:

make
make check
make install
export PATH=$DIR/netcdf/bin:$PATH
export NETCDF=$DIR/netcdf

zlib

Download: http://www2.mmm.ucar.edu/wrf/src/wps_files/zlib-1.2.3.tar.gz and put in your LIBRARIES folder. This is a compression library necessary for compiling WPS (specifically ungrib) with GRIB2 capability. Assuming all the environment variables from the NetCDF install are already set, you can move on to the commands to install zlib. Extract and change into directory:

cd
cd LIBRARIES 
tar -xzf zlib-1.2.3.tar.gz
cd zlib-1.2.3

Configure and make:

export LDFLAGS=-L$DIR/zlib/lib
export CPPFLAGS=-I$DIR/zlib/include
./configure --prefix=$DIR/zlib
make
make check
make install

Build libpng

Download: http://www2.mmm.ucar.edu/wrf/src/wps_files/libpng-1.2.12.tar.gz and put in your LIBRARIES directory.Another compression library necessary for GRIB2 capability. Assuming all above environmental variables have been set, you can build as follows:

cd
cd LIBRARIES 
tar -xzf libpng-1.2.12.tar.gz
cd libpng-1.2.12
./configure --prefix=$DIR/libpng
make
make check
make install

Build JasPer

Download: http://www2.mmm.ucar.edu/wrf/src/wps_files/jasper-1.701.0.tar.gz and put in your LIBRARIES directory. Another compression library necessary for GRIB2 capability. Assuming all above environmental variables have been set, you can build as follows:

cd
cd LIBRARIES 
tar -xzf jasper-1.701.0.tar.gz
cd jasper-1.701.0
./configure --prefix=$DIR/jasper
make
make install
export JASPERINC=${DIR}/jasper/include
export JASPERLIB=${DIR}/jasper/lib

Build WRF

Download the WRF-ARW tar file from the WRf website: http://www2.mmm.ucar.edu/wrf/users/download/get_sources.html. Make sure that the Intel compiler environment is set up, and that environment variables are set:

. /share/apps/intelPS/composer_xe_2015.3.187/bin/compilervars.sh intel64
export PATH=/share/apps/mvapich2intel/bin:$PATH
export DIR=/home/conor/LIBRARIES
export PATH=$DIR/netcdf/bin:$PATH
export NETCDF=$DIR/netcdf
export JASPERINC=${DIR}/jasper/include
export JASPERLIB=${DIR}/jasper/lib

It is important to compile WRFV3 before WPS as WPS makes use of the external I/O libraries in the WRFV3/external directory that are built when WRF is installed. Both WRF and WPS should be built inside the same directory. Make this directory, and change into it:

cd
mkdir WRFV3.9.1
cd WRFV3.9.1

Put the WRF tar.gz file into this directory and unpack it. Then build with the ‘./configure’ command, choosing the option for dmpar INTEL (option 15) and basic nesting (option 1), followed by the ‘./compile’ command:

cd
cd WRFV3.9.1
tar -xzf WRFV3.9.1.1.TAR.gz
cd WRFV3
./configure
Option: 15. (dmpar) INTEL (ifort/icc)
Compile for nesting? (1=basic, 2=preset moves, 3=vortex following) [default 1]: 1
./compile em_real >& log.compile

…it’s took AGES (1 hour 20 minutes), but it worked, producing the following executables:

[conor@orr2 WRFV3]$ ls -hl main/*.exe
-rwxrwxr-x 1 conor conor 43M May 23 15:52 main/ndown.exe
-rwxrwxr-x 1 conor conor 43M May 23 15:52 main/real.exe
-rwxrwxr-x 1 conor conor 42M May 23 15:52 main/tc.exe
-rwxrwxr-x 1 conor conor 50M May 23 15:50 main/wrf.exe

Build WPS

Download the WPS tar file from the WRF website: http://www2.mmm.ucar.edu/wrf/users/download/get_sources.html. Follow similar steps to compile WPS:

cd
cd WRFV3.9.1
tar -xzf WPSV3.9.1.TAR.gz
cd WPS
./configure
  17.  Linux x86_64, Intel compiler    (serial)
./compile >& log.compile

If all goes well, you should get the following executables:

[conor@orr2 WPS]$ ls -hl *.exe
lrwxrwxrwx 1 conor conor 23 May 23 16:05 geogrid.exe -> geogrid/src/geogrid.exe
lrwxrwxrwx 1 conor conor 23 May 23 16:06 metgrid.exe -> metgrid/src/metgrid.exe
lrwxrwxrwx 1 conor conor 21 May 23 16:05 ungrib.exe -> ungrib/src/ungrib.exe

That’s it!