Categories
WRF

WRF on ORR2 with Intel compilers

Orr2 is the HPC cluster of the UCD School of Mathematics and Statistics.

ORR2 architecture

Orr2 consists of 20 compute nodes on the following queues (‘qstat -f’ will show all queues):

  • 2×64.q
    compute-0-0.local, compute-0-1.local
    2 nodes with 64-cores: AMD Bulldozer/Opteron. 2.1GHz, 128GB.
  • 68nht.q
    compute-1-0.local
    1 node with 12 cores: Intel Nehalem. 2.66GHz, 24GB, Infiniband
    compute-1-1.local – compute-1-7.local
    7 nodes with 8 cores: Intel Nehalem. 2.4GHz, 24GB, Infiniband
  • 6x8i.q
    compute-2-0.local – compute-2-5.local
    6 nodes with 8 cores: Intel Core2. 2.5GHz, 32GB.
  • 4x8a.q
    compute-3-0.local – compute-3-3.local
    4 nodes with 8 cores: AMD Shanghai. 2.6GHz, 32GB.

Remove modules:

ORR2 loads an openmpi/gnu module by default. We want to use intel compilers, so remove all loaded modules:

module purge

Set up the Intel compiler environment:

Note the dot at the start of the command – this is important. You need to “source” the file (so it has an affect in your current shell), not just run it.

. /share/apps/intelPS/composer_xe_2015.3.187/bin/compilervars.sh intel64
export PATH=/share/apps/mvapich2intel/bin:$PATH

Check the compilers:

[conor@orr2 LIBRARIES]$ mpif90 --version
ifort (IFORT) 15.0.3 20150407
Copyright (C) 1985-2015 Intel Corporation. All rights reserved.

[conor@orr2 LIBRARIES]$ mpicc --version
icc (ICC) 15.0.3 20150407
Copyright (C) 1985-2015 Intel Corporation. All rights reserved.

Build LIBRARIES:

Although there are newer versions of these libraries available, I’m using the ones linked to from the WRF Download page: http://www2.mmm.ucar.edu/wrf/users/download/get_sources.html Note that log files may contain some errors, but they may not be important for the overall WRF installation. Change into your home directory and create a folder for these libraries:

cd
mkdir LIBRARIES

Each of the libraries below should be built in the LIBRARIES directory

NetCDF

Download: http://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/netcdf-4.1.3.tar.gz and put it in your LIBRARIES folder.

cd
cd LIBRARIES
tar -xzf netcdf-4.1.3.tar.gz
cd netcdf-4.1.3
export DIR=/home/conor/LIBRARIES
export CC=icc
export CXX=icpc
export CFLAGS='-O2 -fpic'
export CXXFLAGS='-O2 -fpic'
export F77=ifort
export FC=ifort
export F90=ifort
export FFLAGS='-O2 -fpic'
export CPP='icc -E'
export CXXCPP='icpc -E'

note: Intel compiler option ‘-fpic’ generates position-independent code, and must be used when building shared objects. Configure, pointing to required install directory:

./configure --prefix=$DIR/netcdf --disable-dap --disable-netcdf-4 --disable-shared

make and check:

make
make check
make install
export PATH=$DIR/netcdf/bin:$PATH
export NETCDF=$DIR/netcdf
zlib

Download: http://www2.mmm.ucar.edu/wrf/src/wps_files/zlib-1.2.3.tar.gz and put in your LIBRARIES folder. This is a compression library necessary for compiling WPS (specifically ungrib) with GRIB2 capability. Assuming all the environment variables from the NetCDF install are already set, you can move on to the commands to install zlib. Extract and change into directory:

cd
cd LIBRARIES 
tar -xzf zlib-1.2.3.tar.gz
cd zlib-1.2.3

Configure and make:

export LDFLAGS=-L$DIR/zlib/lib
export CPPFLAGS=-I$DIR/zlib/include
./configure --prefix=$DIR/zlib
make
make check
make install
Build libpng

Download: http://www2.mmm.ucar.edu/wrf/src/wps_files/libpng-1.2.12.tar.gz and put in your LIBRARIES directory.Another compression library necessary for GRIB2 capability. Assuming all above environmental variables have been set, you can build as follows:

cd
cd LIBRARIES 
tar -xzf libpng-1.2.12.tar.gz
cd libpng-1.2.12
./configure --prefix=$DIR/libpng
make
make check
make install
Build JasPer

Download: http://www2.mmm.ucar.edu/wrf/src/wps_files/jasper-1.701.0.tar.gz and put in your LIBRARIES directory. Another compression library necessary for GRIB2 capability. Assuming all above environmental variables have been set, you can build as follows:

cd
cd LIBRARIES 
tar -xzf jasper-1.701.0.tar.gz
cd jasper-1.701.0
./configure --prefix=$DIR/jasper
make
make install
export JASPERINC=${DIR}/jasper/include
export JASPERLIB=${DIR}/jasper/lib
Build WRF

Download the WRF-ARW tar file from the WRf website: http://www2.mmm.ucar.edu/wrf/users/download/get_sources.html. Make sure that the Intel compiler environment is set up, and that environment variables are set:

. /share/apps/intelPS/composer_xe_2015.3.187/bin/compilervars.sh intel64
export PATH=/share/apps/mvapich2intel/bin:$PATH
export DIR=/home/conor/LIBRARIES
export PATH=$DIR/netcdf/bin:$PATH
export NETCDF=$DIR/netcdf
export JASPERINC=${DIR}/jasper/include
export JASPERLIB=${DIR}/jasper/lib

It is important to compile WRFV3 before WPS as WPS makes use of the external I/O libraries in the WRFV3/external directory that are built when WRF is installed. Both WRF and WPS should be built inside the same directory. Make this directory, and change into it:

cd
mkdir WRFV3.9.1
cd WRFV3.9.1

Put the WRF tar.gz file into this directory and unpack it. Then build with the ‘./configure’ command, choosing the option for dmpar INTEL (option 15) and basic nesting (option 1), followed by the ‘./compile’ command:

cd
cd WRFV3.9.1
tar -xzf WRFV3.9.1.1.TAR.gz
cd WRFV3
./configure
Option: 15. (dmpar) INTEL (ifort/icc)
Compile for nesting? (1=basic, 2=preset moves, 3=vortex following) [default 1]: 1
./compile em_real >& log.compile

…it’s took AGES (1 hour 20 minutes), but it worked, producing the following executables:

[conor@orr2 WRFV3]$ ls -hl main/*.exe
-rwxrwxr-x 1 conor conor 43M May 23 15:52 main/ndown.exe
-rwxrwxr-x 1 conor conor 43M May 23 15:52 main/real.exe
-rwxrwxr-x 1 conor conor 42M May 23 15:52 main/tc.exe
-rwxrwxr-x 1 conor conor 50M May 23 15:50 main/wrf.exe
Build WPS

Download the WPS tar file from the WRF website: http://www2.mmm.ucar.edu/wrf/users/download/get_sources.html. Follow similar steps to compile WPS:

cd
cd WRFV3.9.1
tar -xzf WPSV3.9.1.TAR.gz
cd WPS
./configure
  17.  Linux x86_64, Intel compiler    (serial)
./compile >& log.compile

If all goes well, you should get the following executables:

[conor@orr2 WPS]$ ls -hl *.exe
lrwxrwxrwx 1 conor conor 23 May 23 16:05 geogrid.exe -> geogrid/src/geogrid.exe
lrwxrwxrwx 1 conor conor 23 May 23 16:06 metgrid.exe -> metgrid/src/metgrid.exe
lrwxrwxrwx 1 conor conor 21 May 23 16:05 ungrib.exe -> ungrib/src/ungrib.exe

That’s it!

Categories
ESIPP

Journal Club: 09/05/2017

Article Title: Diurnal variations of land surface wind speed probability distributions under clear-sky and low-cloud conditions.

This article examined the impact of cloud cover on diurnal wind speed variability at different heights, from 10m to 200m above the surface, at a mast in Cabauw, Netherlands.  Oberservations from a backscatter ceilometer were used to determine the cloud cover.  Two subsets of the data were isolated using the ceilometer observations, clear-sky (CLR) and low-level cloudiness (LLC).  These were used to partition the wind speed observations.  Wind speed statistics (mean, standard deviation and skewness) were calculated for the two cloud regimes for each season.

Diurnal variability of mean potential temperature (top), mean wind speed (middle) and standard deviation of wind speed for June-July-August CLR (left) and LLC (right) cloud conditions at six different heights (10m, red; 20m, pink; 40m, yellow; 80m, cyan; 140m, green; and 200m, blue).

It was found that the distributions of wind speed as well as the vertical structure of the wind profile tended to vary depending on the cloud conditions.  Wind speeds tend to be stronger and more variable under LLC conditions, compared to CLR.

 

Thoughts for future research:

– Comparing Cabauw wind speed observations to reanalysis output at different heights.

– Using surface clearness index to partition our data in a similar manner.

– Looking at windrose plots under different cloud conditions.

– Could be relevant to Eadaoin’s research as a method for classifying cloud conditions for her radiation analysis.