Dealing with GRIB2 data

We use python for our data analysis and plotting.

The wonderful xarray package (http://xarray.pydata.org/en/stable/) has no problem with NetCDF, but doesn’t like GRIB2. GFS data are in GRIB2, however, so it would be nicve if we could convert them easily to NetCDF.

We can do this by using the great wgrib2 tool (http://www.cpc.noaa.gov/products/wesley/wgrib2/).

To install wgrib2 (on ORR2 or SONIC, for example), do the following:

  1. Make a directory for wgrib2 and move there:
    cd
    mkdir wgrib2
    cd wgrib2
  2. Get wgrib2 and unpack it:
    wget ftp://ftp.cpc.ncep.noaa.gov/wd51we/wgrib2/wgrib2.tgz
    tar -xzf wgrib2.tgz
  3. Move into the directory, set environment variables and make wgrib2:
    cd grib2/
    export CC=gcc
    export FC=gfortran
    make
  4. Now you can convert a GFS GRIB2 file to NetCDF!
    wgrib2/grib2/wgrib2/wgrib2 gec00.t12z.pgrb2f00 -netcdf GFS.nc

    It’d be a good idea to add the wgrib2 executable directory to your PATH, so you can call it from any directory.

Download GFS forecast

We would like to run WRF forecasts driven by global forecast data from NOAA’s Global Forecast System (GFS: http://www.emc.ncep.noaa.gov/index.php?branch=GFS).

To download a subset of the GFS data use NOMADS g2subs:

http://nomads.ncep.noaa.gov

  • Select the “grib filter” link for the dataset you want.
  • Use “make subregion”: e.g. lon:-55 to 25, lat: 25 to 70
  • Don’t use the analysis data file – it doesn’t contain soil data.
  • You can see which fields are in the GFS file by using the g2print.exe command in the WPS/util directory.
  • Using all levels, all variables, and subregion (lon:-55 to 25, lat: 25 to 70) reduced the size of the analysis file from 176M to 24M. The file size becomes even smaller by requesting less variables.
  • If you want to download a lot of data, you should tick the “Show the URL only for web programming” box, and write a bash script to change the URL for the dates you want, with variable output file name: curl $URL -o ${EM}${FT}.grb

How do you write that bash script?

Say you end up with a URL like this:

URL=http://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25_1hr.pl?file=gfs.t12z.pgrb2.0p25.f000&lev_surface=on&var_TMP=on&subregion=&leftlon=-7&rightlon=-5&toplat=54&bottomlat=52&dir=%2Fgfs.2017041712

This code contains different parts:

  • file=gfs.t12z.pgrb2.0p25.f000 – this is the base file name: model GFS, forecast initial time 12Z, …, forecast hour is 000. The forecast hour is probably what we want to change.
  • dir=%2Fgfs.2017041712 – this is the year, month, day and hour of the forecast initial time. All forecast data are inside this directory.
  • We don’t want to change the rest of the URL: forecast model, level, variables, subregion. If you want to change these, go through the grib_filter web page again and generate a new URL.

If we want to download multiple forecast hours for the same forecast initial time, we could do so with a bash script like this:

#!/bin/bash

# set the base URL in two parts: URL1 and URL2
# leave our forecast hour:

URL1='http://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25_1hr.pl?file=gfs.t12z.pgrb2.0p25.f'

URL2='&lev_surface=on&var_TMP=on&subregion=&leftlon=-7&rightlon=-5&toplat=54&bottomlat=52&dir=%2Fgfs.2017041712'

# Let forecast hour vary from 0 to 24.
# It needs to have three digits, so we start with 1000:

for i in {0..24}
do
  echo $i
  TFCH=`expr 1000 + $i`
  FCH=`echo $TFCH | cut -c2-4`
  URL=${URL1}${FCH}${URL2}
  curl $URL -o GFS${FCH}.grb
done

Note: make sure you are using backticks (`) not single quotes (‘) on the TFCH= and FCH= lines. Single quotes don’t evaluate anything in between them. Backticks run the command between the backticks and return the result, which is what we want here.

Instead of typing these commands into the terminal, you can simply save all of the above into a file, say “getGFS.bash” and then in the terminal type:

bash getGFS.bash

 

Data on ESIPP shared directory

VariableStart DateEnd DateSourceAreaResolutionFrequencyLocation
SLP1980-01-012010-12-31MERRA220S-90N, -180W-180E0.5d N-S, 0.625d E-W1-hourlyMERRA2/SLP
PS1980-01-012005-12-31MERRA249S-60N, -11.875W-2.5E0.5d N-S, 0.625d E-W1-hourlyMERRA2/ASM
QV10M1980-01-012005-12-31MERRA249S-60N, -11.875W-2.5E0.5d N-S, 0.625d E-W1-hourlyMERRA2/ASM
QV2M1980-01-012005-12-31MERRA249S-60N, -11.875W-2.5E0.5d N-S, 0.625d E-W1-hourlyMERRA2/ASM
SLP1980-01-012005-12-31MERRA249S-60N, -11.875W-2.5E0.5d N-S, 0.625d E-W1-hourlyMERRA2/ASM
T10M1980-01-012005-12-31MERRA249S-60N, -11.875W-2.5E0.5d N-S, 0.625d E-W1-hourlyMERRA2/ASM
T2M1980-01-012005-12-31MERRA249S-60N, -11.875W-2.5E0.5d N-S, 0.625d E-W1-hourlyMERRA2/ASM
TS1980-01-012005-12-31MERRA249S-60N, -11.875W-2.5E0.5d N-S, 0.625d E-W1-hourlyMERRA2/ASM
U2M1980-01-012005-12-31MERRA249S-60N, -11.875W-2.5E0.5d N-S, 0.625d E-W1-hourlyMERRA2/ASM
V2M1980-01-012005-12-31MERRA249S-60N, -11.875W-2.5E0.5d N-S, 0.625d E-W1-hourlyMERRA2/ASM
U10M1980-01-012005-12-31MERRA249S-60N, -11.875W-2.5E0.5d N-S, 0.625d E-W1-hourlyMERRA2/ASM
V10M1980-01-012005-12-31MERRA249S-60N, -11.875W-2.5E0.5d N-S, 0.625d E-W1-hourlyMERRA2/ASM
U50M1980-01-012005-12-31MERRA249S-60N, -11.875W-2.5E0.5d N-S, 0.625d E-W1-hourlyMERRA2/ASM
V50M1980-01-012005-12-31MERRA249S-60N, -11.875W-2.5E0.5d N-S, 0.625d E-W1-hourlyMERRA2/ASM
ALBEDO1980-01-012009-12-31MERRA249S-60N, -11.875W-2.5E0.5d N-S, 0.625d E-W1-hourlyMERRA2/RAD
SWGDN1980-01-012009-12-31MERRA249S-60N, -11.875W-2.5E0.5d N-S, 0.625d E-W1-hourlyMERRA2/RAD
SWGDNCLR1980-01-012009-12-31MERRA249S-60N, -11.875W-2.5E0.5d N-S, 0.625d E-W1-hourlyMERRA2/RAD
SWTDN1980-01-012009-12-31MERRA249S-60N, -11.875W-2.5E0.5d N-S, 0.625d E-W1-hourlyMERRA2/RAD

Download MERRA2 Data

As we want to analyse MERRA2 data, we’ll need to download the data first.

Bosilovich et al. have produced a technical report with the file specification for MERRA2, available here.

In order to download MERRA2 data, you need to register with Earthdata Login and then authorize NASA GESDISC Data Access. Instructions are here: https://disc.gsfc.nasa.gov/registration/registration-for-data-access

You should log in to sonic.ucd.ie and download the data from there. Once you’ve downloaded what you want, move your data to our shared folder (so we don’t all download the same thing): /storage-earth/ESIPP/MERRA2

You can download MERRA2 data, using a subsetter to select your area of interest, here.

  • The area that I used for my initial download was:
    West:  -12  North:  60  South:  49  East:  3
  • The select start and end dates. Do only one month at first, to make sure that everything works.
  • Select the parameter(s) you want.
  • Then click ‘Start Search’
  • This ends up showing you instructions on how to download your data using wget:
    • Save the list of URLs  (click on the link on the page) to your local workstation. Change the file name to myfile.dat. Then transfer this (scp ) to sonic – you’ll download the data from sonic.
    • The first time you retrieve data, you’ll need to create two files in your home directory on sonic:
      • create an empty ~/.urs_cookies file.
      • create a ~/.netrc file pointing to urs.earthdata.nasa.gov with your NASA Earthdata login username (USERNAMEHERE) and password (PASSWORDHERE) by placing the following text in the file:

        machine urs.earthdata.nasa.gov login USERNAMEHERE password PASSWORDHERE
    • Now use wget to retrieve the files named in your myfile.dat file (NOTE: this includes the -content-disposition flag):
      wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition -i myfile.dat