NCAR models

Checkout the Source

All the source code for the NCAR models is stored in a subversion repository. The source has been modified to work on our machines and includes scripts that we have created to make it easier to build and run the models. For the CAM model, all the associated data files are also checked into subversion; however, there were too many data files for WACCM and these can be copied from /ptmp/mills/waccm/data on the Samba share attached to cadfael.

1. Go to the directory where you want to build the model.

MAC: On the Macs, the default location that the scripts are expecting for CAM is

cd /Volumes/Data/Models/cam

with NETCDF and MPICH libraries located in

 /Volumes/Data/Libraries

You can override the default location with the environment variables CAM_BASE and CAM_LIBRARY or WACCM_BASE and WACCM_LIBRARY, but it is easier to use the default and it is a good idea to build on a case sensitive volume.

2. Checkout the source code using subversion (svn)

CAM:

svn checkout http://subversion/projects/ncar_models/cam/trunk

WACCM:

svn checkout http://subversion/projects/ncar_models/waccm/trunk

3. WACCM Input data

CADFAEL: If you are running WACCM on cadfael, you need to create a symbolic link to the WACCM data directory (/home/mills/waccm/inputdata/) under each trunk or branch directory:

cd trunk
ln -s /home/mills/waccm/inputdata/ .

MAC: If you are running WACCM on your desktop Mac, you need to copy the inputdata directory to your local machine for WACCM to read in as it runs. The /home/mills/waccm/inputdata directory on cadfael has 35GB of data, but you won't need all of this to run WACCM. A subset of about 1GB of data, containing the data needed to run the base configuration at 4x5 horizontal resolution is available at ftp://laspftp.colorado.edu/pub/bardeen/data . The inputdata directory is on the Samba share attached to cadfael as well, in /ptmp/mills/waccm/inputdata. There is an extra 42GB in a subdirectory there called wa319_smax, which you will not need.

Build the Model

The make script allows you to build the model the same way on several different types of machines. It takes care of the three steps required to build CAM: configure, make, and build-namelist. By default, the script builds the model using the parallelism model (e.g. MPI) that is most efficient for your machine. You can override this with the CAM_MODEL or WACCM_MODEL environment variable.

1. Go to the root directory of the source of your model that you checked out from subversion.

cd /path/to/models/cam/trunk

2. Set any environment variables you would like to override the defaults in the scripts, including CAM_DYN, CAM_RES, CAM_MODEL, CAM_CONFIGURE_OPTS, ... For example, you can use CAM_RES environment variable to switch the model resolution to 10x15.

setenv CAM_RES 10x15

For WACCM, you will likely start out using 4x5 resolution, which is not the default, so type:

setenv WACCM_RES 4x5

You can see all the variables and their default values by looking at the make script.

3. Build the model.

scripts/make-cam.csh

or for WACCM:

scripts/make-waccm.csh

4. The script will display the settings that are being used for the build. If you are happy with them, hit <return>. If you answer 'n' <return> the make will exit without doing anything.

[Charles-Bardeens-G5:Models/cam/trunk] cbardeen% scripts/make-cam.csh 
scripts/make-cam.csh:  Using xlf90_r compiler with gcc
scripts/make-cam.csh:  Set INC_NETCDF to /sw/netcdf/3.6.0-p1/xlf/include
scripts/make-cam.csh:  Set MOD_NETCDF to /sw/netcdf/3.6.0-p1/xlf/include
scripts/make-cam.csh:  Set LIB_NETCDF to /sw/netcdf/3.6.0-p1/xlf/lib
scripts/make-cam.csh:  Set model to OMP
scripts/make-cam.csh:  Set branch to trunk
scripts/make-cam.csh:  Using  fv  dynamics
scripts/make-cam.csh:  Using  10x15  resolution
scripts/make-cam.csh:  Set debug to OFF
scripts/make-cam.csh:  Using namelist:  &camexp nelapse=-1 rest_pfile='/Volumes/Data/Models/cam/trunk/run' mss_irt=0 / &clmexp rpntpath=' Volumes/Data/Models/cam/trunk/run' /
scripts/make-cam.csh:  Continue? (Y/n)

A succesful build will create output similar to the following:

creating /Volumes/Data/Models/cam/trunk/build/Filepath
creating /Volumes/Data/Models/cam/trunk/build/params.h
creating /Volumes/Data/Models/cam/trunk/build/misc.h
creating /Volumes/Data/Models/cam/trunk/build/preproc.h
creating /Volumes/Data/Models/cam/trunk/build/Makefile
creating /Volumes/Data/Models/cam/trunk/build/config_cache.xml
configure done.
scripts/make-cam.csh:  building CAM in /Volumes/Data/Models/cam/trunk/build ...
scripts/make-cam.csh:  Started at Wed Apr 11 09:09:30 MDT 2007
scripts/make-cam.csh:  Completed at Wed Apr 11 09:21:47 MDT 2007

If your make fails, check the output file (build/MAKE.out) to see what happened.

5. If you are rebuilding (i.e. a build directory already exists), you will be asked if you want to reconfigure.

scripts/make-cam.csh: Configure again? (complete rebuild) (Y/n)

Reconfiguring is the safest way to build and will execute the configure script and force everything to be recompiled. If you have only changed source code and not environment variables that control the build, it is faster if you don't reconfigure by entering an 'n' and <return> at this prompt. Changes to most of the environment varaibles (e.g. CAM_RES) require that you reconfigure to build properly. If you need to reconfigure, it is a good idea to delete the build directory before running the make script just to be safe.

6. If a namelist file already exists in the build directory, you will be asked if you want to rebuild it.

scripts/make-cam.csh: Recreate namelist? (Y/n)

If you have made modifications to the namelist file, you should enter 'n' and <return> to this prompt. Otherwise, your namelist file will be overwritten with a default version. If the only changes you have made were to the namelist file, then you don't need to rebuild at all and can immediately run.

Run the Model

The run script, scripts/run-cam.csh handles most of the differences in running the model on the different types of platforms. It can also be used for your first run (initial), subsequent runs (restart), and altered runs (branch).

The CAM_RUN or WACCM_RUN environment variable defines were the model will be executed and where the history files will be stored. It defaults to the directory run/results. It is a good idea to set this variable to a name that describes the type of run your are doing. You can do multiple runs with different namelist values in the same directory tree by changed the value of CAM_RUN or WACCM_RUN.

The number of computational nodes (processors or cores) used are set differently depending on the platform. For your desktop machine using Open/MP or MPI, set the CAM_THREADS or WACCM_THREADS environment variable to the number of threads that you want to use. If you are on a cluster and using a queueing system, then the number of nodes is set in the user's run script. When using more MPI taks than one third of the number of latitudes, the the npr_yz variable in the namelist must be set to do a further decomposition. CAM requires a minimum of 3 latitudes per MPI task.

Initial

An initial run is the first run (timestep 0) for a particular set of runs.

1. Select a name for your run. You should make sure that the directory run/<your run name> does not already exist. If it exists, then the scripts will assume that you want a restart run and not an intial run.

setenv CAM_RUN dust

2. Make any changes you would like to the default namelist file. This is where you indicate how long to run for, what variables to output, and can control many aspects of how the run will happen (e.g. double CO2). The full set of namelist variables and history fields that you can output are defined in the CAM User's Guide. The namelist file used for an initial run is in the build directory (i.e. build/namelist).

3. Run the job. On the clusters you need to submit the job to a queue that will determine when and where your job will run.

scripts/run-cam.csh

On the clusters you need to submit the job to a queue that will determine when and where your job will run.

qsub scripts/run-cam.csh

4. When your run completes, you can check the output of your model in

run/dust/RUNddd.out

or on the clusters

cam.odddd

Where ddd is a series of digits that changes from run to run so that the file names are unique.

Restart

A restart run continues the run from where a previous initial or restart run left off. You can make very few changes to the namelist file in a restart run, only things like nelapse. If you want to change more than that you either need to do a branch run or an initial run.

1. Make sure you specifiy the name of the previous run.

setenv CAM_RUN dust

2. Run the job the same way you did for the intial run.

Branch

Branch runs are used when you want to change something in the namelist file and then continue a run from an existing restart file. For example, you would use this if you wanted to add additional fields to the output file in the middle of a run. Here are the steps to do this using our runs scripts and continuing the branch run in the same run directory and with the same case name as the original run.

1. Make sure you specifiy the name of the previous run.

setenv CAM_RUN 125_lt_gw2_28

2. Make the changes that you want to the namelist file that exists in the run directory.

3. Change the namelist file by adding these fields to the cam section

brnch_retain_casename = .true.
nrevsn = '/home/bardeen/waccm/carma/run/125_lt_gw2_28/125_lt_gw2_28.cam2.r.2027-11-16-00000'

and these fields to the lnd section

brnch_retain_casename = .true.
nrevsn = '/home/bardeen/waccm/carma/run/125_lt_gw2_28/125_lt_gw2_28.clm2.r.2027-11-16-00000'

4. Tell the run script to do a branch run.

setenv CAM_RUNTYPE BRANCH

5. Run the job as usual.

6. Before running the job again, remove the extra fields from the namelist file added in step 3 and unset the CAM_RUNTYPE envrionment variable defined in step 4.