The purpose of this project is to provide a single web portal for requesting CESM experiment runs (including CMIP5 runs), tracking run progress, viewing diagnostics, tracking post-processing status, and viewing ESG published dataset availability.

Questions? Contact Gary Strand

Project Status - May 2012

Phase II - released Beta test period 5/31/2012 - 6/15/2012

Phase II (2.0 Beta) - Functionality Completed :

  • Create a test environment for development on host csegweb
  • incorporate security using SVN / UCAS passwords integrated into application (currently using test SVN; need to update to live SVN via expdb-httpd.conf
  • Add experiment viewing/updating priviledge checks

Phase II - In Progress :

  • Mask all the email links to block spam crawlers
  • Finish adding security checks to add_note, add_diag, add_pub, update_status, and view_cmip5
  • harden host linux for outside access
  • include the latest compset values and add link to Diane's compset_list.pl program that takes tag_name argument
  • Parse old run.txt files
  • Add an option to save a filtered search
  • Add the table management functions
  • Integrate the Experiment DB with the model run scripts to automate data transfers and reduce redundancy
  • Integrate with Adam's public web page to harvest data (http://www.cesm.ucar.edu/experiments/cesm1.0/)
  • Integrate with CCR post-processing status db (http://webint.cgd.ucar.edu/project/ccr/procstat/cgi-bin/index.cgi)
  • Use JQuery and javascript / AJAX to validate form input fields
  • Integrate with a CESM testing DB
  • Integrate with a CESM Performance DB
  • Integrate with a SVN user mgmt dashboard

Phase I - released

Phase I - Functionality Completed :

  • List all experiments and sort columns ascending and descending
  • Add/View/Update/Clone an experiment
  • View/update status
  • View/update notes
  • View/update CMIP5 experiment details
  • Add/View/Update diagnostics links
  • Add/View/Update approvals
  • Get all the filter options working in search
  • Get the free-text search working
  • Add crontab entries to recreate t_filter and t_exp_tmp at 3 a.m.
  • Cloned case_name should be "Copy of case_name"
  • Get Javascript DOM working for parent-child window controls - see approvals to get this working
  • Add row-level locking on update of any tables - should be handled in the DBI and DBD::mysql perl modules
  • Get the sort order up/down arrows working with the free-text searching
  • activate all emails for approvals before going live.
  • Work on the archive_metadata.sh script to only save case directory info to svn repository - tested and stored under tag scripts4_111107
  • Link case_name to case naming convention in add,View,Update, and clone
  • Add crontab entries to call perl scripts to send approval email reminders - based on emailFreq field days
  • add publications management option for links into Opensky
  • Generate the run.txt file
  • Incorporate all historical run data from 4 different sources; CESM run wiki, Matt's old php rundb (http://ccsm-rundb.cgd.ucar.edu/case_list.php), CMIP5 run wiki, Haiyan's decadal perdiction runs.
  • Add automated data pruning reminder messages according to the CESM Data Management and Data Distribution Plan.

Phase I - In Progress :

  • include the latest compset values and add link to Diane's compset_list.pl program that takes tag_name argument
  • Parse old run.txt files

Database Design

The database is built in mysql using these naming conventions:

Database Name: rundb
Database User: u_rundb
Table Name: t_[tablename]
Trigger Name: tr_[triggerName]
Unique ID: [tablename]_id
View Name: v[#]_[tablename]
Index Name: [tablename_variable]_idx

Experiment DB Entity Relationship Diagram

Interactive Forms

The web-based DB interactive forms can all be used on the production site at CESM Experiment Database

Programming specs and conventions

The application is written in perl using a mysql backend DB. The perl programs use templates to format html data with links to CSS for style. The templates are parsed by a cgi program that inserts values loaded from the db into hash arrays.

Work Flow

From the CESM1 Tutorial, the basic work flow for running the model and interacting with the Experiment DB is outlined below.

Basic Work Flow

One-Time Setup Steps

  • Registration and Download - (note: a valid SVN password is going to be needed to access the Experiment DB in the future)
  • Create an Input Data Root Directory
  • Porting

Creating & Running a Case

  • Add a new experiment to the CESM Experiment Database
  • Elect to send run review approvals automatically from the Experiment DB add form
  • Create a New Case using create_newcase
  • Configure the Case
  • Build the Executable
  • Running the Model: Initial Run and Output Data
  • Running the Model: Continuation Runs
  • Manually run the archive_metadata.sh script in the case directory to populate the SVN repository with case details
  • Update the Experiment DB status information for this run

Monitoring & Post-processing a case