Sections
You are here: Home → User Portal → Documentation → IMDI
Integrating Model and Data Infrastructure: IMDI is supporting activities such as source code analysis, model development, compilation, experiment execution and graphical monitoring, as well as result archiving, retrieval, processing, and visualization. The latter facets connect the modelling activities with the ICSU World Data Center for Climate WDCC.

The components

IMDI consists of tools for source code management (SCM), for compilation of ES models (SCE), for execution of ES numerical experiments (SRE), and for model result analysis (MRA).

  • The SCM comprises a subversion (SVN) repository for script and model source codes, html-formatted browsing facilities for source code, and scripts for model source code analysis. The source code analysis includes detection and flagging of FORTRAN 90 coding rule violations and transformation to a non-violating version, all of which are essential for the SCE. It is also possible to generate lists of cpp (conditional compilation) flags used in the model.
  • The SCE is based on the GNU Make utility. It includes portable Makefile templates for compilation, tools for dependency analysis to help with the (g)make variable setting, and for the creation of Makefile prerequisites and cpp flag dependencies.

Model and platform specific compilation scripts are created, which check all components of a coupled model including user controlled libraries and sub-models.

  • The SRE is a toolkit to create model and platform specific scripts for experiment execution with optional tasks for input-data preprocessing, for model integration, for data archiving in file systems or in the WDCC, for graphical monitoring of experiments, and for model output post-processing.

Model results can be analyzed with the help of tools for local data processing and visualization, the WDCC data search and retrieval Graphic User Interface (GUI), and WDCC data retrieval and visualization tools. Specification of Cera/WDCC meta data is supported by the provision of XML templates.

  • The Data Archiving with Model code base and configuration meta data - plans for extending the Cera data model by numerical model describing metadata.
  • There are a few special hints for using GISS with NetCDF datasets  and some GrADS informations made by the former Model and Data group. The link goes to the old M&D WEBpages.
  • The Workflow-Models can be find here.

 

The philosophy

M&D has undertaken the design and realization of an infrastructure for Earth System (ES) data management and modelling activities named IMDI.

The idea behind the design was to provide the user with toolkits which are easy to use, well documented, and give the same look&feel for all ES models and all platforms used within the infrastructure. This should facilitate communication and model as well as data exchange between scientists or institutes.

Instead of reinventing the wheel we are taking existing (popular) software and adapt it to the specific requirements of ES modelers. In order to minimize installation efforts, the software used with the infrastructure is kept to a minimum.

IMDI has a modular design allowing tools to be used independently from another. They are, however, also interoperable, and it is therefore possible to execute routines as suites, starting, for example, with the adaptation of a model source code to generally accepted coding rules and ending with the export of model data into the WDCC archive.
Furthermore, the modular infrastructure supports a modular design of coupled models. This means that IMDI supports the replacement of individual components in coupled models by other components.

 

 A third aspect of modularity arises from the design of the Standard Running Environment (SRE). A SRE experiment consists of a number of different tasks (the ovals in the diagram), each of which can be switched on or off when the experiment is configured (dashed lines: process flow, solid lines: data flow).

 

The history

Some of the development activities of IMDI, such as the modular design of the system just described, the Standard Compile Environment (SCE), and the SRE  as  support of experiment execution, were started in the PRISM EU FP5 Project for Integrated Earth System Modelling.

The achievements of this project have been summarized in a Detailed Documentation prepared for the final project report.
Current tasks of IMDI include the integration of the modelling environment with the data management at M&D, in particular the interfacing of the WDCC  Database (DB) with the SRE. This activity is coordinated within the PSI (PRISM Support Initiative).


Plans and ongoing activities

The creation of metadata before and during the execution of an experiment is a development that has already started and will also be carried out in the future. Metadata are needed not only in order to describe data generated by models, but they also include descriptions of the models themselves and the configurations used in the experiments.

Extension of the Cera data model by an additional scheme describing numerical models is presently being done. The model metadata scheme of Cera will be designed in coordination with other international metadata initiatives, e.g. the Numerical Model Metadata (NMM) project initiative of GO-ESSP and the EU FP7 METAFOR.

It is planned to facilitate the specification of the contents of model metadata by using as much information as possible of that already stored in the SCE/SRE and thereby automate the process and take that burden from the user (more ...).

Cooperation with the PSI initiative is underway to provide a Graphical User Interface for specification of additional metadata content.

 

 

Documentation and Technical Reports

SCE technical report , 7th Edition,   17 August 2007  (pdf) SCE technical report

SRE technical report , 3rd Edition,  28 February 2009  (pdf) SRE technical report

Coding Rules, and Quality Standard technical report, Version 1, 31 May 2002 (ps) PRISM Coding Rules

Powerpoint presentation on SCM/SCE DKRZ/M&D user introduction workshop March 2007 (ppt) IMDI SCM/SCE.

Powerpoint presentation on SRE DKRZ/M&D user introduction workshop March 2007 IMDI SRE.

 
Document Actions