Calibration Data

The set of scripts used to process CMIP5/CMIP6 data for calibrating MAGICC and also powering our CMIP6 data website. Checkout the notebooks directory for examples of how to use the data to make your own plots. If you do build new functionality (particularly in a language other than Python), please make a merge request so that we share the learning.

Getting Started

Data are stored in the data/ folder. This directory is not checked into git as per the previous repository, but is now a staging area for files to be uploaded to S3. Downloaded releases will also go in this folder by default. CMIP6 and CMIP5 data are in different folders to save bandwidth as there will likely be few CMIP5 releases in the near future so the CMIP5 data doesn't need to be copied on every release.

A given release consists of multiple datasets, each of which can be uploaded individually. This allows for the downloading, crunching and stitching to happen on a server, and Malte to upload any extra processing he does locally.

$ tree -L 3 data                                                                                                                              [15:49:33]
data
├── CMIP6
│   ├── 20200423
│   │    ├── ascii
│   │    │      ├── json
│   │    │      └── mag
│   │    └── database
│   └── 20200622
│        ├── ascii
│        │      ├── json
│        │      └── mag
│        └── database
└── cmip6
    └── 20200423
         ├── ascii
         │      ├── json
         │      └── mag
         └── database

The downside of this new methodology is that it requires more disk usage as new releases are now copies rather than. The benifit is that now subsets of data can be downloaded as needed, for example CMIP6 data needs the JSON, MAG and database.

Downloading

Users can download the latest files using the [scripts/manage_data.py]scripts/manage_data.py tool. By default data for the latest release is downloaded.

python scripts/manage_data.py download

Anyone wanting to interact with the data stored in S3 needs credentials for the MAGICC AWS account. These can be obtained from Jared.

Processsing Retractions

Before publishing retracted data should be removed. Currently we only remove the produced MAG files rather than the retracted nc files.

python scripts/find_retractions.py
python scripts/remove_retractions.py

Publishing

There are a number of steps involved in publishing a new release, but most of these are managed by the perform-release-cmip6.sh.

We also publish snapshots of the results to Zenodo. The following steps assume that the ZENODO_TOKEN environment to a valid Zenodo access token.

export RELEASE=20201026
cd data/CMIP6/ascii/mag
for i in average-year-mid-year monthly
do
  echo $i
  tar czf CMIP6_${i}_$RELEASE.tar.gz --exclude="*.log" $i
  bash ../../../../scripts/zenodo-upload.sh 4145924 data/cmip5/ascii/mag/CMIP5_$i_$RELEASE.tar.gz
done