Musings on Climategate

Admitting the Mess: CRU and the Money Game

Posted in Uncategorized by emelks on December 6, 2009

This is interesting. Apparently the Met office, Newcastle University and UAE recognized before 1/2/2007 that their data is a disaster and applied for money from DEPRA to attempt to make sense of it. They also claim they’ll put it all on the web. The file is depra.pdf.

Dealing with the possible consequences of climate change depends on understanding predictions and taking action to mitigate against predicted changes, to adapt, or both. Deciding whether to take action will require weighing up risks and benefits and evaluating alternative strategies. Decision makers will range from individuals, through local government, to national governments and intergovernmental negotiators, and in the public sector alone, cover a gamut of professions from engineers and educators to policy makers and scientists.

Making policy requires access to knowledge, not just the underlying information and data. While data leads to information and knowledge, the steps from data to knowledge in the climate prediction arena can involve handling tens of terabytes of data (in information terms: roughly equivalent to several copies of the British Library’s entire holdings), as well as significant knowledge of the tools (models) used to create the simulations, and a background in both environmental sciences and sophisticated statistics. Managing the underlying data itself is a problem, once data volumes become large enough, hardware and software problems that are rare with small data volumes become common enough that mitigation strategies against failure within the data archive itself are necessary. Holding high volume complex data over time introduces new problems involving format migration and semantic interoperability. Software must also be produced to visualise and extract data (that might be input to other tools such as flood predictions), before producing policy relevant advice.

Defra has funded, and continues to fund, projects which produce climate prediction data, scenarios and advice for the UK climate impacts community. This work is one part of that continuum of research activity, covering the reliable storage of climate data and predictions, and the interfaces to that data to make it usefully available to the impacts community, who themselves provide policy relevant advice. Data will be extracted from the archives of the Met Office Hadley Centre (MOHC) and made available by the British Atmospheric Data Centre (BADC), a national repository for storing digital environmental data for the long term (BADC expertise and additional funding via the National Centre for Atmospheric Science will also ensure that the data will be held for posterity). The result of this phase of the work will be prototype systems coupling interfaces to the data archives developed together by the BADC and Newcastle University (both world leaders in developing web-based interfaces to complex geophysical data). The University of East Anglia and the MOHC will provide expert advice. The eventual goal will be to provide data access to both experts in the climate impacts community and the general public via these interfaces but such deployment will be expected in a future phase of Defra supported activity. The first phase, covered by this proposal, will take eighteen months. The second phase (not covered here), would improve the prototype and then provide and support public access and should begin near month twelve of this project, and continue for at least two years. The project will be carried out in close partnership with the Defra funded UK Climate Impacts Programme (UKCIP) and will contribute to the Intergovernmental Panel on Climate Change (IPCC)’s Data Distribution Centre (DDC). Although there will be three significant components to the work (known as the Data Delivery Package [DDP], the Climate Impacts LINK Project, and the DDC), this activity will eventually (during phase two) create a joined-up resource that serves the whole community, from research scientist to town planner.

The project takes a significant leap forward from its predecessors, exposing cutting edge science involving complex probabilistic datasets and exploiting a Weather Generator (developed in another Defra project) to produce sample time series of weather conditions at specific UK locations in the future. We will also exploit new metadata standards developed both within this project and others with which the project participants are involved. The underlying archives will provide tens of terabytes in reliable network attached storage with multiple gigabit bandwidth to the wider Internet. The data interfaces will be state-of-the art, and, where appropriate, exploit the latest standards-compliant metadata structures and interfaces to make the best use of both technology and experience in other communities. An active climate scientist who is also an expert on data systems will provide UK representation on the IPCC Task Group on Data and Scenario Support (TGICA). To avoid duplication of effort between the components, the entire activity will be supported by a common management infrastructure and technical service layer which will dovetail with existing complex data and information systems at the BADC.

Although this project proposal outlines developments to deliver a system fit for deployment in a phased follow-on project, some aspects of the project, namely LINK and IPCC-DDC components, will include operationally deployed services during this prototype development phase.


This project will provide a prototype system capable of delivering ground-breaking climate change scenarios to the public and policy makers via the web. This will enable interaction with probabilistic climate datasets in a manner that the user can pose a useful question and receive a response that is both informative and retains the uncertainty inherent in climate change predictions. In the follow-up project the deployed system will allow decision-makers and industry to plan their strategies in response to indicative predictions of future climate.

The integration of the DDP with a Weather Generator model will demonstrate how such tools can be employed to add value to climate model output. It will also provide users with access to high temporal resolution data not previously available.

The LINK component (and DDP in the follow-on project) will deliver considerable usage of Defra-funded climate research outputs (UKCIP predict that potentially more than 1000 users will wish to access the DDP system). The users will make use of the data in a variety of ways including: informing policy, making strategic decisions, aiding research, exploring possible climate scenarios and understanding climate models. The IPCC-DDC element will allow climate researchers greater access to data by incorporating existing and new datasets into the BADC’s existing infrastructure.

The following reports will chart progress of the project and provide a commentary of the outputs:

– Quarterly reports, summary Financial Year reports, Annual reports, Final reports to Defra.
– Periodic reports to the TGICA.

Note that the LINK archive will also be of benefit in upcoming IPCC asessment activities (for example, it is expected that the next assessment report will use a distributed archive).

16. Staff effort
(a) Please list the names and grades/job titles of staff and their input to the project together with their unit costs e.g. daily charge-out rates (note 13)

Dr B Lawrence (Project Lead) Band 2 – 108 days at £750/day
Mr A Stephens (Technical Lead) Band 4 – 258 days at £511/day
Dr K Marsh (LINK Manager) Band 4 – 300 days at £511/day
Ms S Latham (Project Manager) Band 4 – 43 days at £511/day

Met Office Hadley Centre:
Science Support Role – 312 days at £356/day
Technical Development Role – 86 days at £356/day

Newcastle University:
Mr G Hobona – 323 days at £355/day
Mr P James – 22 days at £399/day
Technical Asst – 32 days at £103/day

University of East Anglia:
Professor P Jones – 2 days at £344/day
Dr C Harpham – 105 days at £166/day

18. Please give below the address to which payments should be made.

Council for the Central Laboratory of the Research Councils (CCLRC)
Rutherford Appleton Laboratory
OX11 0QX

Declaration (to be completed by a duly authorised signatory of the proposer’s organisation)

Mr Tony Wells
Head of Sales Contracts


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: