Chair: Eileen Shea

Rapporteur: Jill Jaeger

Members: Stephen Andersen, David Cash, Nancy Dickson, Karen Fisher-Vanden, Wendy Franz, Robert Frosch, Milind Kandlikar, Gregory Knight, Stephen Schneider, David Slade

 1. INTRODUCTION

 This report is based on the deliberations of the Working Group during one and a half days. It took as its starting point the background theme paper on this topic, which identifies some significant outcomes of climate assessments during the past 20 years or more, as well as some of the pitfalls that have limited the effectiveness of assessments. The Working Group was conscious of the need to consider the assessment process as a whole and to view outcomes as part of the incremental progress in the assessment process. That is, it is unlikely that a single assessment leads to a particular outcome. An outcome will be affected by multiple factors, as illustrated by the assessment tetrahedron in the theme paper.

 There are several ways of evaluating the effectiveness of an assessment process. For example, the observation that the issue has been moved further along the issue evolution path could be taken as an indicator of an effective assessment process. Similarly, the achievement of some preset goal would be an indicator. Also, by taking a retrospective look at key events and policy decisions in the past, it is possible to ask whether there was any causal connection to an assessment process.

 The report begins by noting the wide range of goals that the sponsors, practitioners and users of assessments might have. With these goals in mind it is possible to look at the characteristics of assessments that will support the achievements of such goals. The report then examines some of these characteristics more closely, since they were considered to be of particular importance in achieving desired outcomes of assessments. This is followed by an examination of some of the main mechanisms through which assessment characteristics affect outcomes. The report finishes by linking the important characteristics of assessments to the pitfalls identified in the theme paper and making recommendations about the design and content of an improved assessment background process.

 In its deliberations, the Working Group had to recognize the differences in ability to participate in assessment and in communication with the user community and public in cultural and economic settings that differ from those of industrialized nations. The group therefore recognized the special needs of assessments in and/or affecting such places and the obligations of sponsor communities to make participation and dissemination impossible.

 The motivations for sponsoring, designing, performing or using the results of assessment are varied. While some may be motivated by a need to clarify a scientific question, others may be motivated by the impending necessity of making a decision. The practitioners and managers of assessments (assessors) and the sponsors and users (users) of assessments can have one or a number of goal(s) with regard to the assessment. Indeed, there are likely to be multiple users for any given assessment process and those users could have conflicting goals. Appendix I to this Working Group report lists the kinds of goals that were suggested by the Working Group in a short discussion at the beginning of its deliberations. The list shows that some goals are specific to the actor (i.e. assessor or user), while others are common to both actor groups. It is likely that having shared goals has an effect on the outcome of an assessment, particularly with regard to whether the outcome is viewed as a success or a failure. If assessors and users have different goals, this will also have implications for outcome. If the goal of the assessor is to prepare for a decision and the goal of the sponsor is to prevent action, one or both of them will view the outcome of the assessment as a failure. It is possible to cluster some of the goals, e.g. a number of them relate to awareness building, others relate to policy initiatives or scientific initiatives.

 In the climate assessments of the past 20 years or more, some goals have dominated:  

• Collect information to provide the basis for a decision (including decisions to do nothing) 
• Synthesize the state of the art (mainly science-based) 
• Define research frontiers, look for missing links 

In past climate assessments, there are also some goals that are rare, in particular those of providing information on mitigation of climate change and adaptation to climate change.

 Using the background theme paper as a basis for discussion, the Working Group identified five particular outcomes of interest in the assessment process for climate change. These were: 

• a change or stabilization of policy position 
• a change of the scope of the debate 
• a change of the position of the issue on scientific or political agendas 
• a change of the research and/or policy community engaged with the issue  
• a change of public awareness of the issue 

The Working Group identified important characteristics of assessment processes that contribute to achieving these outcomes.

If the goal of an assessment is to change or stabilize a policy position or bring about a decision, the Working Group identified several important assessment characteristics that will support the achievement of that goal: 

• recognition of the information needs of the user 
• interaction between assessors and users during the assessment process 
• analysis at the relevant spatial scale and for the relevant sector(s)
• understanding the decision making framework and the information needs of the decision maker (e.g., how the decision maker takes risk and uncertainty into account) 
• inclusion of information on the state of scientific knowledge, levels of certainty etc. 

The characteristics of an assessment process that support the change of the scope of the debate or a reframing of the issue were identified in the theme paper: 

• inclusion of new scientific knowledge, perhaps from disciplines or sources not previously considered 
• inclusion of new participants in the process, bringing new perspectives or new information. 

Moving the issue to the next stage in issue evolution 

Over the past 20 or more years, the climate change issue has shifted from being debated almost entirely in scientific circles to a position on national and international policy agendas. A number of important characteristics of the assessment process have contributed to this shift: 

• elaboration of a clear statement on the state of scientific knowledge 
• elaboration of a clear statement of available response options 
• careful presentation and communication of the results of the assessment 
• careful timing of the release of the results of the assessment to take advantage of external events and relevant deadlines 
• development of new institutions (e.g., the development of an intergovernmental mechanism to replace the self-selected group of experts doing climate assessments at the end of 1980s) 
• involvement of participants with high credibility 
• attention to linkages to other issues  
• development of public sensitivity to the issue.

Changing the research or policy community 

An important outcome of climate assessments has been the development of both research and policy communities interested in the issue of climate change. The characteristics of the assessment process that are important are: 

• development of new institutions 
• entrainment of new participants 
• mobilization of additional funding 
• creation of linkages between the scientific and policy communities 
• creation of international pressure to deal with the issue 
• involvement of stakeholders in the assessment process

Changing public awareness 

If one of the goals of an assessment is to increase public awareness about the issue, there are several ways in which the design of the process can support the achievement of this objective: 

• inclusion of an executive summary or a summary for policy-makers that highlights interesting/relevant material 
• selection of appropriate language, i.e. using language understandable by non-experts 
• careful handling of uncertainty while conveying the state of scientific knowledge and range of available policy options and providing a clear description (explanation) of the nature of uncertainty 
• explicit description of the assessment process (so that the public understands the nature of scientific discourse on the issue but also sees that there has been a process of reaching consensus) 
• careful handling of potentially controversial issues, to minimize criticism in the media  
• provision of detail on the numbers of participants in the assessment process and their disciplinary background and nationality 
• highlighting of new, intriguing aspects of the content and process 
• careful timing of the release of the assessment results 
• awareness of the potentially long-term interest in the results of the assessment 

Public awareness can also be changed through education, so provision of assessment results in a form that can be used as educational material is a long-term investment in increasing public awareness.

 4. WHICH ASSESSMENT CHARACTERISTICS DESERVE PARTICULAR TTENTION?

The previous paragraphs listed a wide range of assessment characteristics that influence outcomes. The discussions of the Working Group showed, however, that some of these characteristics are more important than others. 

 For each of the characteristics identified in the previous section, it should be possible to identify the causal pathways by which they affect the outcome. We do this here for three of the characteristics to illustrate the nature of the mechanisms. 

The case studies carried out by the GEA Fellows provided empirical evidence that successful outcomes depend on assessors being aware of the needs of the users. Regular interaction between assessors and users during the assessment process assures that needs and revised needs can be effectively communicated. Interaction also enhances the building of a common language (shared vocabulary) or an understanding of the special terms used by the different communities. There is also evidence that regular interaction also enhances the development of mutual trust and shared ownership. Furthermore, it allows both sides to learn about the constraints under which the others are operating. For example, the constraints on the users might be the range of policy options they can realistically consider, while the assessors might be constrained by the limits of scientific knowledge. 

All of these factors -- communication of needs, common understanding, and shared ownership -- allow the assessment process to be more effective, especially if the goal is to shift a policy position or reach a decision. 

How does linking to other issues affect the outcome of an assessment process? 

There are a number of ways in which the linking of the content of the climate change assessment process to other issues has helped to achieve the goal of an assessment. Such linkages can raise the salience of the climate change issue to individuals by illustrating why they should be interested. For example, a farmer may well be aware of the implications of and available response options to deal with natural climatic variability, so if the assessment can link the issue of climatic change to the issue of climatic variability, for example by comparing the magnitude of potential climatic change with the intra-annual variability usually experienced, the user will realize why the issue could be important. Similarly, an assessment process can link the issue of climate change to other policy issues on the agenda. For example, by linking the climate change issue with the issue of increasing energy efficiency, the assessment can point out that increasing energy efficiency could simultaneously reduce carbon emissions per unit energy and reduce costs per unit energy.  

The climate change issue can also be linked to other environmental issues. Recent assessments have, for instance, pointed out that energy efficiency measures not only reduce carbon emissions but can also reduce emissions of acidifying substances. Linkages can be made through discussion of technical change, as with energy efficiency measures. The linkage can also be through discussion of behavioral change: encouraging people to take public transportation reduces urban smog and carbon dioxide emissions at the same time and for many people the short term interest is in reducing the health effects of urban smog, not dealing with global climatic change. 

Linking climate change to other scientific issues or other policy debates can increase the saliency of the issue or take advantage of other benefits and improve the effectiveness of the assessment.

Making sure that the results of an assessment process are effectively communicated is an essential element in increasing public awareness of the issue and in informing users and assessors, who are part of the policy process. Shifts in policy positions, implementation of policy measures enhancement of research capacity, changes of funding levels etc all depend on a sufficient level of public awareness of the issue. The design of an assessment process to allow for effective communication means that the risk that the media can misrepresent or distort the results of the assessment can be minimized. In addition, being discounted, disvalued and misinterpreted by potential users of information from the assessment can be avoided through effective communication. By making the process of the assessment transparent, the public can understand that although there continues to be scientific debate and even disagreement on some issues, consensus can be reached on important points and provide the basis for recommendations for research or policy development. 

6. RECOMMENDATIONS FOR THE DESIGN OF EFFECTIVE ASSESSMENT PROCESSES 

The background theme paper for this Working Group identified a number of pitfalls that have marred the effectiveness of the assessment process for the climate change issue. A clear recommendation to improve the effectiveness would therefore be "avoid those pitfalls". On another level, the recommendation could be "ensure that the assessment process is accurate, credible, without compromises, respectful of the actors involved". All of these recommendations suggest things that are under the control of the assessors or users. Even if these recommendations were followed, exogenous events could lead to a failed outcome of the assessment process. For example, if an assessment is being produced for a particular minister who loses his post for other reasons in the week before the assessment is completed and submitted, the outcome could be that the assessment report disappears without a trace, resulting in no action where some had been anticipated. 

The Working Group derived a number of recommendations on improving the outcomes of assessments on the basis of the discussion of the characteristics of assessments that can affect outcomes. These recommendations are strongly related to the broad pitfalls discussed in the theme paper. 

Two recommendations on process -- inclusion of self-evaluation and careful treatment of uncertainty -- were considered to be especially worthy of attention in the further development of the assessment process for the issue of climate change. 

Evaluation While there is little empirical evidence in the GEA case studies on the importance of evaluation, discussion in the Working Group concluded that although there is formal evaluation of progress in other areas, especially business, there is little to no effort to provide for structured evaluation of the assessment process for the issue of climate change issue. That is, there is no formal requirement for internal or external actors to ask the question "How are we doing in the assessment process, an important component of managing the risk of climate change?" and then incorporating the results of that evaluation in the design of the next stage of the assessment process. 

Furthermore, there is no easily available written record of such evaluation, which means that future assessors and managers can not learn from the experience of earlier evaluations unless there happens to be some continuity of people involved in the assessment process. 

Evaluation of the process would allow for an "adaptive assessment process". This should be built in to the design of an assessment process in addition to any scientific review process to determine scientific quality. 

Dealing with uncertainty Assessments of the climate change issue generally deal with uncertainty through finding a consensus among the assessors. This is an unsatisfactory process not only because it ignores the differing levels of scientific expertise among the assessors, which should be taken into account by weighting their subjective judgements of the level of uncertainty, but also because it is not clear that consensus is the best way to deal with the question of uncertainty as the climate change issue matures. This is certainly a topic that requires serious consideration in the design of assessments. 

Being aware of context As the theme paper pointed out, a major pitfall of assessments has been to ignore the political, economic, scientific or general context within which an assessment is performed. Context influences the content of the assessment and the way that it is carried out, as well as the receptiveness to the results of the assessment. An assessment that ignores context can, however, be effective in the longer term for policy making or in the short term for shifting the focus of debate. 

Finding out the needs of potential users The design and process of an assessment must be able to respond to the needs of potential users. It is particularly important to be aware of scale issues (global numbers are of little use to someone wanting to know what to plant in a particular agricultural area) and the kind of expert judgment absolutely essential for making certain decisions. 

Noting past experience. There are several ways in which ignoring past experience can become a pitfall for an assessment. Obviously, ignoring the results of past assessments can lead to repetition and unnecessary duplication of effort . The background theme paper also points out that "institutional history" should not be overlooked. This means that assessments must account for policy instruments already in place when deciding which to add to the portfolio.  

Appropriate participation The selection of participants in the assessment process in terms of their disciplinary background, country origins and role (experts, stakeholders, consultants, peer reviewers etc.) is an important determinant of the outcome of the assessment. For the purposes of building capacity and public legitimacy, participation has to be kept broad and might even include those who are sometimes seen as detractors from the issue. 

Coordinating national and international science-policy frameworks The experience with climate assessments has highlighted the need to coordinate international assessment requirements. As shown in the background theme paper, while the IPCC assessment process requires certain resource, infrastructure, and operating procedures within the governments of participating countries, this is difficult for a developing country like India. The review process of IPCC requires input often on a tight schedule, that is difficult to fulfill at the national level. 

Ensure effective follow-up The design of an assessment process has to ensure that the results of the assessment are correctly disseminated to the appropriate audience(s) in a timely fashion. Often the effectiveness of this communication can be enhanced by taking advantage of external events, such as recent weather extremes (e.g. a heat wave) or a high level political meeting. It is also important to ensure that those that should have access to the assessment really do have access. 

7. CONCLUSIONS

The discussions of the Working Group clearly showed that careful analysis of the assessment process over an extended period of time (20 years or more) can generate useful recommendations on improving assessment processes and outcomes. Similar critical analysis of the assessment process for other issues will be useful in refining the recommendations based on the analysis of the climate issue, as well as uncovering further important suggestions for the design and content of an improved assessment process.

Raise awareness

Synthesize disciplinary perspectives

Assessing whether a target can be met

Enhancing research capacity

Obtaining research funding

Enhancing personal power and status (meeting old friends, collecting frequent flier miles)

Conducting interesting research

Creating problems for further research

Increasing or decreasing support for research

Formulate policy

Provide Information for decisions

Set the stage for technological change

Prevent action

Bludgeon Policy makers

Educate assessors

Gain constituency

Promote Action

Promote political support

Control Content

Justify a previous decision

Increase interaction between user and assessor

Address uncertainties (highlight or downplay)

Reduce uncertainties

Justify a previous decision

Ascertain status of science

Build support via engagement

Support strategic national interest

Prepare for negotiations

Educate the public

Determine political feasibility

Frame the debate

Discover new truths

Link scientific findings

Further political Ends

Link to other issues

Solve a problem or mitigate its effects

Keep an issue on the agenda

Develop a shared understanding of the science

Develop a core set of understanding about the issue

Define new research frontiers

Look for missing links in or between science and policy fields

Increase or decrease support for research

Deal with skeptics