The paper addresses the outcomes of assessments and the elements of context, content and process that are responsible for them. This is followed by a discussion of the pitfalls that have influenced outcomes. Four main outcomes of assessments are identified: 

• a change or stabilization of policy position 
• a change of the scope of the debate 
• a change of the position of the issue on scientific or political agendas 
• a change of the research and/or policy community engaged with the issue  

Elements of context, content and process can affect outcome. Of importance are scientific context, e.g. the state of the scientific community; political context, e.g. the domestic or international political situation; economic context, e.g. the state of the national economy; and other contextual constraints, e.g. the way that science and policy communities are linked. Elements of content include the scope of the assessment and the treatment of uncertainty. The significant elements of process include the assessment design, the participation and the follow-up to the assessment.

 The major pitfalls include a failure in the follow-up to the assessment, ignoring the context within which the assessment is carried out, ignoring the needs of potential users, inappropriate participation and content.

GHG greenhouse gases

IPCC Intergovernmental Panel on Climate Change

This paper poses a number of observations about the outcomes of assessments and their determinants that have emerged from the GEA Project’s retrospective study of 20 years of climate assessments. Our goal is to raise these historical perspectives for discussion among scientists and practitioners who have been engaged in assessments. After a general discussion of the kinds of assessments and the distinct types of actors involved, the paper describes four main outcomes of assessments that have been identified by looking at the climate assessments. The paper then discusses some of the main determinants of these outcomes. This leads to a discussion of a number of pitfalls that have had negative effects on the outcomes of assessments, and the avoidance of which could contribute substantially to improvements in future assessments. The majority of the citations made in this paper are to the papers produced by the GEA Fellows.

In general this paper uses "Assessment" to refer to the entire social process by which scientific knowledge is organized, evaluated, integrated and presented in documents intended to lead to further scientific research and/or policy action. For the purpose of examining the outcomes of assessments, this paper distinguishes among assessment documents, assessment processes that did or did not produce a document and "systems" that link science to policy. This distinction is necessary because in many countries there are no or few formal assessments (studies like those of the US National Academy of Sciences, reports of major meetings etc.) of the kind that have proliferated in industrialized countries, particularly in the United States. (See the Discussion Paper by Kandlikar and Sagar (1997) for a characterization of the assessment system in India).

There is a potentially wide range of outcomes of assessments. While the outcome could simply be a document, it could also be an increase of concern in the general public or some political action. The outcome of an assessment can also be a strengthening or weakening of the research community involved in the assessment. As this paper shows, the outcome of an assessment is not soley dependent upon the way in which it is produced.

Whether the outcome is viewed as a "success" or a "failure" depends on the role of the person or group of persons developing that view. To explore the question of "success" or "failure" further, it is necessary to distinguish between the producers, users and funders of assessments. The funder of an assessment might have the goal of building a research community or of stimulating a policy response and would view the assessment as a success if the original goal were fulfilled. The producer of the same assessment might view the outcome as a failure, if that producer had a different goal when setting out to do the assessment. The users of assessments can also have different views on the success or failure of assessments. In fact the category of "users" is broad and can be subdivided into "recipients" to whom the assessment results are formally presented, and then a group that includes all those who might use an assessment in various ways. The latter category can include other political actors, who might use the assessment to advocate or oppose certain policies, or to advocate delay; scientists or research managers who might use it as a reference or to formulate and fund research priorities; or the public at large. The study of the role of assessments of global environmental change in subnational impact assessment (Cash, 1997) shows that the local users (farmers) need information, which is relevant for short-term decision-making. They do not want data tables and statistical results but rather recommendations of what to do at the farm level in the immediate future. Thus while the funders and producers of a global scale assessment might view the assessment as a success because it fulfilled their goals, some users might judge it to be a failure if it does not have findings that are relevant at the local scale.

 In the following sections, the outcomes of assessments are discussed with reference to three factors: context, process and content. As illustrated in Figure 1, these factors interact with each other. Outcome can be influenced by context, content and process -- but the outcome of an assessment can also affect context, content and process of subsequent assessments. Process, for example the selection of assessors, can influence content, while content, for example the treatment of uncertainty, can influence process. The content and process of an assessment are interrelated and can also be influenced by the context within which the assessment is performed. Elements of context that can be important include the knowledge base, observable phenomena, actors and their interests and perceptions and the relation of the assessment focus to other issues on the agendas of actors involved.

 This paper explores some of the interesting elements of context, process, content and outcome for the climate assessments studied in the GEA Project.

 A major surprise in our research findings is that there has been so little discussion in the research and policy communities on what assessments actually accomplish. One of the goals of this paper is therefore to advance the discussion of assessment goals and outcomes.

 In fact, a wide range of possible outcomes exist. By "outcomes", this paper will mean events or occurrences subsequent to the assessment for which some causal link back to the assessment can be traced. We do not imply that assessments alone "caused" the outcomes we address, or that at least some of them would not have eventually occurred anyway. What we do suggest is that there are a number of ways in which assessments may have helped to shape subsequent events, and that these pathways are worth exploring.

 Some of the outcomes of assessments are intended by the assessors, while some are unintended (e.g. controversy, embarrassment and paralysis). The outcomes can be of relevance to the research community, the policy community and the public. Some of the most significant possible outcomes we have encountered in the GEA study include:

• The assessment affects an observable policy position 
• The assessment changes the scope of the debate by broadening, narrowing or reinforcing the content and/or by broadening, narrowing or maintaining the disciplinary or national participation 
• The assessment moves the issue to the next stage in the issue evolution from a scientific issue to one on political and public agendas and to specific policy action or inaction 
• The assessment builds, maintains or eliminates the research and/or policy community through changes in funding levels, capacity building, disciplinary linkages, national-international links or even job creation or elimination.

 2.1 Affecting An Observable Policy Position

 The GEA study of Fisher-Vanden (1997) concludes that in the US formal (i.e. report-like) assessments have had more effect on policy-makers through indirect channels than direct. It is suggested that some policy relevant parts of the IPCC Second Assessment were written with an intended audience of the advisers of policy-makers rather than the policy-makers themselves. Furthermore, Fisher-Vanden suggests that assessments with recommendations that support an existing view would have a larger impact. This does not imply, however, that assessors should provide policy-makers only with what they want to hear.

 One reason that could explain why assessments rarely affect policy positions directly is that the producers of assessments assume an overarching policy goal that ignores the constraints that policy-makers actually face. Policy goals are not only in response to climate change, so there can be conflicts between policy goals for goals for climate and other goals. Fisher-Vanden suggests that often assessments are used as ammunition by policy-makers, i.e. policy-makers hold up and attract attention to assessments with recommendations that support their existing policy views.

 Lastly, Fisher-Vanden finds that the discussion of policy options and policy instrument choice in assessments tend to overlook the contextual factors that constrain the types of instruments that policy-makers can actually consider. For example, Chapter 11 of the IPCC Second Assessment report provides a comprehensive list of all possible policy instruments without consideration for which instruments are politically feasible. It is necessary to note, however, that since policy-makers may be captured by particular power interests, assessments might have to ignore the question of political feasibility in order just to change the scope of the debate.

 For the issue of climate change the outcome of assessments has generally not been an observable change in policy position. Analysis within the GEA Project has suggested some reasons for this, at least with regard to the position in the US. Some of these points are picked up again in later sections of this paper.

 2.2 Changing The Scope Of The Debate

 Research in the GEA Project encountered several examples of shifts in the content of assessments and the related scientific and policy debates on climate. For example, neither the First nor the Second IPCC assessments discuss human activities that modify the climate at any scale except the global -- the IPCC 1995 report states "One complication with this definition [of climate and climate change] is [that it leaves out] the anthropogenic changes of climate on a restricted space scale, a good example of which is the heat island phenomenon by which highly urbanized areas may have a mean temperature which is higher than it would otherwise have been." In contrast the 1966 assessment of the US National Academy of Sciences listed the urban heat island effect as one of four key areas of climate change, noting that urbanization, like carbon dioxide, constituted a "continuing experiment in climate modification". This is an example of the narrowing of the content of the debate that has occurred in successive assessments of the climate issue. There are, of course, also examples of a widening of the content of the debate, exemplified by the inclusion of greenhouse gases other than carbon dioxide in assessments and by the inclusion of new sectors, like human health, in impact assessments (Long and Iles, 1997). 

There is evidence that changing the content of the scientific debate and/or the participation has affected how the issue is discussed. The case study of Franz (1997) shows that inclusion of government scientists at the Villach conference in 1985 led to the raising of particular policy relevant concerns, which in turn was reflected in the larger international debate. However, Franz’s paper also contains a counter-example in the failure of other GHGs to get into the mainstream of the policy debate on climate change, although the scope of the debate did shift with their inclusion.

 Shifts of the content of the debate have been accompanied by shifts of participation both in terms of the disciplines represented and the countries involved. In the case of climate change, assessments have led to a broadening of the participation from atmospheric scientists and geoscientists to include a very wide number of disciplines. Likewise, participation has broadened to include authors from developing countries and experts from industry and non-governmental organizations. This may in turn result in the injection of alternative perspectives in the scientific and political debate, particularly with the inclusion of perspectives from developing countries.

 A commonly observed pattern in the development of policy issues is that they originate as the concern of a relatively small number of technical people, remain a concern of a relatively narrow group for years, then suddenly blossom onto the public or policy agenda, and eventually settle down to a new -- if inevitably temporary -- equilibrium of sustained attention. This pattern has been followed for the climate issue, as noted in the Overview paper for the GEA study(Chapter I). An important potential outcome of an assessment is the shifting of the issue to the next stage of the evolution of the issue. Equally important from the perspective of some actors could be the absence of such a shift as a result of an assessment. For the climate change issue, two assessments have frequently been cited for having shifted the attention to the issue.

 The Villach 1985 assessment, described and analyzed by Franz (1997), clearly moved the climate change issue at the international level from being an issue discussed almost entirely within the scientific community to an issue on the international policy agenda.

The First Assessment of the IPCC was important in stimulating the negotiation of the Framework Convention on Climate Change. As Agrawala (1997) points out, at its first session in February 1989, the IPCC Bureau adopted a proposal by Working Group I to incorporate a 20-page policy document in its assessment, which would summarize the scientific results and place them into perspective. The Bureau then requested the other two IPCC Working Groups to produce similar policy documents. The policy-makers summary produced by Working Group I is widely regarded as being influential. In particular, according to Agrawala (1997) the statement that under a business-as-usual scenario the world is likely to see " a rate of increase of global mean temperature during the next century...that is greater than that seen over the past 10,000 years" was very influential in catalyzing the decision making process which eventually led to the signing of the Framework Convention on Climate Change in 1992. Similarly, the IPCC 1995 Summary statement on the "discernible influence of anthropogenic activities on global climate" has increased the attention of policy-makers and led to calls for serious action on the climate issue.

A common outcome of assessments has been enhanced capacity building in the research community. The First World Climate Conference (an assessment, in the sense that term is used here) led to the establishment of the World Climate Programme, which in turn led many countries to set up National Climate Programs. At the 1979 World Climate Conference the meteorological community argued forcefully that the solution to climate problems was for local and regional decision-makers to make better use of the knowledge that climatologists already had of climate and climate impacts. Long and Iles (1997) demonstrate that assessments of the impacts of climate change on agriculture, sea-level and human health led to community-building. The most active coalition on sea-level rise, the Alliance of Small Island States (AOSIS) has used assessments to increase awareness, to promote coastal zone management programs and to play an active role in the negotiation of the Framework Convention on Climate Change.

The use of IPCC reports as reference documents by a wide number of people in many different disciplines results in three outcomes that affect the research communities. They provide comprehensive research descriptions for particular fields (with all the appropriate references) where none may have existed. They introduce a researcher to a comprehensive view of another field which significantly overlaps with his/her own, i.e., they help to build multi/ interdisciplinary knowledge They also result in a more nuanced, shared understanding of what the "problem" is within the research community.

 This section discusses elements of three factors that can be shown to affect the outcomes of assessment: the context within which the assessment was performed, the process and the content of the assessment. Although these factors are treated separately here, it is clear that they are linked. Context influences both the process and content of an assessment, while process and content also interact. The list of elements that could potentially affect outcomes is long and this section focuses on the most interesting elements of context, process and content observed to have affected the outcomes of climate assessments.

 3.1 Elements Of Context

 Context is the backdrop against which assessments take place, it is a given condition which at a particular time and place cannot be influenced by the producers or users of the assessment. Moreover, context is continually changing: events at one specific point of time can even influence context in a subsequent period. Different kinds of context can be identified: scientific, political, and economic context can all influence the outcome of an assessment.

The state of the scientific community and the extent of its expertise clearly affect the outcome of an assessment. This is demonstrated in the study of assessments in India (Kandlikar and Sagar, 1997), where limited resources are a barrier for performing research on significant aspects of the climate problem for S. and S.E. Asia (i.e. the Monsoon). Agrawala (1997) illustrates this issue in his discussion of the participation of scientists from developing countries in the IPCC process, which leads to the question of whether IPCC should be focusing some efforts on cultivating research communities in the developing world.

The state of scientific knowledge plays a role in determining what questions are asked by assessments and how the issue is framed, as shown in the papers of Long and Iles (1997). The state of scientific uncertainty also affects the outcome of assessments. Fisher-Vanden (1997) emphasizes the extent to which the state of scientific uncertainty can affect the nature of policy instruments considered. For the US, Fisher-Vanden argues that key scientific uncertainties still exist, which lead US policy-makers to argue for a policy instrument that is both flexible and adaptable allowing for changes as uncertainties are resolved.

Several of the case studies, review by the GEA Project, emphasize the extent to which the general political climate and the interests of powerful actors determine the reception of assessments and their outcome. For example, Franz (1997) shows that the Villach conference took place at a time when several initiatives were taking place at the international level to raise awareness of the climate issue. Fisher-Vanden (1997) shows that political factors have played a major role in determining policy instrument choice. The shift to more conservative views in the US has led to increased acceptance of market-based environmental policy instruments (see also Hahn and Stavins, 1991). An "anti-tax" mood in the US was reflected in the rejection of Clinton's proposed BTU tax in 1993 and leaves policy instruments based on taxes (which from an economic point of view may be more efficient) out of the set of feasible policy instruments from which policy-makers could choose.

The position of issue on the political agenda clearly also affects the outcome of an assessment. In the mid-1980s, according to Franz (1997) the Villach process and conclusions were well suited to an agenda development stage and thus the attention to the issue increased. The Toronto Conference in 1988 occurred at a time when the issue of climate change was being discussed in hearings in the US Senate, the G7 meeting which took place the week before the conference mentioned the issue of climate change. The high position of the issue on national and international agendas at the time meant that the outcome of the conference was followed up internationally and in individual countries.

 The linkage of the issue of climate change to other issues (stratospheric ozone depletion, deforestation, energy crises) has also been an important factor in affecting outcomes of assessments.

The state of the economy can have an influence on the outcome of an assessment -- whether there is a recession or a boom affects the acceptance of any policy recommendations. Similarly, the composition of the economy, e.g., whether the economy is energy intensive or not also has an effect. For example, Fisher-Vanden (1997) points out that the US economy is highly fossil fuel intensive, which leads to the rejection of any policy which significantly restricts fossil fuel use. It is believed that significantly restricting fossil fuel use would not only be detrimental to the US domestic economy but would also have major implications for US competitiveness in the global economy, a primary concern of US policy-makers.

Available resources for assessment activities also influence their outcomes. This factor is emphasized by Kandlikar and Sagar (1997), who note that the nature of the north/south relationship results in donor-driven assessments, which tend to focus on issues that are of primary interest to developed nations, and a concomitant lack of funds and attention to issues of particular interest to developing countries.

In addition to scientific, political and economic factors, the case GEA studies identified other contextual factors that influence the outcome of assessments. An important factor is the way that the science and policy communities are linked in a country (or internationally). Kandlikar and Sagar (1997) point out that in India there is a lack of formalized mechanisms for experts to connect to policy makers (especially in the environment area).

Historical experience is another relevant contextual factor. Fisher-Vanden (1997) points out that in the US experience with past instruments is instrumental in the choice of policy instrument. Much of the stimulus behind the US current proposal for an international tradable permits program is based on the experience with the Acid Rain SO2 Allowance Trading program. Historical experience is also a major factor in explaining the recent shift in emphasis away from joint implementation and institutional instruments such as the GEF to an international emissions trading program.

A number of elements of content affect the outcomes of assessments. The quality and relevance of the material in an assessment determine whether the assessment can lead to a policy shift or move the issue further along. These elements are of course strongly influenced by process, which is discussed further below. Other elements that are of importance are the comprehensiveness, scale and scope of the assessment, including linkages to other issues.

The treatment of uncertainty and of extreme outcomes are also important elements of content. Patt (1997) classifies assessments into three types -- agreement, advisory and advocacy -- and shows that the treatment of an extreme event (melting of the West Antarctic Ice Sheet) differs significantly between these types. This illustrates the limitations that users of assessments have in understanding issues of risk and uncertainty. The treatment of uncertainty clearly differs depending on who is doing the assessment and for what purpose. The users of the assessment need to know what type of assessment they are using (i.e. its origins) in order to interpret the treatment of risk and uncertainty.

The outcome of an assessment is also influenced by the extent of inclusion of conclusions and recommendations and, more generally, by the ratio of technical vs. policy-related material. There is evidence of this in the discussions of the Villach 1985 assessment (Franz, 1997) and the work of the IPCC (Agrawala, 1997). In the latter case, the "policy-makers summaries" provide useful consensus documents but limit the extent to which uncertainties can be communicated.

The GEA case studies identify four main elements of process that affect the outcomes of assessments: the design of the assessment, participation, the communication of the results of the assessment (intended and unintended) and the starting point of the assessment (i.e. at whose initiative was it carried out?)

 The design of the assessment is clearly linked with context -- whether it is a one-shot assessment or an iterative process depends on scientific context as well as economic context.

A major point in the design of the assessment is the treatment of peer review. Agrawala (1997) discusses in detail the pros and cons of the IPCC peer review process. Concluding that the IPCC peer review is more comprehensive by many orders of magnitude than in an average journal, Agrawala shows how the mechanism has evolved since IPCC started and how requirements for peer review differed for the three Working Groups for the first assessment report, because of the different "expert communities" involved. Agrawala notes that between 1990 and 1992 a number of political factors forced the IPCC to ensure that its peer review process was formalized to carry more credibility with government negotiators. A formal set of rules governing peer review was adopted in 1993. Despite these rules, Agrawala points out that there are loopholes which have been exploited by lobbying groups. A major problem is what one IPCC author calls the problem of the "silent majority" -- even if a majority of reviewers support certain conclusions they are unlikely to mention that as part of their review comments, while the few reviewers who want to have a certain point changed explicitly voice their dissatisfaction in their reviews. "This may result in changes in the document based essentially on minority opinions", (Agrawala, 1997). Finally, Agrawala identifies one further problem with the IPCC peer review process -- the fact that the action on specific comments is basically left to the discretion of the writing teams, which might be dominated by opinionated persons who reject certain critical review comments.

 A further element of process for the IPCC assessments is the line by line consensus approval of policy-maker summaries, which is an extremely political process and strongly influenced by political and economic context. The process leads to statements based on "least common denominator conclusions written in carefully hedged language", (Agrawala, 1997).

 The participation in an assessment in terms of who gets to participate and what role they play (authors, consultants, discussants, peer reviewers etc.) obviously affect the outcome of the assessment. A important factor discussed in two of the GEA case studies is the participation of researchers from the developing countries. Kandlikar and Sagar (1997) point out that prior to the formation of the IPCC, Indian scientists and analysts were hardly involved in efforts to raise awareness about climate change in international policy circles. Kandlikar and Sagar conclude that although participation by scientists from developing countries in IPCC assessments has increased, it is still heavily skewed in favor of some industrialized countries. Developing country input does not seem to shape the content and process of the IPCC assessment.

 The study of sub-national agriculture impact assessments (Cash, 1997) shows the importance of ensuring participation of stakeholders in assessments, since the evidence suggests that the results of global scale assessments are not trickling down to local level decision-makers.

 Other elements of participation that are observed to affect the outcomes of assessment include: involvement of governments (Franz, 1997; Agrawala, 1997), international participation (Franz, 1997); inclusion of a broader scientific community; and direct policy maker involvement (Fisher-Vanden, 1997).

 A further significant element of process is the communication of the assessment - intended and unintended. Franz (1997) documents some of the interesting pathways along which the Villach 1995 assessment was communicated. This communication was an important factor in determining the outcome of the assessment, i.e. in increasing public and political awareness of the issue. In contrast, poor distribution of assessment results can be expected to lead to negative outcomes.

 The final element of process identified above is the origin of the assessment. The outcome of an assessment is affected by the factors which govern its initiation. For example, Fisher-Vanden (1997) shows the outcome of assessments that were done at the request of policy-makers. This differs from the outcome of processes initiated by a proactive strategy of stakeholders who identify a problem and ask for an assessment to help reach a solution (Cash, 1997).

The previous section highlighted certain elements of the context, content and process of assessments that have been observed to affect the outcomes of climate assessments. This section identifies six pitfalls that are strongly related to the main elements discussed above and have limited the effectiveness of assessments in the past.

Ignoring the political, economic, scientific or general context within which an assessment is performed can lead to a failure of the assessment from the point of view of the sponsor and the producer. For example, Fisher-Vanden (1997) shows that in the US political and economic factors have played a dominant role in policy instrument choice, an assessment that ignored these factors would produce results that would be irrelevant for immediate policy making. An assessment that ignores context can, however, be effective in the longer term for policy making or in the short term for shifting the focus of debate.

Wrongly assuming the relevance of a particular assessment approach for particular groups of potential users will lead to a failure of the assessment from the point of view of the user. For example, Cash (1997) shows that global level assessments of agricultural impacts ignore the needs of potential users (e.g. the farmers who will have to implement adaptive responses at a local or regional level).

Ignoring past experience with policy instruments or past assessments that have been carried out will lead to a failure of the assessment from the point of view of the user. Kandlikar and Sagar (1997) point out that the donor-driven nature of assessments in India means that the work may be of limited use for internal policy or capacity building and that multiple reports might be written on the same issue (sometimes by the same institution) resulting in a loss of time, effort and talent. Fisher-Vanden (1997) concludes that the importance of "institutional history" should not be overlooked. This means that assessments must account for policy instruments already in place when deciding which to add to the portfolio.

Selecting the wrong participants in the assessment (experts, stakeholders, consultants, peer reviewers etc.) will lead to a failure of the assessment from the point of view of the sponsor, the producer and the user. The case studies of Kandlikar and Sagar (1997), Agrawala (1997) and Cash (1997) show the importance of appropriate participation for the outcome of an assessment.

A mismatch of international assessment requirements and national capabilities will lead to a failure of the assessment from the point of view of all actors. Kandlikar and Sagar (1997) show that while the IPCC assessment process requires certain resource, infrastructure, and operating procedures within the governments of participating countries, this is difficult for a developing country like India. Most of the Ministries in India are caught up with other concerns and climate change is low on their priority list (if present at all); the limited number of administrators and researchers are involved in multiple activities and cannot always find the time for quick response and detailed consideration that the IPCC process often needs.

Poorly planned outreach and communication of the results of the assessment will lead to a failure of the assessment from the point of view of the sponsor and the producer. If the report of an assessment is expensive, people in developing countries will not be able to purchase it and the results of the assessment will not feed into the debate in those countries (Kandlikar and Sagar, 1997). Some assessments had excellent follow-up, e.g. Villach 1985 (Franz, 1997). Some assessments never received adequate distribution and their results were thus forgotten or never seen, e.g. the first World Climate Conference (Jaeger, personal communication).

This paper focuses on the outcomes of assessments and the pitfalls that have negative effects on the outcomes of assessments. The empirical evidence presented is derived from GEA studies of assessments in the climatic change area over the past 20 years or more. The avoidance of these pitfalls could contribute substantially to improvements in future assessments. The points raised in the paper show that from the perspective of "improving assessment processes and outcomes" there are several concrete issues that can be discussed further, in particular to see whether the issues raised by looking at climate assessments are more broadly applicable.  

• what are the various goals for assessments?  
• what criteria can be used to evaluate the success and failure of assessments ? 
• what are the most important factors shaping those outcomes? 
• what are the pitfalls most likely to undermine assessments? 

This paper has shown that the answers to the first two questions are very much dependent on the actor (e.g. producers, users and sponsors) being addressed. The answers to the third question can be focused on elements of context, content and process, bearing in mind that these interact. The observations on pitfalls are directly drawn from the discussion of factors that can affect the outcome of an assessment and indicate that failures to address certain elements of context, content and process have been pitfalls into which climate assessments have fallen. The further exposition of this idea would benefit from additional discussion of questions such as: Are the pitfalls identified in this paper found more generally? Are there other important pitfalls? Have there been conscious efforts to avoid these pitfalls in the past?