Eileen Shea (with contributions from Shardul Agrawala, Milind Kandlikar, Marybeth Long)

This Chapter provides an integrated summary of the deliberations of three working groups organized during the second half of the 1997 GEA Workshop to provide guidance on the emerging national assessment activity being initiated in the context of the U.S. Global Change Research Program (USGCRP). These working group discussions were organized at the request of our Federal agency sponsors in the hopes that some of the insights developed during the week in Bar Harbor might be useful in guiding the evolution of their new activity. To facilitate discussion, Bar Harbor participants were provided with copies of a June 13, 1997 "Proposal for SGCR Preparation and Review of National Assessments" and "Draft Terms of Reference for a National Assessments Working Group of the Subcommittee on Global Change" (also dated June 13, 1997). In addition, the USGCRP agency representatives still participating in Bar Harbor (Richard Ball, Mike Hall, and John Houghton) were present during working group deliberations to field questions and provide additional insights into Federal planning for a national global change assessment.

 The working group discussions of the national global change assessment were organized in the context of the following three challenges:

 Chair: James McCarthy

Rapporteur: Milind Kandlikar

Chair: William Clark

Rapporteur: Marybeth Long

Exploring Issues Related to the Incorporation of Appropriate Review and Evaluation Mechanisms in the Design and Conduct of a National Assessment Process

Chair: Jill Jaeger

Rapporteur: Shardul Agrawala

The following summary represents a synthesis of the key findings and recommendations drawn from the rapporteurs reports from these three working groups.

 Underlying all of the national assessment discussions was an assumption that the proposed new USGCRP assessment activity would be designed and implemented as a dynamic social process which recognizes that: 

• the process of deliberation and interaction among participants will be at least as important in affecting outcomes as any reports which are produced; and 
• the process should be viewed as long-term and iterative -- which sustains an interactive dialogue between scientific and stakeholder/user communities and produces a series of reports whose focus will change over time as the process matures.

 Thus an effective national assessment process will include a systematic sequence of activities through which the scientific community can engage in a formal dialogue with user groups and policy makers in a shared exploration of the challenges and opportunities associated with addressing the consequences of global change.

 All three working groups addressed the need to clearly define the scope and objectives of a national assessment process at the outset. In this context, the 1997 GEA Workshop deliberations would suggest that the emerging assessment activity should be focused on "global change" broadly-defined rather than a more limited assessment of "climate change". In part, this recommendation reflects a recognition that the Global Change Research Act of 1990 (P.L. 101-606), which provides the legislative mandate for the proposed national assessment activity, embraces a broad definition of global change. In addition, the discussions in Bar Harbor suggested that: (i) "global change" encompasses a broad spectrum of issues grounded in established disciplines which are likely to attract diverse experts, policy makers and public interest groups while avoiding the dangers associated with a single set of disciplinary or other interests "capturing" the assessment process; (ii) a "global change" (vs a more limited focus on "climate change") assessment process could break new ground and potentially avoid or diminish pre-existing political controversy by setting climate change in a broader context of multiple environmental stresses; and (iii) a "global change" assessment process may help in framing a public understanding of global change as an important issue with demonstrable regional implications.

 The 1997 GEA Workshop participants suggested that such a national assessment process could be viewed as an exciting and innovative vehicle for exploring the implications of global environmental change for particular people and places in the United States. In this context, a national assessment process could provide information about how global environmental change will manifest itself through natural and social systems at subnational levels--information that is essential if the assessment process aims to address the implications of global environmental change for policy formulation and management decision making at local, regional and national levels. It was felt that using a regional focus to organize a national global change assessment process would have a number of additional benefits, including exciting and unique opportunities to:

• engage the public in the assessment process and, conversely, use the assessment process to improve public understanding about the implications of global environmental change in their local areas; 
• enable the scientific community to survey available research results on the local consequences of global environmental change, link related research efforts and identify critical information gaps and research priorities where further analysis would be especially valuable; 
• utilize local knowledge and expertise in both substantive analyses and communication with users while, at the same time, supporting local, regional and national capacity-building by linking experts from different areas; encouraging new and innovative research; and establishing more effective mechanisms to sustain a two-way dialogue between assessors and assessment users at various national and subnational levels, including; and  
• explore how a number of global environmental factors interact with each other and with natural and social systems--i.e., set issues related to global environmental changes in the context of other environmental, economic and social stresses in a given region.

The participants in the 1997 GEA Workshop noted that there are numerous ways to delineate geographic "regions" including political boundaries (e.g., states, Congressional districts), federal agency jurisdictions (e.g., EPA's regions), ecosystem features and cultural boundaries. Rather than advocating a preferred definition or a top-down predetermination of a set of regions which cover the entire US territory, the Workshop participants encouraged the USGCRP agencies to pursue an innovative and experimental approach to defining regions and illuminating interactions between or among regions. In particular, one Working Group suggested the use of a Request for Proposal (RFP) process to assist national assessment managers in discovering the most critical problems and issues in various regions. Based on such an RFP approach, assessment managers could identify potential linkages between proposed projects and encourage collaborations to develop those linkages.

 Discussions in Bar Harbor suggested that a regional assessment focus could be complemented with sectoral analyses and a national-level synthesis which enabled standardization of data and methods and provided for comparisons across regions. One specific suggestion in this context involved the use of a template or "environmental checklist" which: (i) describes activities or analyses that each regional assessment should undertake; (ii) establishes criteria for selecting sectors to be studied within a region; (iii) provides data and standard reference scenarios for each regional assessment to utilize and standard assessment methodologies that might comprise some, but not all, of the regional study; (iv) articulates a common terminology/shared vocabulary; and (v) establishes standardized data and reporting requirements to assist in aggregating assessment results obtained at the regional level. In addition, the 1997 GEA Workshop participants encouraged the USGCRP agencies to undertake a national assessment process which includes considerations of three key elements of the regional consequences of global change -- impacts, adaptation and mitigation.

The definition of clear objectives for a national assessment process was viewed as an important early step to both guide implementation and provide benchmarks to evaluate performance. Discussions in Bar Harbor identified the following multiple objectives for a national global change assessment process: 

• linking global-scale research and assessment processes with knowledge of local resources and vulnerabilities to improve understanding of global change processes and impacts at both scales; 
• forging direct links between scientific and decision making communities and facilitating the dissemination of information across a wide spectrum of such communities; 
• enhancing interaction among diverse research communities to formulate and undertake research agendas which encourage cross-disciplinary collaboration; 
• increasing public awareness by catalyzing public interest in the local implications of global change; 
• building local capacity for dealing with global change at the local, regional and national levels; and 
• producing reports which describe the current state of understanding of the national implications of global change to inform national and subnational decision making.

 The national assessment working group deliberations at the 1997 GEA Workshop produced a number of suggestions related to issues of participation. Noting that issues of participation are closely linked to the defined scope of an assessment process, the Bar Harbor Working Groups highlighted the value of broad ownership across a wide range of groups and pointed to the value of an initial workshop to delineate the scope of an assessment process and cast a wide net for broad, initial participation. It is interesting to note that the USGCRP agencies have begun such a series of regional "scoping" workshops as a first step in the development of a national global change assessment process. 

Other aspects of assessment participation highlighted during the 1997 GEA Workshop deliberations included: 

• the role that participation plays in influencing the public/political legitimacy of an assessment process, including the importance of effectively engaging private industry and providing for the involvement of "grass-roots" institutions and programs; 
• addressing the challenges associated with defining an appropriate set of stakeholders and effectively identifying their concerns and information needs, with suggestions related to drawing upon previous stakeholder-based initiatives, setting up appropriate mechanisms for "self-identification" on the part of stakeholders, and recognizing that different stakeholders may have different goals; 
• the importance of effective national and regional leadership in an assessment process, particularly as it relates to maintaining both scientific credibility and stakeholder confidence; 
• the value of using existing mechanisms, institutions and local expertise for both scientific contributions and public participation/credibility; 
• the importance of establishing credibility and engaging interest across a range of research and user communities which requires that attention be given to avoiding a tendency for the assessment process to be captured by a specific set of disciplinary or other interests; and 
• recognition that the breadth and range of participation will vary during different stages in the assessment process, ranging from an initial scoping phase where input from a wide range of users is needed to broadly define the problem; through an analysis phase where communities of experts address specific scientific, technical or analytical issues; to a synthesis phase which requires a shift back to increasing interaction between assessors and users as they collectively, and iteratively, evaluate the extent to which the analysis meets the requirements defined in the scoping phase.

 The other major component of a national global change assessment process which received specific attention during the 1997 GEA Workshop involved the importance of incorporating appropriate mechanisms for review and evaluation. Two underlying themes emerged from the working group deliberations: (i) a recognition of the value of incorporating mechanisms for both internal (self) and external evaluation; and (ii) the value of involving stakeholder/user communities in the design and implementation of effective mechanisms for review and evaluation throughout the assessment process. Most of the recommendations related to review mechanisms were developed after exploring the lessons learned from a number of illustrative models, including NRC reviews, the IPCC review process, and publication reviews for interdisciplinary journals. 

 As discussed in Bar Harbor, "review" referred to a/the mechanism through which a wider community of experts and stakeholders have an opportunity to analyze assessment documents (reports) for technical accuracy, coverage of relevant topics and decision making relevance. In this context, the 1997 GEA Workshop participants noted the following: 

• distinct phases of the assessment process will require different approaches to review--employing different techniques, criteria and different combinations of expert and stakeholder participation; 
• independent, third-party monitors of the review process can be useful and are recommended; 
• clients and stakeholders should help define criteria for review and nominate reviewers but it is not recommended that they serve as review monitors or control the actual selection of reviewers; 
• the complex nature of global change will likely require a process which involves multiple rounds of review; 
• experience in other fields suggests that credibility and utility can be enhanced by openly and effectively addressing and communicating issues related to scientific uncertainty, including provisions for the explicit inclusion of dissenting opinions; and 
• high standards of accountability are important to the long-term credibility and viability of a review process and, to that end, it is recommended that the USGCRP maintain complete records of the review process and make those records easily accessible to assessment practitioners, users and scholars.  

Recognizing that undertaking a national global change assessment involves a commitment to a long-term, iterative process, the Workshop participants also identified the value of incorporating appropriate mechanisms for periodic evaluation of the assessment process where "evaluation" refers to a mechanism for performance appraisal of the assessment process as a whole or specific projects undertaken as elements of the national assessment. In this context, the Bar Harbor participants emphasized the importance of incorporating both: (i) "post-hoc" evaluations after the issuance of a major report or the conclusion of some key stage in the assessment process; and (ii) interim, re-evaluations to allow for mid-course corrections during the assessment process. The Workshop participants suggested that the emerging national global change assessment process might consider a mix of internal and external evaluation mechanisms, with internal (self) evaluations and re-evaluations throughout the process punctuated by periodic external evaluations at critical points.  

Regardless of the specific mechanisms employed, the 1997 GEA Workshop participants emphasized the importance of a clear and early articulation of assessment goals and objectives to provide a/the primary benchmark for evaluation. In this context, the Workshop participants provided some illustrative examples of possible evaluation criteria for a national global change assessment process, including: 

• assessor-stakeholder/user interactions; 
• contributions to capacity-building; 
• policy-relevance/impact on decision making; 
• quality of participants and level of participation by key groups; 
• appropriate engagement of stakeholder/user communities; and 
• effectiveness in disseminating assessment results.

 The individual Working Group deliberations and associated plenary session discussion of key findings and recommendations highlighted some of the unique opportunities and special challenges facing those responsible for the design and implementation of an effective National Global Change Assessment Process. The high level of interest among the 1997 GEA Workshop participants suggests that additional efforts to actively engage the communities of assessment scholars, practitioners and users will prove beneficial, particularly in the planning and initial implementation stages. While the discussions in Bar Harbor were certainly not comprehensive, they provided a unique opportunity for a diverse group of experts to consider the design of the proposed global change assessment process in the context of over twenty years of climate assessment experience.