A guinea pig's tale: learning to review end-to-end marine ecosystem models for management applications

ICES Journal of Marine Science ICES Journal of Marine Science (2016), 73(7), 1715– 1724. doi:10.1093/icesjms/fsw047 Quo Vadimus A guinea pig’s tale: learning to review end-to-end marine ecosystem models for management applications Isaac C. Kaplan 1* and Kristin N. Marshall 2 1 *Corresponding author: tel: +1-206-302-2446; fax: +1-206-860-3394; e-mail: Kaplan, I. C., and Marshall, K. N. A guinea pig’s tale: learning to review end-to-end marine ecosystem models for management applications. – ICES Journal of Marine Science, 73: 1 715– 1724. Received 10 June 2015; revised 3 March 2016; accepted 7 March 2016. A shift towards ecosystem-based management in recent decades has led to new analytical tools such as end-to-end marine ecosystem models. Endto-end models are complex and typically simulate full ecosystems from oceanography to foodwebs and fisheries, operate on a spatial framework, and link to physical oceanographic models. Most end-to-end approaches allow multiple ways to implement human behaviours involving fishery catch, fleet movement, or other impacts such as nutrient loading or climate change effects. Though end-to-end ecosystem models were designed specifically for marine management, their novelty makes them unfamiliar to most decision makers. Before such models can be applied within the context of marine management decisions, additional levels of vetting will be required, and a dialogue with decision makers must be initiated. Here we summarize a review of an Atlantis end-to-end model, which involved a multi-day, expert review panel with local and international experts, convened to challenge models and data used in the management context. We propose nine credibility and quality control standards for endto-end models intended to inform management, and suggest two best practice guidelines for any end-to-end modelling application. We offer our perspectives (as recent test subjects or “guinea pigs”) on how a review could be motivated and structured and on the evaluation criteria that should be used, in the most specific terms possible. Keywords: Atlantis, best practices, end-to-end models, evaluation criteria, marine ecosystem models, model performance, peer review. What are end-to-end models, and why is it complex to review them? In recent years, the shift toward ecosystem-based management of marine resources (Pikitch et al., 2004; McLeod and Leslie, 2009) has led to the development of new analytical tools that simultaneously consider multiple human impacts and multiple species. Unlike traditional single species fishery stock assessments (Maunder and Punt, 2013; Methot and Wetzel, 2013) used for tactical management such as setting annual quotas, ecosystem models are typically viewed as strategic tools for exploring qualitative patterns, conducting risk assessment, and ranking policy alternatives (Plagányi, 2007; Fulton et al., 2014). Critically, these tools are intended to capture trade-offs between species, fisheries, and human uses that may occur given future marine management actions or environmental conditions. Evaluation of these trade-offs across species and sectors is a central part of ecosystem-based management such as the Marine Strategy Framework Directive (European Commission, 2008) and US national ocean policy (Interagency Ocean Policy Task Force of the White House Council on Environmental Quality, 2010; Obama, 2010). End-to-end marine ecosystem models (Travers et al., 2007; Fulton, 2010; Rose et al., 2010) are one type of modelling tool that simulates full ecosystems from oceanography to foodwebs and fisheries. These are spatially explicit simulation models that are forced by physical oceanographic models, and that include human actions such as fishery catch, fleet movement, nutrient loading, or climate change. Examples include the Atlantis model (Fulton et al. Published by Oxford University Press on behalf of International Council for the Exploration of the Sea 2016. This work is written by (a) US Government employee(s) and is in the public domain in the US. Northwest Fisheries Science Center, National Marine Fisheries Service, NOAA, 2725 Montlake Boulevard E., Seattle, WA 98112, USA University of Washington School of Aquatic and Fisheries Sciences, PO Box 355020, Seattle, WA 98195, USA 2 1716 Alaska, reviewed by the Center of Independent Experts in 2005 and 2011 (http://www.nefsc.noaa.gov/ecosys/modeling_review.html, https://www.st.nmfs.noaa.gov/science-quality-assurance/cie-peerreviews/cie-review-2005.), and periodic review of Alaskan ecosystem modelling by stock assessment teams and fishery management council subcommittees (Link et al., 2010b). However, the established process for reviewing single species stock assessment models within fisheries management (e.g. NOAA Northeast Fisheries Science Center, 2014; Pacific Fishery Management Council 2012) can serve as a rough template for review of end-to-end models (Figure 1), and the decades of communication between managers and scientists regarding single species models serves as a reminder that much work is to be done by end-to-end modellers in this arena. Our perspective benefits from these previous and ongoing efforts. Below, we summarize a review of an Atlantis end-to-end ecosystem model. The review was held in Seattle, Washington, USA, during June and July 2014. We describe this process to consider how such reviews should be structured and organized (Figure 1), and we propose a very specific set of model evaluation criteria for marine end-to-end models. These differ from previous sets of best practices provided by other marine and terrestrial ecosystem modelers (FAO, 2008; Townsend et al., 2008; Link et al., 2010b; Schmolke et al., 2010; Bennett et al., 2013), which are extremely valuable but not necessarily tailored to end-to-end models. We offer our perspective (as recent test subjects or “guinea pigs”) to other ecosystem modellers, first in terms of how a review could be motivated and structured, and second regarding the evaluation criteria that should be used, in the most specific terms possible. Full materials from the 2014 Atlantis review, including agendas, terms of reference, and reviewer reports, are available at http://www.nwfsc.noaa. gov/research/divisions/cb/ecosystem/marineecology/aem.cfm, and below we synthesize lessons learned and potential evaluation criteria. Motivation for reviewing ecosystem models Though end-to-end ecosystem models have clear utility for marine management, these new purposes, behaviours, and aspects of uncertainty inherent in end-to-end models are unfamiliar to most decision makers. To provide advice within the context of US marine management, Link et al. (2010b) note that ecosystem models’ “credibility will need to be established and the rigor of quality control/ assurance and peer review will need to be at a comparable level as what is done for single species and protected species stock assessments.” By “peer review,” the authors imply multi-day, ex (...truncated)