Testing the sensitivity of fishery assessment scores to changes in the MSC Fisheries Standard requirements

Fisheries may volunteer to be audited against the Marine Stewardship Council's Fisheries Standard to achieve eco-certification for their seafood products. The Standard uses a hierarchical scoring structure with four nested levels: (i) three principles, (ii) nine components, (iii) 27-29 performance indicators, and (iv) 78-93 scoring issues each evaluated at up to three scoring guideposts. Changes to this structure to revise requirements can affect resulting scores and therefore certification outcomes. As part of the MSC's policy development process, we modelled the scoring process and tested the sensitivity of outcomes to changing scoring issues, either one at a time or simultaneously, mirroring the transition of the Standard from v2.01 to v3.0. Results suggest that scoring outcomes resulting from a change depend on three factors: 1) the number of scoring issues at each guidepost of a performance indicator; 2) the weights applied to scoring criteria; and 3) the scores of unaltered scoring issues compared to the scores of altered scoring issues. Some changes showed cascading effects up the hierarchy and changed predicted certification outcomes from pass to fail. Unfavourable scoring outcomes were predicted to be less severe overall when there were more scoring issues at each guidepost and more performance indicators in successive levels. This approach can be used as a tool in the policy development process to evaluate consequences and focus debate around trade-offs involved with perturbations to a hierarchical scoring structure.