WCSAM will hosted a two day workshop on performance of stock assessment methods prior to WCSAM proper. This workshop provided an opportunity to discuss the performance of stock assessment models with many of the world's leading fisheries scientists in an informal and participatory manner. The workshop plans to synthesize findings about model performance. It drew heavily on the current model testing exercise hosted by WCSAM. Participants were encouraged to join the exercise and explore the performance of various assessment methods in the run up to the workshop. We structured this exercise around 5 steps, which involved exploring actual stock assessment data sets and simulated data sets. The data sets were based on stocks from across the world, which exhibit characteristics that challenge current stock assessment methods.
We received, over 40 unique assessment model fits, covering the 12 stocks.
There were two ways to actively contribute to the workshop:
1. Analysing fits to actual and simulated data: this will address the findings of steps 1 to 4 of the WCSAM approach. Using the results from the fits to actual data, we will explore the properties that challenged the model. We will also consider fits to data simulated with observation error, and fits to data generated with both observation and process error. Discussions will likely revolve primarily around attempting to explain any inconsistencies among assessment model fits. The intention is that by having repeatedly applied multiple methods to the same data sets it should be possible to draw general lessons about the performance of various assessment methods.
2. Open forum on the grand questions: We also plan to give time during the workshop for participants to present either results of work done independently or work from step 5 of our simulation exercise. The work must focus on understanding the performance of assessment models and methods in relation to our grand questions. These relate to the utility of ageing data, exploration of length-based assessments, dome selectivity, changes in selectivity over time, importance of contrast in index time series and retrospective patterns. If you wish to provide input from your analysis into the open forum from independent studies, please email WCSAM@ices.dk with the email subject heading "Workshop Open Forum".
Proposed Structure of workshop
The first morning will be spent examining fits to actual stock assessment data and exploring the diagnostics (steps 1 and 2). This analysis will be done on a stock by stock basis. Approximately 15 minutes will be spent on each stock, with stock leaders providing a five minute overview of the fits and diagnostics, and highlighting the main issues. The afternoon of the first day will review model fits to simulated data for the 5 or 6 stocks that have been most explored (steps 3 and 4). An examination of the fits to pseudo-data will provide tests of assessment model self-consistency (e.g., can an assessment method consistently reproduce the same results?) and across-model consistency (e.g., can an assessment method consistently reproduce the results of another model?). Explaining the bias or lack of bias evident in results will likely require modellers to identify the key features of their fit or model, but may also be informed by the discussion about the fits to real data.
On the morning of day 2, step 5 will be discussed in the Open Forum, both with simulation data and independent studies. This format will require quick fire presentations followed by a broader debate that will allow researchers to prepare discussion themes for the main conference. After lunch the workshop will wrap up the findings of both the simulation workshop and the Open Forum. A concluding session will look towards the take home message to be shared at WCSAM, which would include the possibility of extending the simulation testing exercise into the future.
The workshop will produce a discussion document on the challenges to stock assessment models. This document will be used in the following conference during the session on the workshop and also be core to the leading paper in the ICES Journal volume and form a substantial section in the proposed ICES cooperative research report (CRR). It is hoped that the outcome will provide:
A better understanding of cross-assessment variance (perhaps this swamps within-assessment noise).
Identification of model features that generate important differences in fit. For example, can we stop worrying about whether we assume age comps are lognormal or multinomial and concentrate on patterns in M, selectivity, or q?
A better plan for the future on how best to conduct simulation testing.
