Workshop registration closed! The facility to register for workshops has closed. If you have not registered for a workshop, workshop listings will be on the Message Board in the conference foyer of The Burlington Hotel for you to register for workshops that still have spaces available.

How to identify and assess the impact of outcome selection bias in meta-analysis.

Paula Williamson.
Centre for Medical Statistics and Health Evaluation, University of Liverpool, Liverpool, UK.
Doug Altman.
Centre for Statistics in Medicine, University of Oxford, Oxford, UK.

prw doesnotexist@cochrane.org liv.ac.uk


Objective: To provide the review author with a background to the problem of outcome selection bias and how it might lead to misleading conclusions, to demonstrate how a review author might assess the likely presence of such bias in their review, and to present simple techniques for assessing the robustness of the meta-analysis to such bias.

Summary: Within-study selective reporting bias has been defined as the selection, on the basis of the results, of a subset of the analyses undertaken to be included in a study publication (Hutton and Williamson 2000). Sources of such reporting bias will be described, however, the workshop will focus on outcome selection bias. The effect of within study selective reporting of outcomes will be demonstrated.

Direct empirical evidence for the existence of outcome selection bias is accumulating (Chan et al 2004). In a meta-analysis it is often the case that a total number of k eligible studies are identified but only n report the data of interest. The review author needs to examine the remaining (k-n) studies to establish whether the outcome of interest has been collected but not reported. This should ideally involve contact with the original trialists which may result in missing data being made available or it may confirm that the outcome data were not recorded (Williamson and Gamble 2005). However, it is likely that in a subset of these studies, m (<= k-n) say, no such information is forthcoming.

It is important to assess the level of suspicion that selective non-reporting has occurred in these m studies. Methods for the identification of within-study selective reporting in a meta-analysis and an individual study will be described and illustrated using real examples.

If the level of suspicion is high, a useful first stage is to undertake a sensitivity analysis to assess the robustness to extreme selective reporting. Two methods for such an analysis (Williamson and Gamble 2005, Copas and Jackson 2005) will be illustrated and compared using real examples.

Participants will be encouraged to undertake such assessments for examples provided and to discuss the issues for their own reviews.


Level of knowledge required to attend: intermediate.