Effect direction in synthesis without meta-analysis: application of Cochrane Handbook of Systematic Reviews of Interventions 2018 guidance

Background: systematic reviews of intervention effects may draw conclusions based on effect direction rather than statistical pooling of effect estimates. The Cochrane Handbook of Systematic Reviews of Interventions, 2018 (Handbook) now offers guidance on alternative methods for synthesis without meta-analysis. This guidance includes approaches to vote counting based on effect direction, and use of a sign test to determine whether there is any statistically significant evidence of an effect based on effect direction across studies.
Objective: to apply the new Handbook guidance on alternative synthesis methods based on effect direction and
the use of sign tests.
Methods: we re-analysed data for three key outcomes from a published Cochrane Review, which exclusively used
synthesis of effect direction and in which multiple similar outcomes from single studies were incorporated. We
removed reference to statistical significance from the format of the effect direction plot and from the algorithm
for overall within-study effect direction. We used a sign test to examine the probability of observing the given
pattern of positive effect direction across studies if the null hypothesis of even distribution of positive and
negative results were true. We compared the conclusions from this synthesis to the conclusions of the original
review.
Results: the revised effect direction plot (Figure 1) presents results from a synthesis of 10 studies based on the
revised algorithm. Removal of statistical significance from the algorithm to determine an overall effect direction
for multiple similar outcomes within a single study did not change the assessment of effect direction. The sign test
suggests that positive trends in effect direction across studies are statistically significant for housing condition (P
= 0.0039), but not for general health or respiratory health (P = 0.2188). These results are consistent with the
conclusions of the original review, which stated that housing improvements can lead to improvements in health
but that the evidence is inconclusive. We could not include studies in which the effect direction was conflicting or
unclear in the sign test, which reduced the power of the test.
Conclusions: the modified effect direction plot with the addition of a sign test is a feasible method of synthesising
the best available evidence when meta-analysis is not possible and may improve transparency for reviews relying
on effect direction. However, use of the sign test for small groups of studies may be problematic due to lack of
power. Furthermore, the sign test does not consider study size, and risks overlooking evidence where the effect
direction is unclear. The use of a sign test to assess trends in effect direction across studies demonstrates that
vote counting based on effect direction should be interpreted with caution when the number of studies is small
and/or effect direction in some studies is inconsistent.
Patient/consumer involvement: none (application of statistical methods)