## Abstract

Background: systematic reviews of intervention effects may draw conclusions based on effect direction rather than statistical pooling of effect estimates. The Cochrane Handbook of Systematic Reviews of Interventions, 2018 (Handbook) now offers guidance on alternative methods for synthesis without meta-analysis. This guidance includes approaches to vote counting based on effect direction, and use of a sign test to determine whether there is any statistically significant evidence of an effect based on effect direction across studies.

Objective: to apply the new Handbook guidance on alternative synthesis methods based on effect direction and

the use of sign tests.

Methods: we re-analysed data for three key outcomes from a published Cochrane Review, which exclusively used

synthesis of effect direction and in which multiple similar outcomes from single studies were incorporated. We

removed reference to statistical significance from the format of the effect direction plot and from the algorithm

for overall within-study effect direction. We used a sign test to examine the probability of observing the given

pattern of positive effect direction across studies if the null hypothesis of even distribution of positive and

negative results were true. We compared the conclusions from this synthesis to the conclusions of the original

review.

Results: the revised effect direction plot (Figure 1) presents results from a synthesis of 10 studies based on the

revised algorithm. Removal of statistical significance from the algorithm to determine an overall effect direction

for multiple similar outcomes within a single study did not change the assessment of effect direction. The sign test

suggests that positive trends in effect direction across studies are statistically significant for housing condition (P

= 0.0039), but not for general health or respiratory health (P = 0.2188). These results are consistent with the

conclusions of the original review, which stated that housing improvements can lead to improvements in health

but that the evidence is inconclusive. We could not include studies in which the effect direction was conflicting or

unclear in the sign test, which reduced the power of the test.

Conclusions: the modified effect direction plot with the addition of a sign test is a feasible method of synthesising

the best available evidence when meta-analysis is not possible and may improve transparency for reviews relying

on effect direction. However, use of the sign test for small groups of studies may be problematic due to lack of

power. Furthermore, the sign test does not consider study size, and risks overlooking evidence where the effect

direction is unclear. The use of a sign test to assess trends in effect direction across studies demonstrates that

vote counting based on effect direction should be interpreted with caution when the number of studies is small

and/or effect direction in some studies is inconsistent.

Patient/consumer involvement: none (application of statistical methods)

Objective: to apply the new Handbook guidance on alternative synthesis methods based on effect direction and

the use of sign tests.

Methods: we re-analysed data for three key outcomes from a published Cochrane Review, which exclusively used

synthesis of effect direction and in which multiple similar outcomes from single studies were incorporated. We

removed reference to statistical significance from the format of the effect direction plot and from the algorithm

for overall within-study effect direction. We used a sign test to examine the probability of observing the given

pattern of positive effect direction across studies if the null hypothesis of even distribution of positive and

negative results were true. We compared the conclusions from this synthesis to the conclusions of the original

review.

Results: the revised effect direction plot (Figure 1) presents results from a synthesis of 10 studies based on the

revised algorithm. Removal of statistical significance from the algorithm to determine an overall effect direction

for multiple similar outcomes within a single study did not change the assessment of effect direction. The sign test

suggests that positive trends in effect direction across studies are statistically significant for housing condition (P

= 0.0039), but not for general health or respiratory health (P = 0.2188). These results are consistent with the

conclusions of the original review, which stated that housing improvements can lead to improvements in health

but that the evidence is inconclusive. We could not include studies in which the effect direction was conflicting or

unclear in the sign test, which reduced the power of the test.

Conclusions: the modified effect direction plot with the addition of a sign test is a feasible method of synthesising

the best available evidence when meta-analysis is not possible and may improve transparency for reviews relying

on effect direction. However, use of the sign test for small groups of studies may be problematic due to lack of

power. Furthermore, the sign test does not consider study size, and risks overlooking evidence where the effect

direction is unclear. The use of a sign test to assess trends in effect direction across studies demonstrates that

vote counting based on effect direction should be interpreted with caution when the number of studies is small

and/or effect direction in some studies is inconsistent.

Patient/consumer involvement: none (application of statistical methods)

Original language | English |
---|---|

Pages (from-to) | 209-210 |

Journal | Cochrane Database of Systematic Reviews |

Volume | 2020 |

Issue number | Suppl.1 |

DOIs | |

Publication status | Published - 31 Jan 2020 |