9.2.8 Assessment of methodological quality

Research syntheses that are eligible for inclusion in a JBI Umbrella Review must be assessed for methodological quality. Ideally, only high quality systematic reviews should be included in an Umbrella Review. There are a variety of checklists and tools available to assess research syntheses and systematic reviews. Most checklists use a series of criteria that can be scored as being “met” or “not met” or “unclear” and in some instances as “not applicable”. The decision as to whether or not to include a study can be made based on meeting a pre-determined proportion of all criteria, or on certain criteria being met. It is also possible to weight certain criteria differently. Decisions about a scoring system or any cut-off for exclusion should be made in advance and agreed upon by all reviewers before critical appraisal commences. The protocol, therefore, should detail how selected research syntheses will be assessed for quality, e.g. use of a predetermined cut off score.

It is the JBI policy that all systematic reviews need to be critically appraised using the standard JBI critical appraisal instrument for Systematic reviews and Research Syntheses that is available in Appendix 10.1 of this chapter (further details regarding the appraisal questions can be found in Appendix 10.2). For a JBI Umbrella Review the assessment criteria are available for selection in the JBI SUMARI software. The tool is designed to be used with two independent reviewers conducting the critical appraisal of each research synthesis selected. Reviewers are blinded to each other’s assessment and assessments can only be compared once initial appraisal of an article is completed by both reviewers. Where there is a lack of consensus, discussion between reviewers should occur. In some instances it may be appropriate to seek assistance from a third reviewer. The source of the JBI critical appraisal tool for research syntheses should be cited in the protocol (Aromataris et al., 2015).