3.6.6 Assessment of methodological quality

Qualitative studies that are eligible for inclusion in the review must be assessed for methodological quality. There are a variety of checklists and tools available to assess research syntheses and systematic reviews. Most checklists use a series of criteria that can be scored as being “met” or “not met” or “unclear” and in some instances as “not applicable”. The decision as to whether or not to include a study can be made based on meeting a pre-determined proportion of all criteria, or on certain criteria being met. It is also possible to weight certain criteria differently. Decisions about a scoring system or any cut-off for exclusion should be made in advance and agreed upon by all reviewers before critical appraisal commences. The protocol, therefore, should detail how selected studies will be assessed for quality, e.g. use of a predetermined cut off score.

All included studies need to be critically appraised using the standard JBI critical appraisal instrument for qualitative research that is available in Appendix 3.1 of this chapter (further details regarding the appraisal questions can be found in Appendix 3.2). The assessment criteria are built into JBI SUMARI. The tool is designed to be used with two independent reviewers conducting the critical appraisal of each research synthesis selected. Reviewers are blinded to each other’s assessment and assessments can only be compared once initial appraisal of an article is completed by both reviewers. Where there is a lack of consensus, discussion between reviewers should occur. In some instances it may be appropriate to seek assistance from a third reviewer. The source of the JBI critical appraisal tool for research syntheses should be cited in the protocol.

NB: If the best available evidence for your question is narrative text, expert opinion or policy rather than qualitative research, these studies should be analysed using the text and opinion module of JBI SUMARI. Such reviews become a JBI Textual Evidence Review (see 5. Systematic Reviews of Textual Evidence) rather than a qualitative review of evidence, and therefore the review title, question and criteria should be reviewed against the expectations of a textual evidence review.