7.3.6.3 Assessment of methodological quality

Assessment of methodological quality, or critical appraisal, is a process conducted in systematic reviews to establish the internal validity and risk of bias of studies that meet the review inclusion criteria. JBI has developed a number of tools for assessing the quality of various quantitative study designs that are appropriate to use in systematic reviews assessing questions of etiology & risk (see Appendix II).

The protocol should indicate which tool is going to be used that match the included study designs when determining methodological quality of papers to include in the review. JBI appraisal tools should be used preferentially; if not clear reasoning should be provided. Critical appraisal tools should be cited in the protocol and should be appended if the tools are modified in any way. Critical appraisal must be conducted by two reviewers independently of each other. The reviewers should then meet to discuss the results of their critical appraisal for their final appraisal. If the two reviewers disagree on the final critical appraisal and this cannot be resolved through discussion, a third reviewer may be required.

When detailing the methods of the review report, the section on appraisal should detail the approach to critical appraisal, not the assessment results, and should be consistent with the protocol. The approach to critical appraisal process should include information on what constitutes acceptable levels of information for appraisal and whether the decision to include or exclude studies following critical appraisal is based on meeting a predetermined proportion of criteria or weighing criteria differently. The authors of the review should state a priori in the review protocol the criteria used to determine the inclusion or exclusion of poor quality studies. The authors have to make explicit and agree on criteria to determine whether a study is of good, moderate or poor quality, and based on these criteria or a combination of criteria, the authors can decide whether to include only good quality studies or all studies irrespective of the quality. However, the importance of these criteria (e.g. selection, measurement bias, confounding) will vary with study type and problems specific to the review question.

The report should detail the criteria that were considered when determining the methodological quality of papers considered for inclusion in the review. In the systematic review, appraisal questions should be presented with the results, or appended.