5.2.6 Assessment of quality

The goal of critical appraisal is to assess the methodological quality of a study and to determine the extent to which a study has excluded or minimized the possibility of bias in its design, conduct and analysis. The focus on limiting bias to establish validity in the appraisal of quantitative studies is not possible when dealing with textual evidence. In appraisal of text, the opinions being raised are vetted, the credibility of the source investigated, the motives for the opinion examined, and the global context in terms of alternate or complementary views are considered.  Validity in this context therefore relates to what is being said, the source and its credibility and logic; and consideration of the overt and covert motives at play.

There are JBI standardized appraisal tools based on textual type appropriate for JBI reviews of textual evidence. JBI systematic reviews are required to use these JBI standardized appraisal tools. Reviewers should refer in the review protocol to the JBI standardized critical appraisal checklists and provide references for these checklists. It is not necessary to provide these checklists in appendices of the review protocol. If non-JBI appraisal tools are proposed, then these tools should be briefly described and correctly referenced. In this case, an explicit justification for the use of non-JBI appraisal tools should be provided in the review protocol.

Two reviewers should perform independent assessment of retrieved papers using the standardized checklists developed by JBI. Any disagreements are solved by consensus or by the decision of a third reviewer. Reviewers should specify that they plan to report in narrative form and in tables the results of quality assessments, for each aspect of quality for each individual paper and the overall quality of the entire set of included papers. This phase of the review should not be treated as a rapid ‘box ticking exercise’ on checklists, but rather as a complex, profound, critical, systematic, thorough examination of the quality of each included text, a solid foundation for an appropriate synthesis of the results.

The review (and protocol) should specify if and how the results will be used for the exclusion of papers from the review. For example, if papers judged of low quality will be excluded from the review, the details of the circumstances under which such decisions will be made and the explicit criteria or decision rules should be explicitly provided, including explanations for what is considered low quality by reviewers. It is the decision of the review team if they want to exclude from the review papers judged of low quality. Reviewers should explain and justify their criteria and decision rules. The decision as to whether or not to include a text can be made based on meeting a predetermined proportion of all criteria, or on certain criteria being met. It is also possible to weight the different criteria differently. The decisions about the scoring system and the cut-off for inclusion of a paper in the review should be made in advance and be agreed upon by all participating reviewers before assessments commence. The review protocol should specify if and how the results of critical appraisal will be used in the synthesis of the results. JBI reviewers are encouraged to read the article by Porritt et al³⁸ regarding study selection and critical appraisal.

This section of the review should include the results with the three different JBI textual evidence critical appraisal checklists, embedded in the JBI SUMARI software, whether it is narrative, expert opinion or policy (or consensus guideline). The primary and secondary reviewer should discuss each item of assessment for each textual type included in their review. In particular, discussions should focus on what is considered acceptable to the needs of the review in terms of the characteristics of the textual evidence. The reviewers should be clear on what constitutes acceptable levels of information to allocate a positive appraisal compared with a negative, or response of ‘unclear’ or ‘not applicable.’ This discussion should take place before conducting the assessment, as each paper should be assessed independently by both reviewers. The quality assessment tool should be referenced accordingly.

The explanation for the JBI SUMARI text and expert opinion critical appraisal tool is detailed below. Ongoing consideration by the methodology group is to have three separate critical appraisal tools for the different types of text; narrative, expert opinion and policy.

Important note: These critical appraisal tools are presented separately according to the textual source (narrative, expert opinion or policy), but please be aware that these are not currently available in the JBI SUMARI software. However, if you plan to use the separate critical appraisal tools, please cite as detailed. Currently in JBI SUMARI is the Text and Expert Opinion critical appraisal tool.¹¹