5.2.8 Textual evidence synthesis
- Carolyn Phillips
The JBI approach to the synthesis of textual evidence derived from sources other than research follows the qualitative evidence approach based on pragmatic and transcendental thought. This process of textual synthesis replicates the JBI approach to the synthesis of qualitative evidence as articulated by Lockwood and colleagues.43 Following critical appraisal and data extraction of the three various types of text: narrative, expert opinion or policy, conclusions will be synthesized together, depending on the nature of the clinical question. If the three various types of text are included in the systematic review, a decision will be made by the review team whether these are presented in their separate textual types, or synthesized together. This should be outlined transparently in the a priori protocol. As the process relates to textual findings rather than numeric data, the need for methodological homogeneity, so important in the meta-analysis of the results of quantitative studies, is not a consideration.
The aim of textual evidence synthesis is to: firstly, assemble conclusions; secondly, categorize these conclusions into categories based on similarity in meaning; and thirdly, to aggregate these to generate a set of statements that adequately represent that aggregation. These statements are referred to as synthesized findings and they can be used as a basis for evidence-based practice. In order to facilitate this process, as with ensuring a common understanding of the appraisal criteria and how they will be applied, reviewers need to discuss synthesis and work to common understandings on the assignment of categories, and assignment to synthesized findings.
This section of the report should include how the findings were synthesized. Where evidence synthesis is possible, textual conclusions should be pooled using JBI SUMARI. The units of extraction in this process are specific conclusions stated by the author/speaker and the text that demonstrate the argument or basis of the conclusion. Conclusions are principal opinion statements embedded in the paper and are identified by the reviewer after examining the text in the paper; the conclusion is the claim or assertion of the author. It is for this reason that reviewers are required to read and re-read the paper closely to identify the conclusions to be entered into JBI SUMARI. Conclusions should be extracted as verbatim statements from the author.
The processes for categorization and formulating synthesized findings mirror that of the JBI SUMARI qualitative approach of synthesis. For a more detailed discussion of synthesis, reviewers are encouraged to read the section on data synthesis for qualitative studies. 43
Data synthesis should involve the synthesis of conclusions to generate a set of statements that represent that aggregation, through assembling the conclusions rated according to their credibility, and categorizing them on the basis of similarity in meaning. These categories should then be subjected to a meta-synthesis in order to produce a single comprehensive set of synthesized findings that can be used as a basis for evidence-based practice. Where textual pooling is not possible the findings can be presented in narrative form.
Prior to carrying out data synthesis, reviewers first need to establish, and then document:
their own rules for setting up categories
how to assign conclusions to categories
how to aggregate categories into synthesized findings.
In JBI SUMARI, a reviewer can add conclusions to a study after an extraction is completed on that paper. The JBI approach to synthesizing the conclusions of textual or non-research studies requires reviewers to consider the credibility (logic, authenticity) of each report as a source of guidance for practice; identify and extract the conclusions from papers included in the review; and to aggregate these conclusions as synthesized findings.
The most complex problem in synthesizing textual data is agreeing on and communicating techniques to compare the conclusions of each publication. The JBI approach uses the SUMARI software, which involves categorizing and re-categorizing the conclusions of two or more studies to develop synthesized findings. Reviewers should also document these decisions and their rationale in the systematic review report. Many textual based reports do not state conclusions explicitly. It is for this reason that reviewers are required to read and re-read each paper closely to identify the conclusions to be generated into JBI SUMARI.
Each conclusion should be assigned a level of credibility, based on the congruency of the conclusion with supporting data from the paper where the finding was found. Textual evidence has three levels of credibility; thus, the reviewer is required to determine if, when comparing the conclusion with the argument the conclusion represents evidence that is:
Unequivocal (U) - relates to evidence beyond reasonable doubt which may include conclusions that are matter of fact, directly reported/observed and not open to challenge.
Credible (C) - relates to those conclusions that are, albeit interpretations, plausible in light of the textual data and theoretical framework. As the conclusions are interpretive they can be challenged.
Not Supported (NS) - is when the conclusions are not supported by the textual data.
In the systematic review report, it may be set out in the following way.
Papers were pooled using JBI SUMARI. This involved a three stage process: 11
Extraction of Level 1 author’s conclusions from full text articles and rating each according to its assessed validity (unequivocal, credible, not supported).
Categories were developed and assigned (Level 2 conclusions) based on similarity of meaning of Level 1 conclusions.
A set of synthesized conclusions were developed (Level 3 conclusions) after subjecting the categories to meta-synthesis. This represents the synthesis of Level 1 and Level 2 conclusions.
Please note: For JBI textual evidence reviews, not supported findings should not be included in the synthesis process. They may be presented separately in the extraction table, or in the Appendices.
