5.1.2 JBI methodological approach to Textual Evidence Systematic Reviews
- Carolyn Phillips
Of interest to contemporary commentators is the systematic review of policy statements and documents; patient stories/narrative; the opinions of experts and expert bodies; and the varying, competing discourses associated with science, expertise and patient experience. Some commentators 8-10 argue for the conduct of narrative reviews; that is, a review that “….deals in plausible truth. Its goal is an authoritative argument, based on informed wisdom that is convincing to an audience of fellow experts.” 10 (page 3)
It is important to differentiate here between the ‘narrative synthesis’ advocated by these writers and the systematic review, or synthesis, of textual data, which may include data derived from narrative, expert opinion and policy documents or consensus guidelines. Popay et al 8 in their detailed examination of narrative synthesis, describe a process of synthesizing diverse data fields (for example text, quantitative studies, qualitative studies) by:
“…Bringing together evidence in a way that tells a convincing story of why something needs to be done, or needs to be stopped, or why we have no idea whether a long-established policy or practice makes a positive difference is one of the ways in which the gap between research, policy and practice can start to be bridged. Telling a trustworthy story is at the heart of narrative synthesis”. 8 (page 5)
While JBI acknowledges the importance of taking an inclusive approach to what counts as evidence, the conceptualization of synthesis as it relates to narrative is quite different to the views expressed above. We contend that a systematic review approach (as opposed to the approach embodied in narrative synthesis) to searching for, appraising, extracting and synthesizing data derived from text (i.e., non-research data) should utilize a structured and pre-determined framework to establish the legitimacy of the evidence included in the review.
Historically, JBI’s methodology as it related to this body of evidence (i.e., narrative, opinion and text), was grounded in discourse analysis and the evidence was defined and treated as a relatively homogenous data source.11 The focus of appraisal was on authenticity and the ability to ascertain the possible motivating factors driving alternate views. It sought to assess the credibility of the expert voice and make decisions as to whether the arguments put forth were logical.
We contend that three related, but distinctive sources of textual evidence exist in the form of narrative, expert opinion and policy. For the purpose of synthesis, we suggest that it is essential to acknowledge the unique nature of these data sources, particularly in relation to critical appraisal because the specific strategies/questions required to effectively interrogate the legitimacy and authenticity of these three data sources is quite different.
The central questions of truth and power as posited by Foucault, 12 remain legitimate in assessing the quality of a textual evidence data set, irrespective of source. Authors’ (be they experts expressing an opinion or contributing to the development of policy or narrative) attempts to represent reality may still be prone to being selective about inclusion or exclusion of information in order to serve an agenda. Thus, elements of discourse analysis in this vein remain an important underpinning of this methodological stance. This aligns with the premise that each data source (narrative, expert opinion or policy) is responsible for the presentation of an argument (in some form or another) and thus Toulmin’s Model of Argumentation 13 offers some important insight into how such text might be appraised.
Appraising or analyzing an argument, or the process of argumentation, is of much interest to social scientists, philosophers and scholars in the humanities (but historically much less so in the health sciences). There are thus numerous processes and models presented in the literature. In critically appraising textual evidence, assessing the ‘quality’ and logic of an argument is of some importance. For our purposes, we suggest the use of Toulmin’s model.13 The model conceptualizes an argument as a process that makes a claim based on data. This model breaks arguments into six different components ensuing the argument relates to a warrant (i.e., cause or reason), backing to support the warrant, a qualifier and a reservation or rebuttal. This model describes the beginning of any argument as containing three fundamental elements: the claim, the data or grounds, and the warrant.
The claim is the conclusion you wish the audience to accept; it’s the proposition you want the audience to believe is true or justified or right. For example: Successful emergency management requires both competent on-site and remote healthcare practitioners.14
The data (or grounds) are the facts and opinions; the evidence used to support your claim. For example: Patient safety and outcomes are related to staffing levels.15
The warrant is the connection leading from the data to the claim. The warrant is the principle or the reason why the data justify (or warrant) the claim. For example: When an emergency department lacks either competent on-site or remote staff poor outcomes can result.14
In addition to these three elements, there are three other optional elements that may or may not be present depending on the type of argument advanced and the nature of the audience to be persuaded.
The backing is the support for the warrant - the supporting material that backs up the principle or reason expressed in the warrant. Backing is especially important if the warrant is not accepted or believed by the audience. For example: Staffing policies, legislative support for safe staffing levels, staffing ratios.15
The qualifier is the degree to which the claim is asserted; it’s an attempt to modify the strength or certainty of the claim. The qualifier is used only when the claim is presented with less than total certainty. For example: There should be more support for emergency management staffing to prevent some unintended poor outcomes.14
The reservation (or rebuttal) specifies those situations under which the claim might not be true. For example: Although many institutions having severe staffing challenges, supporting a staffing model that includes both competent on-site and remote healthcare practitioners can improve outcomes.14
Usually, these six parts of an argument are laid out in diagrammatic form to further illustrate the important relationships.
