5.2.6.4 Assessment of quality: Evidence from policy/consensus guidelines

Policy, for our purposes, refers to a deliberate set of principles designed to guide decisions and achieve rational outcomes. In health care, a policy or consensus guideline is essentially a statement of intent that is often then implemented as a procedure or protocol. Critical appraisal of policy and consensus guidelines draws on features of discourse analysis that seek to identify the degree to which the text being reviewed has ‘authority’ in-so-far as its purpose and its focus on serving the best interests of health care recipients; and the quality of the policy or guideline.

Discourse analysis is characterized by a wide range of approaches stemming from a number of theoretical bases. The term discourse itself refers to expressing oneself using words and discourse analysis attempts to describe, interpret, analyse and critique positions reflected in text.  Critical discourse analysis more particularly studies written texts to uncover discursive sources of power, dominance, inequality, and bias. The JBI approach to critically appraising evidence from policy merely draws on the techniques of discourse analysis, rather than subscribing to, or committing to, its philosophical bases.

In terms of the quality of guidance, an international team of guideline developers and researchers, known as the AGREE Collaboration (Appraisal of Guidelines, Research and Evaluation) established the AGREE II Instrument ⁴¹ which provides a framework to assess the quality of guidelines. Evidence-based guidelines require a strict methodological approach for development, but there is also a need for consensus guidelines, drawn from expert opinion and nominal group processes (eg Delphi methods) in reaching consensus.  

It is important to take heed of Sutcliffe and Court, ³⁰ who assert the importance of acknowledging that evidence is not the only factor which influences policymaking and guideline development. Each stage of the development cycle, a number of different factors will also affect the outcome including a policymaker’s or guideline developer’s own experience, expertise and judgement; at an institutional level, institutional capacity; and the pressure to process information quickly. They argue that policymaking and guideline development is neither objective nor neutral; it is an inherently political process.

The validity of evidence from policy and consensus guidelines in this context therefore relates to what is being said, the source and its credibility and logic; a consideration of the overt and covert motives at play; the processes of policy/guideline development; and the degree to which external evidence is considered.