Critical appraisal
Doctors should be able to critically assess guidelines and research results before clinical practice is changed. The purpose of a critical appraisal is to make sure whether one can trust, understand and apply guidelines or studies in practice (15). The online training course www.kunnskapsbasertpraksis.no provides a more detailed introduction to critical appraisal of professional guidelines, systematic reviews and primary studies with varying types of design. The course includes checklists as tools to ask the right questions about matters such as the reliability of the studies.
The GRADE system (Grading of Recommendations Assessment, Development and Evaluation) (e-Box 1) is an alternative to traditional critical appraisal with the aid of checklists (16). This is a method for the systematic and transparent appraisal of the quality of research-based knowledge and for the development of professional guidelines. The GRADE system has been developed by organisations that participate in the development of guidelines and systematic reviews, including the Cochrane Collaboration. The Norwegian Directorate of Health recommends the use of GRADE for development of professional guidelines (17).
BOX 1
GRADE (Grading of Recommendations Assessment, Development and Evaluation)
System for quality assessment of research evidence and development of professional guidelines (www.gradeworkinggroup.org/society/)
GRADE defines quality of evidence as confidence in the results (effect estimates) generated by the studies (16). The five following factors¹ may reduce the quality of the documentation from high to low:
Risk of bias: Was the randomisation concealed, who was blinded, was there major attrition?
Consistency: Are the results consistent across individual studies in the meta-analysis?
Applicability: Can we transfer the results to our clinical practice?
Precision: How precise are the effect estimates as expressed by the confidence intervals?
Publication bias: May key studies have remained unpublished?
In the development of recommendations in clinical practice guidelines, GRADE emphasises a systematic process based on an integrated assessment of four factors when moving from evidence to either strong or weak recommendations:
The balance between desirable and undesirable consequences of the intervention studied, expressed in absolute effect estimates.
Quality of the evidence: How much confidence do we place in the effect estimates?
The patient’s values and preferences: What would patients have chosen if they had been well informed about desirable and undesirable consequences of the treatment alternatives?
Costs: Does this treatment represent a sensible use of healthcare resources?
The strength of the recommendation reflects the following:
A strong recommendation indicates that the desirable consequences clearly outweigh the undesirable consequences, and is based on the assumption that virtually all well-informed patients most likely would have opted for this treatment.
A weak recommendation indicates a finer balance between advantages and disadvantages, and is based on the assumption that most well-informed patients would have opted for this treatment.
¹ Applies to studies with a randomised, controlled design. Observation studies start at low quality, but may end up with high quality on the basis of three factors that increase our confidence in the effect estimates.
Evidence-based textbooks and guidelines are intended to provide balanced recommendations by integrating research evidence and clinical expertise, patient preferences and other contextual factors, for example availability of resources (17). New standards and criteria for guidelines require, for example, a systematic review of the evidence and a balanced assessment of advantages and disadvantages of various treatment alternatives (18, 19). The development of guidelines is a demanding task. This can be illustrated by a study of a random sample of international guidelines, less than half of which complied with established criteria for trustworthiness (9). The situation is no better with regard to Norwegian guidelines and local protocols.
Well-executed systematic reviews that summarise existing research evidence through meta-analyses provide the best knowledge base for the recommendations found in guidelines and should be used as a basis for clinical decisions in the absence of trustworthy guidelines. Criteria for critical appraisal of guidelines include clearly formulated questions, explicit inclusion criteria, systematic literature searches, critical appraisal of individual studies and summaries of results with the aid of adequate methods, such as meta-analyses. Effect estimates from meta-analyses are shown as a square diamond and provide the relative effect estimate for treatment (e-Figure 4).
Individual studies are placed at the bottom of the knowledge pyramid, since they are not collated with other studies and since they also may include sources of error or other factors that limit their applicability in clinical practice (20). Nevertheless, individual studies may occasionally – after a systematic review of the documentation and a critical appraisal – constitute the best knowledge base for clinical practice (21).
The strong recommendation made by UpToDate for providing enteral nutrition through a nasojejunal tube to our patient Eva who suffers from pancreatitis is based on a well-executed, systematic Cochrane review and meta-analysis. The status of the Cochrane review as the best current research evidence for the question on enteral nutrition in the case of pancreatitis is confirmed by its position high up in the knowledge pyramid, at the level below evidence-based textbooks and guidelines (e-Figure 3).
e-Figure 4 shows a forest plot from the Cochrane meta-analysis for the outcome of mortality. Enteral nutrition in the case of pancreatitis results in a relative risk reduction of 50 % and an absolute risk reduction of 8 % (number needed to treat, NNT = 12) when compared to total parenteral nutrition. The surgeon interprets this as an impressive effect of enteral nutrition, in terms of relative as well as absolute effect.
The absolute effects of a treatment are often less impressive than the relative ones. A prime example is provided by mammography screening, whose introduction reduced the relative risk of death from breast cancer by 10 %. The corresponding relative risk reduction, however, amounts to less than 0.5 % (22). As important as being able to interpret relative and absolute effects is the ability to balance advantages and disadvantages by identifying effects on other outcomes that are important for the patient.
The surgeon decides to read the Cochrane review, and quickly retrieves the results of the meta-analysis summarised in a «GRADE Summary of Findings table». e-Figure 5 shows an excerpt from this table, in which the authors use the GRADE system to summarise relative and absolute effects of treatment and the quality of the evidence across the outcomes that are important for the patient.