Evaluating, synthesizing and grading of the evidence
Each relevant resource (guideline, systematic review, primary study) is then critically appraised with respect to methodology, significance and applicability of the results independently by two authors against standard appraisal checklists with a third arbiter for cases of disagreement. Appraisal tools, for example AGREE instrument for guidelines and critical appraisal proformas (for example those developed by the Critical Appraisal Skills Programme (CASP)) are available for this purpose.
Once the evidence has been critically appraised it is translated into evidence-based recommendations. The strength of recommendation for an intervention depends on:
- the balance between benefits, risks and burdens
- the quality if the evidence for the intervention.
Each recommendation is graded using a modification of the grading system developed by the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) Working Group.
The GRADE system incorporates an assessment of the quality of the evidence and the strength of recommendation.
Quality of recommendation
The quality of evidence for each clinical question is determined after considering study design, study quality, consistency, and applicability of the evidence to the target population.
GRADE uses the following definitions of quality of evidence:
High = Further research is very unlikely to change our confidence in the estimate of effect.
Moderate = Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate.
Low = Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate.
For recommendations regarding therapeutic interventions, randomised controlled trials (RCT) with low risk of bias provide high quality evidence, observational studies provide low quality evidence and case series or case reports also constitute evidence of low quality. The GRADE system allows for downgrading RCT evidence or upgrading evidence from observational studies based on the following specified criteria:
Factors that downgrade the level of evidence:
- limitation to the quality of the study, for example unmasked assessors of subjective outcomes or high loss to follow up
- important inconsistency
- some uncertainty about directness (‘generalisability’ to the target population)
- imprecision of the estimate
- high probability of reporting bias
Factors upgrading the level of evidence:
- strong or very strong evidence of association
- dose-response effects
- all possible confounders would decrease the observed effect
Observational studies can occasionally provide moderate or even high quality evidence, for example situations where the magnitude of a treatment benefit is large and consistent.
Strength of recommendation
The strength of recommendation is classified as strong or weak. When determining the strength of recommendations, consideration is given to the balance between desirable and undesirable effects, the quality of the evidence, and variability of or uncertainty about patient values and preferences.
Where guideline authors are confident that most patients would benefit from a particular management strategy, they will make a strong recommendation. Strong recommendations are usually underpinned by high quality evidence, but there may be situations where strong recommendations are made in the presence of moderate or low quality evidence if the benefits clearly outweigh the risks. However, in situations where there is uncertainty regarding the magnitude of a treatment effect or the risks and benefits are finely balanced, a weak recommendation should be made.
Search Clinical Management Guidelines.