A Research Guide for VTPMD 6181: Public Health Practice – Assessment: Evaluating Evidence in Public Health

Abstract

Starting with the proposition that "if we want more evidence-based practice, we need more practice-based evidence," this article (a) offers questions and guides that practitioners, program planners, and policy makers can use to determine the applicability of evidence to situations and populations other than those in which the evidence was produced (generalizability), (b) suggests criteria that reviewers can use to evaluate external validity and potential for generalization, and (c) recommends procedures that practitioners and program planners can use to adapt evidence-based interventions and integrate them with evidence on the population and setting characteristics, theory, and experience into locally appropriate programs. The development and application in tandem of such questions, guides, criteria, and procedures can be a step toward increasing the relevance of research for decision making and should support the creation and reporting of more practice-based research having high external validity.

Abstract

Public health interventions tend to be complex, programmatic, and context dependent. The evidence for their effectiveness must be sufficiently comprehensive to encompass that complexity. This paper asks whether and to what extent evaluative research on public health interventions can be adequately appraised by applying well established criteria for judging the quality of evidence in clinical practice. It is adduced that these criteria are useful in evaluating some aspects of evidence. However, there are other important aspects of evidence on public health interventions that are not covered by the established criteria. The evaluation of evidence must distinguish between the fidelity of the evaluation process in detecting the success or failure of an intervention, and the success or failure of the intervention itself. Moreover, if an intervention is unsuccessful, the evidence should help to determine whether the intervention was inherently faulty (that is, failure of intervention concept or theory), or just badly delivered (failure of implementation). Furthermore, proper interpretation of the evidence depends upon the availability of descriptive information on the intervention and its context, so that the transferability of the evidence can be determined. Study design alone is an inadequate marker of evidence quality in public health intervention evaluation.

Abstract

Background

There is an increasing push for ‘evidence-based’ decision making in global health policy circles. However, at present there are no agreed upon standards or guidelines for how to evaluate evidence in global health. Recent evaluations of existing evidence frameworks that could serve such a purpose have identified details of program context and project implementation as missing components needed to inform policy. We performed a pilot study to assess the current state of reporting of context and implementation in studies of global health interventions.
Methods

We identified three existing criteria sets for implementation reporting and selected from them 10 criteria potentially relevant to the needs of policy makers in global health contexts. We applied these 10 criteria to 15 articles included in the evidence base for three global health interventions chosen to represent a diverse set of advocated global health programs or interventions: household water chlorination, prevention of mother-to-child transmission of HIV, and lay community health workers to reduce child mortality. We used a good-fair-poor/none scale for the ratings.
Results

The proportion of criteria for which reporting was poor/none ranged from 11% to 54% with an average of 30%. Eight articles had ‘good’ or ‘fair’ documentation for greater than 75% of criteria, while five articles had ‘poor or none’ documentation for 50% of criteria or more. Examples of good reporting were identified.
Conclusions

Reporting of context and implementation information in studies of global health interventions is mostly fair or poor, and highly variable. The idiosyncratic variability in reporting indicates that global health investigators need more guidance about what aspects of context and implementation to measure and how to report them. This lack of context and implementation information is a major gap in the evidence needed by global health policy makers to reach decisions.

This review evaluated the impact of packages of interventions intended to reduce neonatal mortality and morbidity in developing countries. The authors concluded that the evidence is weak and further research is needed. The review was based on a thorough literature search and these conclusions appear justified.

Abstract
The past decade has witnessed widespread interest in the development of policy and practice that is better informed by evidence. Enthusiasm has, however, been tempered by recognition of the difficulties of devising effective strategies to ensure that evidence is integrated into policy and utilized in practice. There is already a rich but diverse and widely dispersed literature that can be drawn upon to inform such strategies. This article offers a guide to this literature by focusing on six main interrelated concerns: (1) the types of knowledge relevant to understanding research utilization/evidence-based practice (RU/EBP) implementation; (2) the ways in which research knowledge is utilized; (3) models of the process of utilization; (4) the conceptual frameworks that enable us to understand the process of RU/EBP implementation; (5) the main ways of intervening to increase evidence uptake and the effectiveness of these; (6) different ways of conceptualizing what RU/EBP means in practice.