How Do We Know We Are Effective?

by Dick Senese and Cornelia Butler Flora

In times of scarce resources, accountability in the use of resources is increasingly important. How do we know that the investments made in our programs by our staff, our funders and our volunteers are worth that expenditure of human, social and financial resources? Do our funders, staff and volunteers share the goals of our programs? Programs that do not have clear goals or goals that do not resonate with the folks who provide the resources will not continue to be funded. Effectiveness means that the program actually achieves those goals.

We often implement programs based on relatively little evidence of effectiveness. Because work with families and communities seems so important, we spend more time on problem-solving techniques than on whether or not the issue was resolved and on the impacts of our intervention on families and communities. For example, when the NCRCRD did its inventory of leadership programs (http://www.ag.iastate.edu/centers/rdev/newsletter/Vol25No1-2001/flora.html), we found very little evaluation of the programs beyond the initial training. But practitioners keep implementing these programs. And, for many involved with rural development, there is the feeling that the work they do has such face value, little evidence of effectiveness is needed.

There are two major reasons that we have such low levels of evidence of program effectiveness. First, we have scarce resources. We lack the time to follow up, as we are moving ahead with our next program efforts. We lack the skills, in that we are interested in delivery, not research. We lack the money to hire people with the skills and the time to look at impacts. Second, we can never prove causality—only that things are NOT related (the famous null hypothesis). Often we do not have data on the situations we are trying to improve. Even when we can show the initial condition of the family or community, the date we started our program, and a present condition (time order) and that the condition improved between the initial condition and our measure of the present condition (co-variation), we still cannot claim that we caused the change (elimination of rival causal factors).

Like programs offered by other organizations, Extension programs can be thought of as having a variety of program elements. These elements include such things as the (a) program theory (logic model), (b) specific activities in which participants engage, (c) the tool used to evaluate quality of delivery, and (d) approaches used to evaluate the outcome reached and the impact achieved. There is a decision point for determining each of these elements. By being clear on the quality of evidence for each element of the program, the program’s scholarship is both transparent and traceable thus assuring quality programs that are based on scholarship and research—a necessary hallmark of Extension programming. Further, focus is placed on implementing accurately, reaching planned outcomes and achieving sought-after impact.

The Community Vitality program area of the University of Minnesota Extension Service developed a matrix of levels of knowledge of program effectiveness. This matrix is based on work by both the Center for the Application of Prevention Technologies and the Minnesota Department of Children, Families and Learning (see Table 1.) By understanding where each of our programs is on the matrix, we can begin to increase our knowledge of program effectiveness, which in turn increases the efficient use of resources and helps us generate outside resources.

Levels of Evidence for Effectiveness of Program Elements

Level One — The people in the organization believe that the program and/or specific program elements (e.g., program theory, activities and/or evaluation methods) are valuable, but the actual evidence of its effectiveness is anecdotal. For example, the program and/or specific program elements may have received an award or honor from a professional organization (possibly for reasons unrelated to effectiveness), but does not have concrete data to establish credibility of the program and/or particular program element. Confidence in the program and/or specific program elements that rely on this level of evidence may be described as “not reliable.”

Level Two — Evidence for the program and/or specific program elements (e.g., program theory, activities and/or evaluation methods) have appeared in a non-refereed professional publication or journal. For example, a program and/or specific program elements may be described in a publication that does not use expert peer review process for its editorial decisions. The program and/or specific program elements may have conducted process or outcome evaluation that has been published in a professional newsletter, but not been subject to appropriate scholarly review. Confidence in the programs and/or specific program elements that rely on this level of evidence may be described as “weak” or at best as having “some confidence” that the program will actually produce intended outcomes and impacts.

Level Three — The program and/or specific program elements (e.g., program theory, activities and/or evaluation methods) have been examined in an expert-peer consensus process. Programs or elements of programs that have been described in refereed journals, for example, would be designated at this level. At this level of evidence, the program and/or specific program elements have been shown to produce some credible evidence of claims related to outcome and impact. Confidence in programs and/or specific program elements that rely on this level of evidence may be described as having “some confidence” or “fairly good confidence” that the program will actually produce intended outcomes and impacts.

Level Four — Where programs and/or specific program elements (e.g., program theory, activities and/or evaluation methods) have undergone an expert-peer consensus process in the form of a qualitative or quantitative meta-analysis, additional evidence of effectiveness is achieved. In other words, multiple studies of the same program and/or specific program elements are reviewed on several factors, including methodological rigor, implementation characteristics and findings. When a program and/or specific program elements receive support across a broad array of program interventions and evaluation strategies, higher confidence is gained that the program is solidly defensible. Also, when strategies are consistently linked to positive outcomes, confidence that the positive outcomes relate to the program activities is gained. Confidence in programs and/or specific program elements that rely on this level of evidence may be described as having “fairly good confidence” or “strong confidence” that the program will actually produce the intended outcomes and impacts

Level Five — Replications of the program and/or specific program elements (e.g., program theory, activities and/or evaluation methods) have appeared in several refereed professional journals or other similarly refereed sources. These replications may have occurred within similar or differing settings. This level of evidence is the best evidence of the effectiveness of the program’s logic model in that it can be replicated across venues and populations, demonstrating credible useful results that have been extended to other contexts. Confidence in programs and/or specific program elements that rely on this level of evidence, may be described as having “strong confidence” or “high confidence” that the program will actually produce the intended outcomes and impacts. This level assures that the program and/or specific program elements are fully defensible.

The different types of evaluation (see Table 2) are important in helping us resolve the conundrum of causality. The process, formative and summative evaluations help link what we do to the impacts. But without the impact evaluations, we cannot show that our programs are critical investments in times of austerity at the federal, state and local levels.

The approach further assumes that the program has clearly delineated inputs, outputs, outcomes and impacts that are defined in an operational manner. Given such operational definitions, the program can choose on which of this is most important to focus evaluation efforts. The focus of evaluation will gradually shift from output only to increasingly include outcome and eventually impact goals. With the ability to understand program impact and inputs comes the ability to meaningfully analyze programs using a cost-benefit framework. In short, by understanding the interdependent roles of program effectiveness and evaluation coupled with an appreciation of program development, Extension and other community change agents can accurately and confidently claim credit for the outcomes and impacts of its programming and engage in a dynamic quality improvement process for its programming. As more and more funders move to performance- based budgeting, such evidence of effectiveness will be even more important for program improvement and continuation.

References

Central Center for the Application of Prevention Technologies (Central CAPT). 2000. “Levels of Evidence.” A Road Map for Science-based Prevention Participant Materials, pp. 1.7.

Minnesota Department of Children, Families & Learning. September 1999. State incentive grant funding: Request for proposals and resource guide. St. Paul, MN.

Next Article

Return to Inside this Issue (Vol. 26, No. 2, 2003)

Return to Rural Development News Index

North Central Regional Center for Rural Development
Iowa State University
107 Curtiss Hall
Ames, IA 50011-1050
(515) 294-8321, (515) 294-3180 fax

For more information about the Center, contact pkcooper@iastate.edu.
For questions, comments or concerns about the NCRCRD website, contact khetland@iastate.edu.

Last updated May 12, 2006 .