Evaluating the Impact of Program Evaluation

Abstract

Nonprofit organizations are often asked to show their effectiveness by achieving participant outcomes. Further, they are often encouraged to use the data-driven decision-making (DDDM) process evaluation strategy to improve outcomes. However, not a great deal of research demonstrates the efficacy of DDDM as a strategy for outcome improvement. This research aims to examine the strategy, not just the results level, but also the ability of an organization to design and implement a DDDM strategy. To conduct this research, a case study was done of an out-of-school (OST) program required to implement a DDDM strategy focused on improving student academic outcomes. The case study uses a mixed-methods approach with a qualitative and quantitative analysis. The qualitative portion includes staff interviews and a document review. This part of the research is focused on the design and implementation of the DDDM strategy. For the quantitative analysis, the program’s student outcomes in Math and ELA will be compared to those of another OST program that did not implement a DDDM strategy. Although the program staff struggled with the strategy's design and implementation, the academic outcomes for the students in the program using the DDDM outcomes were better. These findings demonstrate that DDDM may be a worthwhile strategy for nonprofit organizations seeking ways to improve their participant outcomes. If this is true, there should be a consideration for more resources provided to nonprofits to support the implementation of the strategy.