Have your questions answered through some of the key resources created at tamaracks evaluation masterclass. May 29, 2007 this chapter highlights the continued salience of weisss questions about theory. Debate on the appropriate methods for conducting impact. Using the theory based approach avoids black box impact evaluations. U sing theories of cha nge for development, research and evaluation video interaction guidance 10 rooted in professional. Organisations already using a theory of change approach will have a head start in assessing impact. Calls for rigorous impact evaluation have been accompanied by the quest not just to.
Adoption of modern beekeeping and its impact on honey. A range of quantitative, qualitative and theorybased approaches for defining the. Using theory of change in the development, implementation and. What is it, how does it work, and how can it be used to evaluate impact. Applying program theorydriven approach to design and. It is widely accepted that a theory based approach to impact evaluation, one that maps out the causal chain. In parallel to these trends, behavioral economics has. Her majesty the queen in right of canada, represented by the president of. It draws on concepts and approaches used in theory based program evaluation and the more limited experience of theory based research evaluation, providing a brief overview of conceptual strengths and limitations of other methods. Evaluation and application of andragogical assumptions. Theory based evaluations can be used with individual cases such as a single policy change or with multiple cases, such receiving capacity development support. How should we choose methods for impact evaluation. Although the framework does include broad guidance on the use of theory, it contains little practical guidance for implementers and there have been calls to develop a more comprehensive approach.
Some think a common conceptual and operational understanding has been elusive, while others point to a completely consistent basic concept regardless of slight. Theorybased approach the monitoring and evaluation toolkit. The choice of the approach or method to use depends upon a number of factors. Choosing the evaluation method often depends on the type of effort being evaluatedprograms and direct services are best served by some approaches, while others best serve advocacy.
A theory based approach to impact evaluation maps out the causal chain from inputs to outcomes and impact, and tests the underlying assumptions. The current study was to show the results of using the method of theory driven approach to program planning to design and evaluate a teachers professional development. In addition, toc is being used for impact evaluation at a programmatic level, providing an. It is particularly useful when the intention is to learn from an impact evaluation conducted at one site and then apply these lessons to another site. Since the 1990s, many works have refined the theory based approach dressing it up with new names such as contribution analysis, elicitation method or general elimination methodology. Combinations of program theory approach and stakeholder orientation are very rare. Impact evaluation a model guidance and practical examples. Howard white identifies six principles for the successful application of the approach mapping out the causal chain programme theory, understanding the context, anticipating heterogeneity, rigorous evaluation of impact using a credible counterfactual, rigorous factual analysis and using mixed methods. Knowles 1984 acknowledged the work of eduard lindeman as having a profound influence on his approach to adult education. It sets ie into the wider setting of evidence based policy. The trunk is built on a dual foundation of accountability and systematic social inquiry. There are several different approaches to theorybased evaluation. There are several different approaches to theory based evaluation.
At the first sight it may looks contrary to the nativism. Approaches and methods of theory based impact evaluation. Among semantic, syntax, and pragmatics, the usage based theory emphasizes the primary role of pragmatics in human communication. The evaluation is designed to test the theory to see if it holds true. Lewins threestage model of change2 focuses on the driving forces that facilitate or hinder change, and how those involved in the change agree that the change is necessary, collaborate towards the desired result and. A refined method for theorybased evaluation of the. When to use theorybased evaluation theory based evaluations may be used alongside almost any other type of evaluation. A theory of change is a building block for impact evaluations and should be used in some form in every impact evaluation. An impact evaluation approach which unpacks an initiatives theory of change, provides a framework to collect data on immediate, basic changes that lead to longer, more transformative change, and allows for the plausible assessment of the initiatives contribution to results via boundary partners. For the vanguards, dealing with high levels of complexity and uncertainty, theory based evaluation offers a robust approach to measuring impact. Given these potential benefits, participation in evaluation is often presented as a winwin situation see box 2. A prospective, theorydriven process of intervention. In a 1995 paper, weiss described the potential contribution of this approach to the evaluation of ccis. Approaches, tor and methods for impact evaluation interact.
Introduction to mixed methods in impact evaluation. Although other kinds of evaluation may cover one or more of the features in the list below, an impact evaluation would normally be expected to cover all of them stern 2015, perrin 2012, rogers 2012, actionaid 2016. An assumption is made that an action is purposeful. Theorybased evaluation in complex environments polish agency. In cflat, we have been using theory based approaches since 2000. This chapter aims to help readers identify what is distinctive about impact evaluation. Ive helped to develop a theory of change for a project or program. Theory based approaches are a logic of enquiry, which complement and can be used in combination with most of the evaluation designs and data collection techniques outlined in program evaluation methods. We will show you how theory based impact evaluation can match the rigour associated with counterfactual impact evaluation and when it might be a better approach to determining how interventions can be more effective. Impact evaluation is an assessment of how the intervention being evaluated affects. We employed a theorybased evaluation approach for each of the five outcome evaluations presented in belcher et al. The impact evaluation model can support this process by helping.
It is widely accepted that a theorybased approach, one that maps out the causal. Jul 05, 2014 the medical research councils framework for complex interventions has been criticized for not including theory driven approaches to evaluation. To develop the awareness and knowledge of what theory based evaluations can offer, we recently asked frans l. Ive helped to develop a theory of change and use it for monitoring andor evaluation. Pdf universiti teknologi malaysia utm was established in 1972 and is one of the well known research universities ru in malaysia. Participants can be difficult to track long term and individual or group factors which draw people to participate may also cause drop out or irregular participation from a project and even more so from a structured study. It can also provide a rich process and impact framework to guide implementation and evaluation, addressing barriers to implementation, and incorporating the rationale behind approaches taken and contextual influences. The logic model is a key tool to support this approach. Developing the evaluation framework through a participatory and iterative process enables the involvement of policy actors and facilitates co. There is greater understanding of counterfactual logics, the approach to causal inference that underpins experimental approaches to ie. They can be used in any sector and with any kind of work.
Program theory and the evaluation of a program s theory have gained interest in the evaluation field. It is a way of structuring and undertaking analysis in an evaluation. Methods theory based impact evaluation according to white 2009, theory based evaluation, which examines the assumptions underlying the causal chain from inputs to outcomes and impact, is a wellestablished approach that has long. Choosing an impact evaluation method 187 determining which method to use for a given program 187 how a programs rules of operation can help choose an impact evaluation method 188 a comparison of impact evaluation methods 193 finding the smallest feasible unit of intervention 197 chapter 12.
In particular, our evaluations were informed by the aspen. In contrast with black box impact evaluation approaches, which only report mean differences in outcomes between treatment and comparison groups, theory based impact evaluation involves mapping out the causal chain from inputs to outcomes and impact and testing the underlying assumptions. Overview there is a wealth of literature available. Theory of change page 5 the education sector1 identified four different research based theories to inform the evaluation. A theory based evaluation design tests the validity of the assumptions. In parallel to these trends, behavioral economics has increasingly attracted. Theory driven evaluation methods branch of theory tree developed by sociologists huey chen and his advisor, peter rossi chens book published in 1990 normative theory prescriptive, what should be causative theory descriptive, what is research based. Aug 07, 2018 process evaluations, like other theory based approaches to evaluation such as realist evaluation and theory of change approaches investigate the underpinning theory of the intervention. Map out the causal chain program theory which explains how the intervention is expected to lead to the intended outcomes. Theorybased impact evaluation international institute for. Evaluation is an active component of change management, ideally achieving a balance of meaningful practical application and methodological rigour. Applying theory based impact evaluation tbie approach.
Ive used an existing theory of change to guide monitoring and evaluation. Applying a theory of change approach to the evaluation of. Introduces a new approach to the general evaluation field called theory based stakeholder evaluation tse model. Choosing appropriate designs and methods for impact. This approach to evaluation focuses on theoretical rather than methodological issues. So, in order to ensure that we are all on the same page, box 1 summarizes how the terms are used in these guidance notes. What questions does impact evaluation seek to answer.
This means a realist evaluation will always be working with an explicit project, programme or policy theory. The underlying rationale for choosing a participatory approach to impact evaluation can be either pragmatic or ethical, or a combination of the two. Theory based evaluation and objective based evaluation have gained tremendous popularity in the last decade because of its power to augment an evaluation s capacity to assess program impact without strong experimental designs and the contemporary. This article lays out the application of a theory based approach to the evaluation of social fund projects, with a focus on the issues of subproject sustainability and institutional development. The benefits of a mixed methods approach 4 part ii. Theorybased approaches to evaluation use an explicit theory of change to draw conclusions about whether and how an intervention contributed to observed results. It is widely accepted that a theory based approach to impact evaluation, one that. Designs need to build on causal inference approaches each of which have their. Although the commitment to theory based impact evaluation is there in principle, few. The resulting coaching approach was informed by models of teacher coaching9 and figure 7 initial theory of change interview mapped as a flowchart for use as the basis of return interviews 11 theory based methodology.
It is widely accepted that a theory based approach to impact evaluation, one that maps out the causal chain from inputs to outcomes and impact. A theory of change articulates explicitly how a project or initiative is intended to achieve outcomes through. An impact evaluation approach without a control group that uses narrative causal statements elicited directly from. A refined method for theorybased evaluation of the societal impacts.
Introduction to mixed methods in impact evaluation interaction. Using the impact evaluation model will give you a holistic view of how an intervention is working and will enable you to build a persuasive case for its impact, based on both qualitative and quantitative evidence. Guidance for transport impact evaluations the tavistock institute. If one does not exist before the evaluation then it will need to be developed as part of the evaluation. Developing a theory or model of how the intervention is supposed to work 6 8. We now identify at least three reasons to begin the design and evaluation of a cci with a good theory of change. A theorybased evaluation of a program is one in which the selection of program features to evaluate is determined by an explicit conceptualization of the program in terms of a theory, a theory which attempts to explain how the program produces the desired effects. Deciding which theorybased evaluation approach is appropriate. Social, ethical and political considerations for impact evaluation 50 5. For example, a theorybased evaluation often forms an important part of an impact evaluation, a case based evaluation or a realist evaluation. Impact evaluation is the main means for empirically testing what actually happens when interventions are implemented. In a new guide on impact evaluation, stern makes this point about when a fully.
While there is little agreement either on the terminology or the concepts, there is consistency in the main messages and agreement on the value of theory based. The main purpose of ie is to improve evidence based policy. A theory based evaluation is usually based on an explicit theory of change or logic modelthat explains the theory of a development intervention. While it is a highly recommended approach by wb because of its robustness and being dubbed as the gold standard of evaluation, they themselves admit based on their evaluation experience, that with the complexity of sbm policies and intervention and despite the fact that randomization can be applied, it is not possible to attribute the. In comparison, in the case of the counterfactual impact evaluation cie complemen. One best evaluation approach quantitative data is the. Theory based evaluations are particularly relevant for policy makers as they explain why an intervention works or not in a given context allowing for generalisation. These include but not limited to formal surveys, performance indicators approach, theory based evaluation, participatory methods and costbenefit and costeffectiveness analysis. The model can also encourage team members to engage more fully in the planning, design and implementation of services, which will. Decide the structure of the workshops based on the local context and the types of stakeholders.
There is a wealth of literature available on theory based evaluation and impact evaluation tbe, but experts disagree on whether tbe is a common and clearly defined approach. Applying a theory of change approach to the evaluation. Approaches include theory based evaluation weiss, 1995, 2000, theory driven evaluation chen, 1990, and contribution analysis mayne, 2001. Needless to say, it is not only our views that are evolving. The basic idea is to use the programs rationale or theory as the basis of an evaluation to understand the programs development and impact smith, 1994, p. Tbe tools for impact evaluation of sd and governance programmes 4. Mixed methods have the same objectives, ask many of the same questions and draw on all of the impact evaluation tools and techniques described in guidance note 1 of this series gn1. The limitations of an exclusive reliance on quant or qual evaluation approaches 3 1. All develop a theory of change for the intervention and then verify the extent to which the theory matches what is observed.
Since the 1990s, many works have refined the theory based approach dressing it up. Evaluations using quantitative impact evaluation approaches. Theory based stakeholder evaluation tse, instead of fusing the theories of different stake holders into a single unified theory, keeps the program theories of diverse stakeholder groups sepa rate from each other and from the program theory embedded in the institutionalized intervention. This study was the first year of a twoyear project which applied a program theory driven approach to evaluating the impact of teachers professional development interventions on students learning by using a mix of methods, qualitative inquiry, and quasiexperimental design. Theory based evaluation tbe is versatile but is only useful insomuch as it is tailored to the intervention in question.
Building stakeholder participation into an impact evaluation needs to be meaningful and feasible, with ethical dimensions carefully considered. International initiative on impact evaluation, 3ie email. If it does, the task of the evaluator is to produce a plausible case, with evidence, that shows what has. Theory based evaluations can be used with individual cases. Nov 23, 2018 and weaknesses of various evaluation approaches and ways to design and apply them in practice. Theory based methods take many forms and are referred to in a variety of ways for example program theory, implementation theory and realistic evaluation, but usually incorporate a theory of change in some form. An impact evaluation approach which unpacks an initiatives theory of change, provides a framework to collect data on immediate, basic changes that lead to longer, more transformative change, and allows for the plausible assessment of the initiatives. Calls for rigorous impact evaluation have been accompanied by the quest not just to find out what works but why.
One of the main challenges in undertaking impact evaluations of transport. Christie o ur evaluation theory tree is presented in figure 2. Leeuw, from the maastricht university, to draft guidance outlining under which circumstances and how this approach. A theory based evaluation is usually based on an explicit theory of change or logic model that explains the. The various links in the chain are analyzed using a variety of methods, building up an argument as to whether the theory has been realized in practice. According to the canadian government 2012, theory based evaluation is an approach to evaluation i.
1438 995 1796 409 705 1249 72 1190 816 655 1819 659 1478 1043 450 995 209 924 193 108 1240 838 1043 735 387 178 182 1301 1724