REBECCA SERO, Ph.D. – Evaluation Specialist, Washington State University Extension
JO ANN WARNER – Associate Director, Western Extension Risk Management Education Center
Extension Risk Management Education (ERME) ssponsored projects help farmers and ranchers manage risk and achieve long-term success by providing important educational programs that help them be successful and, hopefully, pass on their heritage to the next generation. With the prospects of shrinking federal budgets ahead of us, however, it is vital that we are able to demonstrate how and to what extent ERME projects help sustain agriculture in the U.S.
Establishing performance measures at the beginning of a project, ideally during the design phase, is the first step to demonstrating the effectiveness of your project. As is often the case, farmers and ranchers may not realize the intended results until long after the project has ended, making it difficult for you to observe those results or collect actual data. With additional advanced planning and the collection of baseline data, however, you can estimate potential results for your participants.
Begin with the End in Mind
Establishing clear end goals for your participants is key to setting up a successful monitoring and performance measurement system. Identifying and teaching new agricultural or financial management practices is just the first step to getting your participants on a path to sustainability. Knowing how many participants adopted those new practices or changed their behavior is also important.
The effectiveness of your program, however, lies in the magnitude of benefits or outcomes that your participants realized as a result of adopting new practices or changing behavior. Even though you may not be able to observe or collect actual outcome data, identifying your intended outcomes (benefits to the participants) and collecting relevant baseline data will enable you to estimate the extent to which your project mitigated a particular risk.
Our earlier article on Program Evaluation & Results Verification (Winter 2017 Newsletter) addressed how to approach the verification of producer results through the lens of program evaluation. Establishing a valid and reliable performance measurement system prior to initiating a project or program is the foundation to monitoring and evaluating your project throughout and beyond its duration.
Performance measures include a mix of output AND outcome measures. Output measures measure the things produced as we achieve our objectives. They are tangible products that can be counted as a result of our activities. Outcome measures measure the results from our outputs. They are often inferred from our activities. They describe the change in condition that we want to see when we reach our goal and they typically start with a value change (e.g., increase in X). Outcome measures answer the questions: what did the target audience gain at the end and who benefited. Beneficiaries can include the participant, their household, and society.
The ERME online application and reporting system (RVS), defines results as a progression of actions that begin with learning (knowledge and awareness gained), moves to adoption of practices (behavioral changes and skills acquired), and ends with implementation. In this article, we are proposing that you begin your design planning with the end bundle of benefits and work backwards into the type of data that you will need to assess the benefits or outcomes. We provide an example of this logic using the Production Risk category.
The ERME program aspires to help America’s farmers and ranchers improve and achieve agricultural economic viability by mitigating five types of risk. In our example, the introduction of new and more efficient grazing practices aims to reduce input costs for farmers and ranchers. The purchasing of hay can be a significant cost to agricultural producers. Keeping animals on pasture longer not only reduces those costs, but can also improve animal health outcomes. In Figure 1, we lay out the objective, one potential output measure, and several possible outcome measures. The outcome measures speak to multiple and related benefits, ultimately benefiting the producers’ bottom lines.
For readers who are comfortable using Logic Models, we illustrate (Figure 2) how this information fits into this framework. Note that the long-term outcomes link back to ERME’s overarching goal of economic viability for farmers and ranchers.
Logic Model within a Performance Measure Framework
Getting to Outcomes Using a Theory of Change Approach
Logic models have long been used to help design new programs and projects. A complementary approach to logic models is the Theory of Change. While both approaches, if used properly, facilitate program design, the Theory of Change emphasizes causal linkages with an emphasis on the end goals and the multiple paths to those goals.
A theory of change is used by an organization or a program to explain the process of how a change will occur as a result of interventions (activities). The theory of change is a causal model that requires underlying assumptions to be clearly stated. In this causal model, activities become necessary pre-conditions that must occur before a long-term outcome can be achieved. Taplin and Clark (2012) state that “A theory of change describes the types of interventions (a single program or coordinated initiative) that bring about the outcomes… Each intervention is tied to an outcome in the causal framework, revealing the often complex web of activity required to bring about change.” Described another way, a theory of change explains how and why a change is expected to occur by linking a program’s outcomes to its activities.
In a theory of change, you begin by identifying your end goal; this is the societal impact and/or public value that will come about as a result of your program. Once you have determined your long-term impact, you work backwards, moving to long-term program outcomes and then medium-term outcomes. Once your outcomes have been established, you then begin to identify what must happen in order for the outcomes to be obtained. These outputs are defined as the strategies that will be employed, the tactics to be undertaken, and the training activities that will occur.
Using Mitigating Production Risk as an example, the overall societal impact of the ERME program is to work with the USDA to ensure a secure supply of safe, nutritious, and affordable food. There are also benefits that producers receive as a result of participating in programming. You would measure these benefits through the long-term results (also known as outcome measures) obtained by producers. For production risk, there are two primary ways to this: ensuring profitability by (1) increasing yields and (2) reducing input costs. Because the theory of change is a causal model, we know that in order for these results to be obtained, there are certain outputs that have to occur. Figure 3 shows a Theory of Change for Production Risk addressing the risk management areas and topics under the production risk category (ERME Western Center RFA – 5 (pdf)).
Introduction to Risk Management (pdf)
When creating your theory of change, it is often useful to answer the following questions (Forti, 2012):
- Who are you seeking to influence or benefit (target population)?
- What benefits are you seeking to achieve (results)?
- When will you achieve them (time period)?
- How will you and others make this happen (activities, strategies, tactics, trainings, etc.)?
- Where and under what circumstances will you do your work (context)?
- Why do you believe your theory will bear out (assumptions)?
Verifying Your Program’s Outputs and Outcomes
For participants to collectively advance the Extension Risk Management Education (ERME) goals of your project, you will need to collect data that determines how they applied and benefited from the risk management principles and education outlined in your project. This will involve collecting baseline measures that correlate to the desired results. Baseline measures become the standard against which you will measure the changes that are occurring as a result of your program. For example, continuing with the activity example outlined in Figure 1, if a desired result is for producers to reduce feed costs, then data needs to be collected prior to the start of the project that identifies the cost of feed (this becomes your baseline measure for cost). By then comparing this baseline measure to the cost of feed at certain points of time during the project, you begin to validate the feed cost reductions achieved by producers as a result of the programming being implemented.
There are a number of ways to establish these baseline measures. If the measures you need to track are producer specific, then conducting a short survey (e.g. prior to the start of a workshop with attendees) might provide enough information. If your baseline measures are regionally focused, then you should explore existing resources online to see if you can obtain the data you need. For example, the USDA’s National Agricultural Statistics Service (NASS) conducts a wide assortment of surveys each year on a wide variety of topics, including “agricultural production, economics, demographics and the environment.” Additionally, the NASS also conducts the Census of Agriculture – every five years – which provides agricultural data for every county within the United States.
Regardless of what baseline measures you choose to use, the critical piece is to create an ongoing feedback system as part of your performance measurement process. You should establish a way to compare current conditions to the baseline measures as your program progresses through the grant. In other words, do not wait until the end to collect all of your measurements and data collection. Instead, evaluation should be conducted at multiple points in time. The advantage of using a theory of change is that you have established, in advance, what must be happening in order for the program to be successful. By using your theory of change to guide your evaluation, you will know – prior to the program beginning – what needs to be measured and how to measure it.
The goal of ERME is to support programs that help producers achieve economic viability. As competition for shrinking federal dollars increases, ERME seeks to maintain its competitiveness by improving our capacity to demonstrate, not merely assert, the value of our programs to the agricultural sector and the American public. Adopting a variety of tools and developing a more rigorous approach to performance measurement is one step that we can take in this effort to demonstrate that our programs deliver positive and significant benefits.
Forti, M. (2012). Six theory of change pitfalls to avoid. Stanford Social Innovation Review.
National Institute of Food and Agriculture – USDA (2017). Measuring the Public Value. Planning, Accountability, and Reporting. (Mimeo).
Taplin, D. & Clark, H. (2012). Theory of change basics: A primer on theory of change (pdf). ActKnowledge.