JO ANN WARNER – Associate Director, Western Extension Risk Management Education Center
REBECCA SERO, Ph.D. – Evaluation Specialist, Washington State University Extension
Program evaluation is essential to the success of the Extension Risk Management Education (ERME) program. ERME’s online application and reporting system integrate results verification – methods the project team will use to evaluate and measure producer results (outcomes) – into the project’s scope of work. In this article, we will address the mechanics of results verification through the lens of program evaluation. By doing so, we will demonstrate how it can be effectively used to both document risk management impacts and accumulate stories of success from farmers and ranchers.
The worth of public value level impacts are linked inextricably to your evaluation findings (Kalambokidis, 2013). Evaluation seeks to gain insights about your target audience – to find out what works for them and what does not (Patton, 2008). It documents improvements and changes in practice to strengthen the risk portfolios of farm and ranch operations; and to assess how well program participants are able to meet their risk management goals and objectives. Well thought out program evaluation builds the risk bearing capacity for farmer and rancher participants; as well as within your program and community of stakeholders, and strengthens accountability to the funding organization and other partners.
Another primary purpose of evaluation is to evaluate the programs assets and weaknesses such that appropriate changes can be made for improving the quality of the programs offered and to assess how well it did to accomplish the stated objectives (Patton, 2008). This may determine how you will plan for and develop future programs.
The evaluation process (and design) begins when conceiving of and planning a new program or project and involves several stages throughout the life of a project and after it has ended. The information that you seek from participants will change at each stage of program delivery; as will the type of evaluation best suited to obtaining the desired information. The types of evaluation critical to outcome/results based programs such as ERME are needs assessment, outcome evaluation and impact evaluation. Process evaluation is integral too as it measures how well the project team and collaborators are executing the program and their accountability towards participants reaching the desired outcomes.
Program team members would conduct a process evaluation in order to understand what happens during the implementation of a program (Saunders, 2015). The results from a process evaluation can then serve as a road map for how you want your project to be implemented and how you want your team and partners to function.
What you have laid out in your scope of work (Project Steps) and what you actually do can sometimes be two different things. The Progress Notes section of an ERME Progress report makes it easy to record detailed data on the processes associated with program delivery. For example, questions asked might be: Was the program delivery consistent?; Were the activities delivered as initially conceived?; Were adequate tools and resources provided for participants to help them achieve changes in practice?; Was it a good learning experience for participants?; and How did the participants view the exchange? Additional questions to ask might include whether participants had adequate access to the team and partners and whether your project team was able to reach the anticipated number of producers.
Good process evaluation can provide a clear path for what you want to do next; it allows you to document your strengths and weaknesses as an organization and/or program, to determine your level of effectiveness and how to structure improvements (Saunders, 2015).
The ERME program places a high priority on an applicant organizations ability to determine producer demand – understanding and then articulating what the participants’ motivations are for taking part in your proposed project or training. A needs assessment can be used to understand the characteristics and priorities of your target population (Witkin & Altschuld, 1994). This may be especially helpful if you are starting from scratch and have no prior knowledge of your target audience, or are one or two steps removed from them. A needs assessment can also help secure credibility within the community. It can also identify potential challenges and/or barriers to participants’ ability to achieve the desired outcomes/results. This may result in modifications to curriculum, delivery methods and/or other approaches in order to make programming more appropriate for your audience (Witkin & Altschuld, 1994).
A needs assessment allows your target audience to define the relevance (or not) of the risk management topic(s) being proposed and then to provide valid evidence back to you that the issue needs to be addressed. Stakeholders who are aligned with your target audience (commodity and/or grower organizations), and others within the sphere of influence of your target audience, may also provide value in helping to define relevant factors which support the need for training or education. Having an established relationship with your farm/ranch audience can make it easier to conduct a needs assessment or you may already have benchmarking data in the way of their knowledge levels and other factors that support the need for future programs.
Outcome Evaluation is used to determine the (risk management) changes in practice achieved by your farm/ranch audience because they participated in your program. It attempts to link the changes they made to a specific part of the program. The Results and How will you verify section of the ERME application makes it easy to link these changes.
Conducting an outcome evaluation helps the project team to determine to what extent participants are making the proposed changes and if the program delivery is actually helping the farm/ranch participants to achieve their goals. It also shows the degree to which participants are benefitting from the training or program activities. Outcome evaluation also enables the project team to identify what is working well for participants and what pitfalls may be getting in the way of their success (Hoggarth & Comfort, 2010). Finally, another strength of outcome evaluation is that it documents unintended or unexpected results. Sometimes these successes can have as much if not more value to participants than the outcomes initially proposed by the project team (Morell, 2005)
Outcome evaluation is critical to the success of an ERME project because it measures/verifies short and medium term results achieved by participants throughout and by the end of an 18-month project. Figure 1 shows a progression of proposed risk management outcomes/results and identifies at what point in the program delivery it is anticipated that participants will be able to make these changes.