Journal of the NACAA
Volume 13, Issue 1 - June, 2020
Evaluating Perceived Benefits of a Long-Running Extension Program: Lessons from a Regional Restoration-Based Conference
- Dettenmaier, M., Forestry Extension Educator, Utah State University Cooperative Extension
Dupéy, L.N., Regional Natural Resources Educator, Forestry, UW-Madison Division of Extension
Kuhns, M., Extension Forestry Specialist, Utah State University Cooperative Extension
McAvoy, D., Extension Assistant Professor, Utah State University Cooperative Extension
Demonstrating the value of Extension programming is vital, especially when justifying long-running programs and conferences. We sent an Internet survey to previous conference attendees (n = 1,277) to examine if and how attending an annual conference impacted their professional lives. Most respondents (76%) agreed their professional actions were positively influenced, and 58% agreed the natural resource(s) they work with have been positively impacted because of their attendance. Results from this study demonstrate how precise evaluative tools can demonstrate the value of a long-running Extension conference, improve the capacity and reach of an existing Extension program, and justify the existence of a reoccurring Extension program over time.
The idea for a regional restoration-based conference was first implemented in fall 2004 when Utah State University (USU) hosted a conference titled “Managing Aspen in Western Landscapes” at Southern Utah University. This conference was planned by a partnership that included USU researchers and Extension specialists, several county Extension agents, a U.S. Forest Service researcher, and others who felt the need to have a forum for sharing scientific information and experiences on management and restoration of aspen forests in the West. That first conference included one day of indoor presentations and 2 days of field trips in aspen forests. Based on the success of this conference and interest from state and federal land managers for a land restoration conference focused on challenges facing western ecosystems, we developed the Restoring the West (RTW) Conference.
Utah State University Extension Forestry and several partners held the first RTW Conference in September of 2006 and every year following (14 years total, 12 years of data is included in this paper). Themes have varied widely, from aspen ecology and management, to restoring sagebrush ecosystems, to climate change, to fire (Appendix A). The audience has included state and federal land managers and resource professionals, landowners, researchers, and students. Since 2006, we have averaged 181 attendees per year. The lowest attendance was 115 in 2008, while the highest attendance was 283 in 2015. Currently, the conference consists of two days of presentations with keynote speakers, a poster session, and an off-campus social reception. We chose the speakers each year based on the specific conference theme, their expertise, recommendations by peers, recent research, and management involvement.
Evaluating Extension programs is essential to documenting their benefits. This is important to Extension’s long-term sustainability as there is increasing pressure to produce and report on the substantive, tangible benefits of publicly funded programs (McClure et al., 2012; Lamm et al., 2013). Evaluation becomes even more important when Extension programs persist for many years, as is the case with RTW (Hatchfield et al., 2013). Conducting follow-up surveys one to five years following the occurrence of an Extension program can help educators measure program quality, quantify benefits (Larese-Casanova, 2018), and measure outcomes related to perceived changes in behavior over time (Koundinya et al., 2016). Follow-up surveys can provide beneficial feedback for Extension practitioners, however, evaluations that measure long-term benefits are rarely documented (Workman & Sheer, 2012).
The RTW planners (and authors of this paper) administer an online survey immediately following every RTW conference to measure short-term impacts and capture perceived changes in knowledge, skills, and abilities as a result of attending the conference. These results provide useful feedback that guides future conference content, themes, and logistics. While this information is useful, we were interested in understanding the longer-term impacts resulting from attending RTW, so in 2018 we surveyed RTW attendees spanning the 12-year period from 2006 to 2017. We did this with an online survey that attempted to measure broad, long-term impacts of RTW attendance for all previous attendees. Our findings demonstrate the value and importance of conducting long-term evaluations on continuing Extension programs.
Purpose and Objectives
The RTW Conference mission is to bring together resource professionals, landowners, academics, and land managers to facilitate collaboration on pertinent natural resource management and restoration issues. The 2018 follow-up survey measured perceived impacts from attending this conference. The purpose of this study was to determine if and how RTW conferences have impacted our attendees. Specifically, we aimed to:
- assess if the goals of RTW were reflected in attendee experiences,
- assess the specific areas of attendees’ professional lives that were impacted by RTW attendance,
- estimate the amount of land that attendees managed or impacted,
- determine if attendees perceived a positive change in the condition(s) of the natural resources they managed or studied due to their attendance, and
- determine the value of conducting follow-up surveys to evaluate long-term Extension program impacts and compare that to single-event surveys administered shortly after a program.
The 12 RTW Conferences that were included in this study had a total of 1,996 registrants, or 1,277 registrants with duplicates removed. In our outreach emails we indicated our desire to evaluate the conference and to determine if it was meeting the needs of attendees. We included a link to an online Qualtrics survey and asked them to take the survey (Appendix B). We sent the first email on May 21, 2018; 20 of those emails bounced and 296 surveys were started. On May 28, 2018 we sent the email to 1,172 registrants; 13 of those emails bounced. We sent a final email to the remaining 1,029 registrants on June 14, 2018; 23 of those emails bounced. We sent the survey in three batches to remind people to take the survey and to give everyone a chance to complete the survey if they had started it but had not completed it (Qualtrics has unique links for each person that takes the survey which allows users to completed partially finished surveys instead of starting over). Of the 1,221 valid surveys sent, we received 175 completed surveys, representing a 14.3% response rate.
Did Attendee Experience Reflect RTW Goals and Objectives?
We asked respondents to rate how strongly they agreed or disagreed that the objectives of RTW were met in the conference(s) they attended. While some (19%) were neutral or disagreed, most (81%) agreed or strongly agreed that the conference successfully “brought together resource professionals, landowners, academics and land managers to facilitate collaboration on pertinent natural resource management and restoration issues” (Figure 1).
Figure 1. Proportion of respondent’s indicating different levels of agreement with the statement “The mission of Restoring the West was met in the conference(s) I attended”.
We asked the question: “In what way has your attendance at past RTW Conferences influenced your professional actions? (e.g., management decisions, research, teaching, monitoring protocols, treatment prescriptions)” (Figure 2). The majority (76%) of respondents reported that attending RTW had a positive influence on these professional actions.
Figure 2. Responses to the question “In what way has your attendance at past RTW Conferences influenced your professional actions? (e.g., management decisions, research, teaching, monitoring protocols, treatment prescriptions).
To investigate this topic further we asked about specific ways RTW impacted respondents’ professional actions. Most (81%) agreed that RTW generated topical discussion and nearly two thirds (64%) agreed that RTW facilitated collaborations (Figure 3).
Figure 3. Levels of agreement with the statement: "Please rank how strongly you disagree or agree with the following statements as they relate to your professional life. My attendance at RTW: Facilitated collaborations, Generated topical discussion, Influenced my management or research decisions, Established inter/intra-agency relationships, Improved inter/intra-agency partnerships”.
Did Attendance at RTW Positively Impact Natural Resources?
We asked respondents if they agreed that their attendance at RTW positively impacted the condition of the natural resources they work with (Figure 4). Approximately 58% percent indicated they somewhat or strongly agreed with this statement, 34% were neutral, and 8% either strongly or somewhat disagreed with this statement.
Figure 4. Respondents’ levels of agreement with statement that “My attendance at Restoring the West positively impacted the condition of the natural resource(s) I work with”.
How Many Acres Were Managed by Attendees Decisions?
We asked two questions to determine the number of acres impacted by respondents’ attendance at RTW. First, we asked respondents to estimate the amount of land they work with to gauge approximately how many acres may be impacted due to the existence of RTW (Table 1). Next, we examined the data from the 101 individuals that indicated their attendance at RTW positively impacted the natural resources they worked with and combined the acres they manage. The most conservative estimate of the number of acres managed by this group of respondents is over 229 million acres.
Table 1. Amount of land that survey respondents work with
|Number of acres||Number of respondents||Proportion of respondents|
|Less than 1,000||10||6%|
|1,000 to 100,000||29||17%|
|100,000 to 1 million||36||21%|
|1 to 20 million||49||28%|
|More than 20 million||14||8%|
Our study has some limitations that future studies could address to enrich and improve the ability to make inference and draw connections between participating in an Extension program and public benefits. We had a lower than desired response rate (14%). The low response rate in our study could have been improved by reducing the number of survey questions. Inserting an assurance that a survey only has 4 questions (or will only take 4 minutes) in the subject line of an email containing a survey is one tactic that has helped us increase survey response rates in the past. The low response rate prevented us from being able to analyze survey data by conference year; instead we had to analyze responses over the life of the conference. If this survey was replicated and had a higher response rate, Extension professionals could:
- investigate if differences exist between people that attended for multiple years and one-time attendees,
- evaluate if and how perceptions of benefits from attending RTW changed over time, and
- assess how respondents’ professions and/or the amounts of land they manage impacted their answers.
We asked questions in a way that emphasized perceived behavior changes resulting from RTW (e.g., my attendance at RTW had the following impact on my areas of work). Therefore we cannot say with certainty that an actual impact or change in condition was observed, instead, we rely on the perceptions of the attendees. Our study demonstrates the value of conducting long-term surveys for reoccurring programs as such surveys provide rich data for program administrators and professionals.
Our findings have several implications for Extension programming and program evaluation.
- Long-running Extension programs can have long-term impacts; however, the only way to quantify these impacts is to administer multiyear follow-up surveys. While somewhat coarse, the estimated impacts from RTW are substantial. Extension professionals may find value in conducting similar multiyear surveys to measure outcomes and perceived benefits, especially for annual programs.
- Most respondents agreed that the RTW objectives are being met. However, there is a need to improve the experience of the 19% that were neutral or disagreed. To reach this group, we plan to provide multiple open-ended questions on all future surveys to understand how to better meet these objectives. Understanding how to improve experiences for people who engage with Extension is important information. Positive experiences can lead to continued engagement from the public. If improving the experiences of those engaging with Extension is a goal, we suggest incorporating similar questions to program evaluations.
- We conducted this survey 12 years after the first RTW. Combining brief annual surveys with follow-up surveys can further clarify if an Extension program is meeting its established objectives. To encourage long-term attendance and increase survey response rates, it may be useful to conduct follow-up surveys that include in-depth questions at shorter intervals (i.e., every 3 years instead of every 10). This approach would allow us to combine the annual feedback with broader three-year data when planning future programs. Also, conducting long-term surveys more frequently increases the likelihood that attendees will recall specifics from the conferences in question and may increase survey response rates. We suggest adopting this schedule for evaluating annual programs or conferences. Long-term surveys can provide important information to Extension practitioners and can justify the existence of specific programs.
Our study demonstrates the professional benefits realized by past RTW attendees. RTW brings together professionals from across the region to discuss and collaborate on pertinent natural resource issues. Regional conferences allow for conversation and collaboration on landscape-scale issues – the scale at which these ecosystems and issues operate.
Hatchfield, G. A., Bau, D. B., Holcomb, C. R., & Craig, J. W. (2013). Multiple year Extension program outcomes & impacts through evaluation. Journal of Extension, 51(1), Article 1FEA2. Available at: https://www.joe.org/joe/2013february/a2.php
Koundinya, V., Klink, J., Deming, P., Meyers, A., & Erb, K. (2016). How do mode and timing of follow-up surveys affect evaluation success? Journal of Extension, 54(1), Article 1RIB1. Available at: https://www.joe.org/joe/2016february/rb1.php
Lamm, A.J., Israel, G. D., & Diehl, D. (2013). A national perspective on the current evaluation activities in Extension. Journal of Extension, 51(1), Article 1FEA1. Available at: https://www.joe.org/joe/2013february/a1.php
Larese-Casanova, M. (2018). The importance of evaluating long-term impacts: Utah Master Naturalist Program as a case study. Journal of Extension, 56(6), Article 6RIB3. Available at: https://joe.org/joe/2018october/rb3.php
McClure, M. M., Furhman, N. E., & Morgan, A. C. (2012). Program evaluation competencies of Extension professionals: Implications for continuing professional development. Journal of Agricultural, Extension 53(4), 85-97.
Workman, J. D., & Scheer, S. D. (2012). Evidence of impact: Examination of evaluation studies published in the Journal of Extension, 50(2), Article 2FEA1. Available at: https://joe.org/joe/2012april/a1.php
Restoring the West Themes and Attendance by Year
|2017||Forest Restoration: What’s Working, What’s Not?||160|
|2016||Climate, Disturbance, and Restoration in the Intermountain West||156|
|2015||Restoration and Fire in the Interior West||283|
|2014||Down by the River: Managing Resilient Riparian Corridors||234|
|2013||Change Agents and Managing for Forest Resilience||154|
|2012||Balancing Energy Development and Biodiversity||151|
|2011||Sustaining Forests, Woodlands, and Communities through Biomass Use||152|
|2010||Managing Plant and Animal Conflicts||168|
|2009||Peaks to Valleys: Innovative Land Management for the Great Basin||205|
|2008||Frontiers in Aspen Restoration||115|
|2007||Sagebrush Steppe Restoration||218|
Survey Questions, Response Categories, and Answer Choices in Survey Instrument Administered in May 2018, to All Past Restoring the West Conference Attendees
|Survey question||Response categories||Answer options|
|To the best of your ability, please indicate which years you have attended the Restoring the West (RTW) Conferences.||2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017||
|In what way has your attendance at past RTW Conferences influenced your professional actions? (e.g., management decisions, research, teaching, monitoring protocols, treatment prescriptions)||
|Please rank how strongly you disagree or agree with the following statements as they relate to your professional life, interactions with other attendees, interactions with other people, or interactions with groups back home. My attendance at RTW:||
Please indicate how strongly you disagree or agree with the following statement:My attendance at RTW positively impacted the condition of the natural resource(s) I work with.
My attendance at RTW had the following impact on my areas of work or interest; select “Don’t work with” if the area does not apply:
|Approximately how much land do you manage, work with, or impact (e.g., manage, conduct research on, etc.)? If this number has changed over time, select the amount you currently manage.||
|The mission of Restoring the West is to bring together resource professionals, landowners, academics, and land managers to facilitate collaboration in natural resource management and restoration. Please rate how strongly you agree or disagree that the above objective was met in the RTW conference(s) you attended?||