Item No. 1 of 1

INVESTIGATOR: Obudzinski, J.

PERFORMING INSTITUTION:
NATIONAL SUSTAINABLE AGRICULTURE COALITION, THE
110 MARYLAND AVE NE STE 209
WASHINGTON, DISTRICT OF COLUMBIA 20002

EVALUATING BEGINNING FARMER PROGRAMS: IDENTIFYING OUTCOMES, IMPACTS AND FACTORS RELATING TO SUCCESS

NON-TECHNICAL SUMMARY: The Beginning Farmer and Rancher Development Program is the only national program seeking toexplicitly train the next generation of farmers. Since BFRDP was first created, over $100 millionhas been invested to support over 200 new farmer training projects impacting farmers in every stateacross the country. And while each project is required to track and report outcomes and impacts, todate, no national-level analysis has ever been completed of BFRDP as a whole, in order to assessthe short and long-term outcomes and impacts the program has had since its creation, and to betterunderstand the factors that lead to more successful new farmer training projects. Our project seeksto conduct the first ever comprehensive evaluation of BFRDP funded projects to date, by analyzingqualitative and quantitative data from project reports, conducting additional evaluation activities asneeded, and assessing the long-term impacts of completed projects for which data has not yet beencollected. With this project, we seek to better understand both the short and long-term outcomesand impacts of BFRDP as a whole. As interest in new farmer training continues to grow across thecountry, and more new farmer training projects launch every year, it is imperative that we betterunderstand how to design the most effective and successful new farmer training programs, as wellas demonstratethe impacts this federal investment has had on growing the next generation offarmers.

OBJECTIVES: The purpose of this project is to evaluate the beginning farmer programs funded by the BFRDP, to provide clarity on their outcomes and impacts for reporting to interested stakeholders (such as policymakers), and to provide recommendations for improvement to NIFA program management and future grantees and other beginning farmer practitioners.The primary objectives of the evaluation are:Identifying and summarizing short, medium and long-term outcomes and impacts of funded projects;Identifying factors contributing to successful outcomes; andMaking recommendations to improve future outcome reporting and evaluation methodologies.

APPROACH: Phase 1: Content analysis of existing BFRDP final project reports: Phase 1 will include:1) pre-assessment of data to identify outputs and outcomes to be summarized via analysis, and;2) full content analysis.First, data usability (quality, availability, uniformity) will be assessed from a sample of BFRDP final reports and available aggregated data from the CRIS system Typologies of education project variables will be created for coding purposes. A plan for full content analysis will be developed - including sampling, coding and analysis procedures. Second, data from all CRIS project annual reports will be coded using the content analysis plan. If feasible, statistical analysis of the data will be performed.The objective of the content analysis will be to identify projects with high rates of key outcomes and look for themes related to project success. A visual model will be created to illustrate outcomes related to identified outputs and activities.Phase 2: Follow-up data collection: Phase 2 will include two surveys:1) an online survey of project directors/managers, and;2) an online survey of BFRDP program participants.The evaluation focus, data collection strategy and instruments will be designed with input from the project's advisory team. Statistical analysis of survey data will be performed at the close of each survey. The objectives will be to determine outcomes which have been identified since the completion of the BFRDP project grant, to understand more about impacts and program specifics, and to assess the current farming status of program participants and identify factors related to program success.

PROGRESS: 2016/06 TO 2018/06
Target Audience:Although this evaluation was commissioned by USDA, we intend that it be useful to a broader audience of stakeholders who have an interest in new farmers, education, and the effectiveness of federally funded programs. The final evaluation report, Cultivating the Next Generation: An Evaluation of the Beginning Farmer and Rancher Development Program, is divided into 8 sections which each address a distinct component of this evaluation. Each section is intended to meet the diverse needs and interests of various stakeholders - including policymakers, the general public, farmers, practitioners and grantees, and USDA. Please refer to Table 3 in the final report to find specific areas of interest. The report is available online at http://sustainableagriculture.net/publications/bfrdp. Changes/Problems:Phase I of the content analysis portion of the project included a preliminary assessment of the data from the CRIS reports. This assessment revealed uneven, nonstandard, and incomplete reporting of outputs, outcomes and other topics in the sample of reports assessed. A decision was made to go ahead with the coding of the data, in spite of major concerns about the validity of aggregated output and outcome totals, because of the desire to report these totals in a final report. Some of the information desired for the final report, such as the number and success of partnerships, many of the outputs and outcomes, was either not included or unanalyzable. Given the results of the Phase I assessment, it was evident that a project-to-project comparison of outputs and outcomes to determine "success" would not be possible. Instead the concept of Implementation Analysis (IA) was used as a proxy to rank the projects' success by rating how clearly and completely the project was described, implemented and reported. The IA rating scores were then used to broadly sort the projects into two tiers. The original proposal included conducting a very short project leader survey, and a more involved farmer participant survey. However, once the project started and priorities were further discussed with NIFA and the Advisory Team, other research method options for conducting phase 2 were explored. Given limitations identified with conducting a fully representative farmer survey (given the current budget and timeline, results would only be attributable to English speaking farmers and those from intensive programs), the evaluation activities were modified. The farmer survey was dropped in favor of a more extensive project leader survey. Additionally, the inclusion of successful case vignettes were added to the project deliverables. Due to delays in year one, the publication date of the final report was delayed until October 2017 and the related outreach and dissemination occurred in Q5 and Q6 of the project, rather than in Q4. What opportunities for training and professional development has the project provided?During year two, the Project Team presented our findings to current BFRDP grantees at the 2017 Project Director's meeting, and facilitated a follow up discussion on how to improve evaluation and outcome reporting with grantees. All grantees in attendance received a copy of the report and major findings were shared during a plenary presentation. Additionally, this evaluation was circulated and used as a reference by the Project and Evaluation Team, including NIFA program staff, who will continue to share and utilize the best practices and recommendations with future project grantees. How have the results been disseminated to communities of interest?Significant project activities in year two focused on publishing and disseminating the report findings. The draft of the final evaluation was circulated to the Advisory Team for their input in Q5, which was then incorporated into a final evaluation report. The final report, Cultivating the Next Generation: An Evaluation of the Beginning Farmer and Rancher Development Program, was published in October 2017. The report is available online at http://sustainableagriculture.net/publications/bfrdp and printed hard copies of the Executive Summary were also made available and distributed by the Project Team. Project leaders developed an outreach plan, in coordination with the Advisory Team, to distribute hard and electronic copies of the evaluation to practitioners, beginning farmers, policymakers and other interested stakeholders throughout year two. Team members utilized traditional and online media sources to share the project's findings, including through national and local press, organizational newsletters, blog posts, and social media. Project team members distributed hard copies of the Executive Summary at annual conferences held nationwide that attracted both beginning farmers, practitioners, and other stakeholders. The project team presented report findings at the 2017 BFRDP Project Director's Meeting and distributed hard copies to all project grantees in attendance. Project team also shared with policymakers - including NIFA program staff, other USDA personnel and Congressional offices as requested. What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

IMPACT: 2016/06 TO 2018/06
What was accomplished under these goals? Goal 1. Identifying and summarizing short, medium and long-term outcomes and impacts of funded projects: The preliminary assessment of the existing CRIS data on funded, completed BFRDP standard grants and a record-by-record assessment of the 119 grant records were completed in year one, after verifying that the data set was complete. The content analysis of the funded BFRDP projects was also completed in year one. The results were summarized into a final evaluation report that was published in year two. The final report includes a summary of short, medium and long-term outcomes and impacts of funded projects, including the following three major findings: 1) BFRDP has been successful in meeting its legislative mandate. Our findings show that farmers are at the core of BFRDP projects, with nearly every project we evaluated including farmers either in project development or implementation. Our findings illustrate a wide variety in educational content and highlight the diversity and complexity of new farms and the farmers themselves. Of the completed projects evaluated, all but one included at least one partner or collaborator, and the majority of project leaders surveyed found that their partners made a significant contribution to the project's success. In total, over half of all projects and 53% of total funding supported projects focusing on socially disadvantaged beginning farmers and ranchers as a primary audience. Our findings demonstrate that, on the whole, projects appear to be regionally balanced, with some variation from year to year. 2) BFRDP is helping to grow the next generation of farmers. Our findings showing that over 60,000 beginning farmers have been directly impacted by BFRDP during the grant period we evaluated. Almost all projects focused on farmers in their first 5 years of farming, with a significant focus on those farmers starting out at a small scale. More than two-thirds of projects offered intensive programs, lasting months or even several years, designed to move aspiring farmers quickly into production or at least well on their way. BFRDP project leaders estimate that on average, over half of BFRDP project participants have started farming, with nearly three-quarters being more prepared to farm and more successful in their farming endeavor. 3) BFRDP is building a national infrastructure, new models, and best practices to train and support new farmers. Besides directly training the next generation of farmers, our findings show that BFRDP funding has also helped projects build capacity, fill a critical gap, create an extensive collection of tools and resources, and develop, expand and replicate successful models. Goal 2. Identifying factors contributing to successful outcomes: As part of the process of identifying organizations with successful outcomes, Implementation Analysis ratings were assessed for each of the 119 BFRDP project records in year one and used to develop vignettes of successful projects. These vignettes were finalized and included in the final evaluation report published in year two. Projects selected for vignettes highlight the diversity of program types, including at least one of the following: an intensive program, a university driven effort, an example of effective partnerships, a focus on immigrant and socially disadvantaged audiences, a veteran focus, a variety of educational methods (apprenticeships, incubator, mentoring, workshops, etc.), a general audience, and projects from different regions. Additionally, the final report includes detailed findings on best practices identified from interviews conducted on successful projects. These include best practices to ensure successful partnerships, tactics for working with different audiences (experience level, background, demographics, immigrants, veterans, etc), best practices for different educational methods (classes, workshops, field days, apprenticeships, etc), and specific strategies to help new farmers succeed. Goal 3. Making recommendations to improve future outcome reporting and evaluation methodologies: In addition to the activities and accomplishments listed in year one, results from the preliminary and content analysis, project leaders survey and subsequent interviews conducted in year one were used to develop recommendations for improvements needed in outcome reporting and evaluation. The final report that was published in year two included a detailed analysis of some of the challenges the project team came across when assessing project evaluation data. These findings are presented in the final report and include specific recommendations for USDA program leaders and project grantees in how to improve evaluation methodologies and outcome reporting. In addition to the activities and accomplishments listed above, the project leader survey was developed to provide further insight into each of the three primary research goals listed above. The survey was intended to augment the data from the content analysis in order to identify outcomes that were not reported in CRIS/REEport (goal 1), get at best practices/factors contributing to successful outcomes (goal 2), and to improve evaluation reporting (goal 3). The survey was developed, pre-tested, finalized (with input from NIFA and the Advisory Team), and implemented during year one. There were 68 responses for a 57% response rate. The survey respondents were reasonably representative of the sample as a whole. Respondents who reported having additional evaluation reports or presentations, and were willing to share them, were contacted in year one. Reports with data collected during the project, that included mid-term outcomes, were highlighted and made available for use in the content analysis. Follow-up data collected a year or more after the project ended were summarized. Quantitative data were analyzed in Qualtrics software (for basic frequencies), Excel, and SPSS. Qualitative questions on best practices were analyzed inductively for themes (Patton, 1990) by a project evaluator and research assistant, then reconciled. Excel was used to review data and code identified themes. Other evaluation team members, as well as the advisory team, reviewed results of qualitative and quantitative data analyses. The evaluation team incorporated feedback and re-coded some thematic analyses. Results were drafted and incorporated into the draft evaluation report.

PUBLICATIONS (not previously reported): 2016/06 TO 2018/06
Type: Other Status: Published Year Published: 2017 Citation: Cultivating the Next Generation: An Evaluation of the Beginning Farmer and Rancher Development Program (2009-2015). National Sustainable Agriculture Coalition. October 2017. Available: http://sustainableagriculture.net/publications/bfrdp

PROGRESS: 2017/06/15 TO 2018/06/14
Target Audience:Although this evaluation was commissioned by USDA, we intend that it be useful to a broader audience of stakeholders who have an interest in new farmers, education, and the effectiveness of federally funded programs. The final evaluation report, Cultivating the Next Generation: An Evaluation of the Beginning Farmer and Rancher Development Program, is divided into 8 sections which each address a distinct component of this evaluation. Each section is intended to meet the diverse needs and interests of various stakeholders - including policymakers, the general public, farmers, practitioners and grantees, and USDA. Please refer to Table 3 in the final report to find specific areas of interest. The report is available online at http://sustainableagriculture.net/publications/bfrdp. Changes/Problems:Due to delays in year one, the publication date of the final report was delayed until October 2017 and the related outreach and dissemination occurred in Q5 and Q6 of the project, rather than in Q4. What opportunities for training and professional development has the project provided?During year two, the Project Team presented our findings to current BFRDP grantees at the 2017 Project Director's meeting, and facilitated a follow up discussion on how to improve evaluation and outcome reporting with grantees. All grantees in attendance received a copy of the report and major findings were shared during a plenary presentation. Additionally, this evaluation was circulated and used as a reference by the Project and Evaluation Team, including NIFA program staff, who will continue to share and utilize the best practices and recommendations with future project grantees. How have the results been disseminated to communities of interest?Significant project activities in year two focused on publishing and disseminating the report findings. The draft of the final evaluation was circulated to the Advisory Team for their input in Q5, which was then incorporated into a final evaluation report. The final report, Cultivating the Next Generation: An Evaluation of the Beginning Farmer and Rancher Development Program, was published in October 2017. The report is available online at http://sustainableagriculture.net/publications/bfrdp and printed hard copies of the Executive Summary were also made available and distributed by the Project Team. Project leaders developed an outreach plan, in coordination with the Advisory Team, to distribute hard and electronic copies of the evaluation to practitioners, beginning farmers, policymakers and other interested stakeholders throughout year two. Team members utilized traditional and online media sources to share the project's findings, including through national and local press, organizational newsletters, blog posts, and social media. Project team members distributed hard copies of the Executive Summary at annual conferences held nationwide that attracted both beginning farmers, practitioners, and other stakeholders. The project team presented report findings at the 2017 BFRDP Project Director's Meeting and distributed hard copies to all project grantees in attendance. Project team also shared with policymakers - including NIFA program staff, other USDA personnel and Congressional offices as requested. What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

IMPACT: 2017/06/15 TO 2018/06/14
What was accomplished under these goals? Goal 1. Identifying and summarizing short, medium and long-term outcomes and impacts of funded projects: The preliminary assessment and content analysis of the funded BFRDP projects were completed in year one. The results were summarized into a final evaluation report that was published in year two. The final report includes a summary of short, medium and long-term outcomes and impacts of funded projects, including the following three major findings: 1) BFRDP has been successful in meeting its legislative mandate. Our findings show that farmers are at the core of BFRDP projects, with nearly every project we evaluated including farmers either in project development or implementation. Our findings illustrate a wide variety in educational content and highlight the diversity and complexity of new farms and the farmers themselves. Of the completed projects evaluated, all but one included at least one partner or collaborator, and the majority of project leaders surveyed found that their partners made a significant contribution to the project's success. In total, over half of all projects and 53% of total funding supported projects focusing on socially disadvantaged beginning farmers and ranchers as a primary audience. Our findings demonstrate that, on the whole, projects appear to be regionally balanced, with some variation from year to year. 2) BFRDP is helping to grow the next generation of farmers. Our findings showing that over 60,000 beginning farmers have been directly impacted by BFRDP during the grant period we evaluated. Almost all projects focused on farmers in their first 5 years of farming, with a significant focus on those farmers starting out at a small scale. More than two-thirds of projects offered intensive programs, lasting months or even several years, designed to move aspiring farmers quickly into production or at least well on their way. BFRDP project leaders estimate that on average, over half of BFRDP project participants have started farming, with nearly three-quarters being more prepared to farm and more successful in their farming endeavor. 3) BFRDP is building a national infrastructure, new models, and best practices to train and support new farmers. Besides directly training the next generation of farmers, our findings show that BFRDP funding has also helped projects build capacity, fill a critical gap, create an extensive collection of tools and resources, and develop, expand and replicate successful models. Goal 2. Identifying factors contributing to successful outcomes: As part of the process of identifying organizations with successful outcomes, Implementation Analysis ratings were assessed for each of the 119 BFRDP project records in year one and used to develop vignettes of successful projects. These vignettes were finalized and included in the final evaluation report published in year two. Projects selected for vignettes highlight the diversity of program types, including at least one of the following: an intensive program, a university driven effort, an example of effective partnerships, a focus on immigrant and socially disadvantaged audiences, a veteran focus, a variety of educational methods (apprenticeships, incubator, mentoring, workshops, etc.), a general audience, and projects from different regions. Additionally, the final report includes detailed findings on best practices identified from interviews conducted on successful projects. These include best practices to ensure successful partnerships, tactics for working with different audiences (experience level, background, demographics, immigrants, veterans, etc), best practices for different educational methods (classes, workshops, field days, apprenticeships, etc), and specific strategies to help new farmers succeed. Goal 3. Making recommendations to improve future outcome reporting and evaluation methodologies: In addition to the activities and accomplishments listed in year one, results from the preliminary and content analysis, project leaders survey and subsequent interviews conducted in year one were used to develop recommendations for improvements needed in outcome reporting and evaluation. The final report that was published in year two included a detailed analysis of some of the challenges the project team came across when assessing project evaluation data. These findings are presented in the final report and include specific recommendations for USDA program leaders and project grantees in how to improve evaluation methodologies and outcome reporting.

PUBLICATIONS: 2017/06/15 TO 2018/06/14
Type: Other Status: Published Year Published: 2017 Citation: Cultivating the Next Generation: An Evaluation of the Beginning Farmer and Rancher Development Program (2009-2015). National Sustainable Agriculture Coalition. October 2017. Available: http://sustainableagriculture.net/publications/bfrdp

PROGRESS: 2016/06/15 TO 2017/06/14
Target Audience: Nothing Reported Changes/Problems:Phase I of the content analysis portion of the project included a preliminary assessment of the data from the CRIS reports. This assessment revealed uneven, nonstandard, and incomplete reporting of outputs, outcomes and other topics in the sample of reports assessed. A decision was made to go ahead with the coding of the data, in spite of major concerns about the validity of aggregated output and outcome totals, because of the desire to report these totals in a final report. Some of the information desired for the final report, such as the number and success of partnerships, many of the outputs and outcomes, was either not included or unanalyzable. Given the results of the Phase I assessment, it was evident that a project-to-project comparison of outputs and outcomes to determine "success" would not be possible. Instead the concept of Implementation Analysis (IA) was used as a proxy to rank the projects' success by rating how clearly and completely the project was described, implemented and reported. The IA rating scores were then used to broadly sort the projects into two tiers which was made available to the project team. The original proposal included conducting a very short project leader survey, and a more involved farmer participant survey. However, once the project started and priorities were further discussed with NIFA and the Advisory Team, other methodsfor conducting phase 2 were explored. Given limitations identified with conducting a fully representative farmer survey (given the current budget and timeline, results would only be attributable to English speaking farmers and those from intensive programs), the evaluation activities were modified. The farmer survey was dropped in favor of a more extensive project leader survey. Additionally, the inclusion of successful case vignettes were added to the project deliverables. What opportunities for training and professional development has the project provided?None to date. How have the results been disseminated to communities of interest?Findings from the evaluation activities completed during year one will be distributed in year two, primarily through outreach at conferences, online, social media, and through other outreach mechanisms to communities of interest. The evaluation report drafted during year one identified five primary stakeholders, including: general public, farmers, practitioners and grantees, USDA/NIFA, and policymakers. What do you plan to do during the next reporting period to accomplish the goals?In year two, the evaluation team will continue drafting the evaluation report and will finalize by Q6. Once finalized, the report will be published and distributed widely to stakeholders and communities of interest.

IMPACT: 2016/06/15 TO 2017/06/14
What was accomplished under these goals? Goal 1. Identifying and summarizing short, medium and long-term outcomes and impacts of funded projects: The preliminary assessment of the existing CRIS data on funded, completed BFRDP standard grants was completed in Q1. A record-by-record assessment of the 119 grant records was completed in Q3, after verifying that the data set was complete. A preliminary content analysis of the data was completed and provided to NIFA and the Advisory Committee for review in Q4. The results were summarized into a first draft of the final evaluation report, with feedback from NIFA and the Advisory Committee, in Q4. Goal 2. Identifying factors contributing to successful outcomes: As part of the process of identifying organizations with successful outcomes, Implementation Analysis (IA) ratings were assessed for each of the 119 records to produce an IA score for each record. IA ratings were completed in Q3. The rating scores were used, along with input from NIFA and the Advisory Team, in the selection for projects for the case vignettes developed during Q4. The primary purpose of the vignettes was to highlight general examples of project success, innovation, and diversity to help stakeholders understand program impact and useful practices. A short list of successful projects was identified if they, a) had an implementation analysis score of 16 or above OR were identified as generally successful by the advisory/evaluation teams. Projects were narrowed further if they b) had reported medium-term outcomes and c) had at least one average score in the Project Director's Survey (if they took the survey). Projects selected for vignettes highlight the diversity of program types, including at least one of the following: an intensive program, a university driven effort, an example of effective partnerships, a focus on immigrant and socially disadvantaged audiences, a veteran focus, a variety of educational methods (apprenticeships, incubator, mentoring, workshops, etc.), a general audience, and projects from different regions. Only one project selected did not have an IA score of 16 or more, but demonstrated solid medium-term outcomes, and provided an excellent example of a particular type of program. Project leaders from all 6 identified projects were willing to participate in the case vignettes. The lead evaluator reviewed CRIS/REEport reports, project websites, as well as other project reports offered by the leaders and developed questions for each project based on available information. Interviews were conducted over the telephone, and questions generally covered basic information about the project, project outcomes and successes, and what led to project success. Vignettes were written by one evaluator, reviewed by the evaluation and advisory teams, revised, and incorporated into the draft evaluation report. Goal 3. Making recommendations to improve future outcome reporting and evaluation methodologies: Recommendations will be drafted during year two of the project. In addition to the activities and accomplishments listed above, the project leader survey was developed to provide further insight into each of the three primary goals listed above. The survey was intended to augment the data from the content analysis in order to identify outcomes that were not reported in CRIS/REEport (goal 1), get at best practices/factors contributing to successful outcomes (goal 2), and to improve evaluation reporting (goal 3). The survey was developed, pre-tested, finalized (with input from NIFA and the Advisory Team), and implemented during Q1 and Q2. There were 68 responses for a 57% response rate. The survey respondents were reasonably representative of the sample as a whole. Respondents who reported having additional evaluation reports or presentations, and were willing to share them, were contacted in Q3. Reports with data collected during the project, that included mid-term outcomes, were highlighted and made available for use in the content analysis. Follow-up data collected a year or more after the project ended were summarized. Quantitative data were analyzed in Qualtrics software (for basic frequencies), Excel, and SPSS. Qualitative questions on best practices were analyzed inductively for themes (Patton, 1990) by a project evaluator and data collection assistant, then reconciled. Excel was used to review data and code identified themes. Other evaluation team members, as well as the advisory team, reviewed results of qualitative and quantitative data analyses. The evaluation team incorporated feedback and re-coded some thematic analyses. Results were drafted and incorporated into the draft evaluation report.

PUBLICATIONS: 2016/06/15 TO 2017/06/14
No publications reported this period.

PROGRESS: 2016/06/15 TO 2017/06/14
Target Audience: Nothing Reported Changes/Problems:Phase I of the content analysis portion of the project included a preliminary assessment of the data from the CRIS reports. This assessment revealed uneven, nonstandard, and incomplete reporting of outputs, outcomes and other topics in the sample of reports assessed. A decision was made to go ahead with the coding of the data, in spite of major concerns about the validity of aggregated output and outcome totals, because of the desire to report these totals in a final report. Some of the information desired for the final report, such as the number and success of partnerships, many of the outputs and outcomes, was either not included or unanalyzable. Given the results of the Phase I assessment, it was evident that a project-to-project comparison of outputs and outcomes to determine "success" would not be possible. Instead the concept of Implementation Analysis (IA) was used as a proxy to rank the projects' success by rating how clearly and completely the project was described, implemented and reported. The IA rating scores were then used to broadly sort the projects into two tiers which was made available to the project team. The original proposal included conducting a very short project leader survey, and a more involved farmer participant survey. However, once the project started and priorities were further discussed with NIFA and the Advisory Team, other research method options for conducting phase 2 were explored. Given limitations identified with conducting a fully representative farmer survey (given the current budget and timeline, results would only be attributable to English speaking farmers and those from intensive programs), the evaluation activities were modified. The farmer survey was dropped in favor of a more extensive project leader survey. Additionally, the inclusion of successful case vignettes were added to the project deliverables. What opportunities for training and professional development has the project provided?None to date. How have the results been disseminated to communities of interest?Findings from the evaluation activities completed during year one will be distributed in year two, primarily through outreach at conferences, online, social media, and through other outreach mechanisms to communities of interest. The evaluation report drafted during year one identified five primary stakeholders, including: general public, farmers, practitioners and grantees, USDA/NIFA, and policymakers. What do you plan to do during the next reporting period to accomplish the goals?In year two, the evaluation team will continue drafting the evaluation report and will finalize by Q6. Once finalized, the report will be published and distributed widely to stakeholders and communities of interest.

IMPACT: 2016/06/15 TO 2017/06/14
What was accomplished under these goals? Goal 1. Identifying and summarizing short, medium and long-term outcomes and impacts of funded projects: The preliminary assessment of the existing CRIS data on funded, completed BFRDP standard grants was completed in Q1. A record-by-record assessment of the 119 grant records was completed in Q3, after verifying that the data set was complete. A preliminary content analysis of the data was completed and provided to NIFA and the Advisory Committee for review in Q4. The results were summarized into a first draft of the final evaluation report, with feedback from NIFA and the Advisory Committee, in Q4. Goal 2. Identifying factors contributing to successful outcomes: As part of the process of identifying organizations with successful outcomes, Implementation Analysis (IA) ratings were assessed for each of the 119 records to produce an IA score for each record. IA ratings were completed in Q3. The rating scores were used, along with input from NIFA and the Advisory Team, in the selection for projects for the case vignettes developed during Q4. The primary purpose of the vignettes was to highlight general examples of project success, innovation, and diversity to help stakeholders understand program impact and useful practices. A short list of successful projects was identified if they, a) had an implementation analysis score of 16 or above OR were identified as generally successful by the advisory/evaluation teams. Projects were narrowed further if they b) had reported medium-term outcomes and c) had at least one average score in the Project Director's Survey (if they took the survey). Projects selected for vignettes highlight the diversity of program types, including at least one of the following: an intensive program, a university driven effort, an example of effective partnerships, a focus on immigrant and socially disadvantaged audiences, a veteran focus, a variety of educational methods (apprenticeships, incubator, mentoring, workshops, etc.), a general audience, and projects from different regions. Only one project selected did not have an IA score of 16 or more, but demonstrated solid medium-term outcomes, and provided an excellent example of a particular type of program. Project leaders from all 6 identified projects were willing to participate in the case vignettes. The lead evaluator reviewed CRIS/REEport reports, project websites, as well as other project reports offered by the leaders and developed questions for each project based on available information. Interviews were conducted over the telephone, and questions generally covered basic information about the project, project outcomes and successes, and what led to project success. Vignettes were written by one evaluator, reviewed by the evaluation and advisory teams, revised, and incorporated into the draft evaluation report. Goal 3. Making recommendations to improve future outcome reporting and evaluation methodologies: Recommendations will be drafted during year two of the project. In addition to the activities and accomplishments listed above, the project leader survey was developed to provide further insight into each of the three primary research goals listed above. The survey was intended to augment the data from the content analysis in order to identify outcomes that were not reported in CRIS/REEport (goal 1), get at best practices/factors contributing to successful outcomes (goal 2), and to improve evaluation reporting (goal 3). The survey was developed, pre-tested, finalized (with input from NIFA and the Advisory Team), and implemented during Q1 and Q2. There were 68 responses for a 57% response rate. The survey respondents were reasonably representative of the sample as a whole. Respondents who reported having additional evaluation reports or presentations, and were willing to share them, were contacted in Q3. Reports with data collected during the project, that included mid-term outcomes, were highlighted and made available for use in the content analysis. Follow-up data collected a year or more after the project ended were summarized. Quantitative data were analyzed in Qualtrics software (for basic frequencies), Excel, and SPSS. Qualitative questions on best practices were analyzed inductively for themes (Patton, 1990) by a project evaluator and research assistant, then reconciled. Excel was used to review data and code identified themes. Other evaluation team members, as well as the advisory team, reviewed results of qualitative and quantitative data analyses. The evaluation team incorporated feedback and re-coded some thematic analyses. Results were drafted and incorporated into the draft evaluation report.

PUBLICATIONS: 2016/06/15 TO 2017/06/14
No publications reported this period.