Is Meaningful Evaluation of Public ED Efforts Possible?

More rigorous evaluations of local economic development programs and policies are feasible argues a recently released working paper by Timothy Bartik, a senior economist at The W.E. Upjohn Institute for Employment Research. Evaluating the Impacts of Local Economic Development Policies On Local Economic Outcomes: What Has Been Done and What Is Doable? stresses the importance of evaluation in local economic development. The ability to distinguish the true impacts of proposed policy in providing economic development benefits is crucial in having well-functioning state and local governments, the report claims.

Bartik contends that in the U.S., rigorous evaluation does occur, but it is the exception rather than the rule. When these evaluations are completed, Bartik states, they usually show that services to small and medium-sized firms can be effective in advancing the performance of these organizations. Economic development strategies targeting distressed areas tend to be ineffective if they only provide moderate resources, but are more successful if they offer broad resources over an extended time frame, the author suggests.

Many agencies assert that they do, in fact, perform evaluations on their programs; however, they really do not, Bartik explains. It has become commonplace for state and local economic development organizations to assemble large quantities of data on program activities such as jobs created. These job creations are then declared as a program impact, which inaccurately presumes that none of the economic activity would have occurred without the program. Also, agencies will detail local economic conditions, such as jobs created during a specific time period or reductions in the unemployment rate, and claim again such progress as program impacts. Once again, by assuming improvements in the local economy are due exclusively to local economic development policies, this approach is flawed.

Bartik stresses the importance of "outcome impact" evaluations. Understanding fully how and why the policy has its estimated impacts are critical to evaluations, he says; therefore, evaluations should include estimates of how program impacts may change with any variation in the structure or administration of the program. A model evaluation, argues Bartik, would not only report on local business activity but also the impact on the economic well-being of local residents — the definitive target of local economic development policies.

Numerous ways in which more rigorous evaluations can be performed are presented in the paper. These include random experimentation, statistical analysis of program users and comparison groups, surveys and focus groups, and linking regional econometric models with fiscal impact and local labor market models. Bartik suggests that encouragement for more rigorous evaluations must come from outside pressure and funding. This outside pressure can come from legislatures, governmental audit bureaus, and higher levels of government.

One of the many reasons that evaluations are not performed enough is the fear of negative evaluations being used as a pretext to terminate a program, Bartik concludes. As a result, economic development agencies may not pursue thorough evaluation. The article states rigorous assessment is more likely to be used in the future if the results are used to improve programs, rather than end them.

The working paper can be downloaded at: http://www.upjohninst.org/publications/wp/03-89.pdf

Resources for Evaluating Programs

To encourage more thorough and useful review of technology-based economic development programs, SSTI added several exceptional titles on evaluation to its current publication catalog including:

• The Handbook of Practical Program Evaluation
• Evaluation Models
• The Balanced Scorecard: Translating Strategy into Action
• Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Public and Nonprofit Policies and Programs
• Improving the Economy, Efficiency, and Effectiveness of Not-for-Profits: Conducting Operational Reviews

Ordering information is available in SSTI's Publication Catalog, Resources for Building Tech-based Economies, which is available online at: http://www.ssti.org/Publications/Onlinepubs/catalog.pdf