Assessing Incubator Performance: NBIA Releases Toolkit to Aid Impact Measurement

As with every public policy or program to promote economic development, TBED initiatives can fall victim to critics’ concerns regarding the value of these approaches if performance measurement is not an integral component of your efforts. Fair assessment of impact, though, remains a thorny issue for many TBED strategies because of the early stage of investment (e.g. support for university research, entrepreneurship education or even seed capital).

 

Even when one does measure the impact of a specific program or policy, additional potentially legitimate concerns can be raised for how well your performance compares to similar efforts in other parts of the state or country. Or, the lack of any sense of a control group of companies or entrepreneurs who did not participate in the initiative to use to benchmark the difference made by the effort can lead to unwarranted criticism from skeptics.

 

This last criticism recently was levied against Washington's state network of incubators in a report issued by the state’s Joint Legislative Audit and Review Committee (JLARC) at the request of the Washington State Legislature. JLARC investigated incubators across the state including information about their location and number, Washington’s role in supporting and financing incubators, and the performance of the state’s incubators.

 

Incubators are a fundamental element of many local and regional TBED strategies. The Washington State experience provides an interesting case study of value cross TBED approaches and a useful tool to help prevent similar scenarios from occurring elsewhere.

 

In Washington’s case, the problems started with definitions: What is an incubator? The report cited previous work by the Washington Department of Community, Trade, and Economic Development (DCTED) to infer there were 53 current or potential sites proclaiming themselves as incubators in 2005. An alternate perspective estimated 24 incubators in the state, as the report’s authors interpreted information from the Washington Association of Small Business Incubators (WASBI), whose definitions for incubators are more complex than they are for those sites that may offer only office space.

 

The lack of common definitions across assessments makes measurement of their aggregate impact difficult and comparisons across regions futile. Judging the benefits of incubators, JLARC claimed they were not able to determine if any impact did exist. Additionally, the report states that due to a “lack of credible comparative analysis” of the performance of incubators in the existing literature, the central question of what would happen to firms without the presence of an incubator to assist them is left unanswered. While the report found several examples of literature reporting best practices, it concluded that research needs to be performed to examine if businesses would not have succeeded “but for the incubator."

 

The “but for the incubator" argument illustrates a significant challenge for most public TBED policies, because comparisons with control groups can be virtually impossible. In the case of business incubation, should the control group be all small businesses? Only those in similar technology sectors? Or only those with similar product development or service orientations? Geographic considerations for the control group also must enter into the equation. What about some indication of types of services/assistance received?

 

A 2003 National Benchmarking Analysis of Technology Business Incubator Performance and Practice, prepared for the soon-to-be-defunct Technology Administration within the U.S. Department of Commerce, explored the issue of comparing 79 incubators and identified several factors that have limited value for benchmarks because of the challenges presented by the types of issues identified above.

 

Dinah Adkins, President of the National Business Incubation Association (NBIA), recognizes the challenge for incubators, but admits, when looking to other economic development policies for models, she could not find this level of scrutiny satisfactorily addressed for other small business economic development investments or programs, either.

 

The success of business incubation, however, is clear to NBIA, whose past research in the 1997 report Business Incubation Works has found that 87 percent of all incubator graduate firms are still in business. The organization compares this value to U.S. Small Business Administration data, which state after four years only 44 percent of small businesses are still functioning. Exploring the effectiveness of incubators is an ongoing task, one that requires the continuous tracking of metrics and programs designed to assist new businesses. In the hope of increasing the number of incubators that are tracking pertinent data, last week the NBIA released a toolkit to assist in this endeavor.

 

In Measuring Your Business Incubator’s Economic Impact, NBIA Director of Publications Meredith Erlewine states that without tracking outcomes, incubators may be missing their opportunity to explain to the public and potential funders the importance of their programs. Additionally, a lack of following incubator companies may contribute to doubt about the industry’s effectiveness. If the majority of incubators tracked a handful of key metrics, she contends, the NBIA could demonstrate the industry-wide impact of incubators. This approach to measuring their impact may balance those looking for “but for the incubator” examples to demonstrate efficacy.

 

The toolkit references 10 basic metrics that incubation programs should track, which should be done annually for all clients and graduates for at least five years after a business leaves the program. These 10 measures include, among others, the number of people currently employed full and part-time, current monthly salaries and wages, debt and equity capital, gross revenues, and grant funds collected. In the future, NBIA will use these measures in industry-wide surveys. Also provided are steps for data collection, which recommend explaining to incubator clients their data will be used only in aggregate and that their participation in data collection is required on a regular basis, both before and after graduation. Additional topics within the document describe advanced metrics to use, tips to convince clients to provide data, two case studies, and recommendations on how to perform impact reporting and spinning results.

 

Measuring Your Business Incubator’s Economic Impact is packaged with surveys for incubator clients and graduates and includes a spreadsheet to assist with the collection of results. The toolkit is available as a PDF free-of-charge for NBIA members or for purchase by nonmembers at http://www.nbia.org/impact/index.php. Hard copies with a CD-ROM also are available for purchase.

 

Small Business Incubators: Review of State Policy, Funding, and Incubator Performance is available from the website of the Washington Joint Legislative Audit and Review Committee at:

http://www.leg.wa.gov/reports/07-10.pdf

 

National Benchmarking Analysis of Technology Business Incubator Performance and Practice is available at: http://www.technology.gov/reports/TechPolicy/NBIA/2003Report.pdf