Measuring Impact

Evaluation results help to refine programming, energize investments, build collaboration, and multiply impact when shared with other communities.

Proven economic outcomes are an effective way to gain the attention of public stewards.

Evaluation as a learning tool

Evaluation is a learning tool that funders and grantees use to measure an initiative’s impact on a community. It is a way for people to learn what works and what doesn’t. NEI, learning as it evolved its overall economic development strategy, also evolved its evaluation and impact approach over the length of its investments.

How to establish a solid evaluation plan

Establishing an evaluation protocol that measures the impact of support and investments involves some key steps:

Hire an external evaluator

 

An evaluation consultant will develop an evaluation process and strategy that…

  • engages the initiative’s stakeholders to ensure that all parties are aligned in terms of outcomes;
  • immerses its evaluators with the funding team to connect the evaluation framework and specific measurable objectives with the design of the overall initiative;
  • narrows the scope to what is most critical to measure;
  • determines the data collection methods;
  • builds a platform to receive the reporting; and
  • creates a communications plan for sharing results and for stakeholder reporting.

Assess grantee readiness

Many grant-funded neighborhood organizations (business support organizations) have not done rigorous program evaluation in the past. When funders and communities introduce milestones, metrics, and financial reports for evaluation, the process can seem daunting. 

Allowing time to transition business support organization grantees to a more detailed, quantitative, and responsive approach to reporting is necessary. Multiple meetings with each grantee will ready them for the transition; for some it will be relatively painless, but for others, it will be a significant struggle.

Measure what matters

Many evaluation plans are principally focused on outcomes with a small layer of process evaluation. Evaluations should capture whether milestones were achieved; classes, events and workshops happened; expected outcomes occurred; and changes could be attributed to the program or program activities.

Performance metrics, data and analysis allow for well-informed funding decisions to drive continuous improvement and long-term impact in any given community. The core purposes of an evaluation are to make judgments about a funded program, to improve its effectiveness, and/or to inform programming decisions.

Use data for decision making

Private or public funders need to fulfill their external stakeholder requirements while being empowered to make strategic internal decisions and improvements. The commitment to track and communicate results will help others to understand the goals of any economic initiative and to follow incremental progress toward achieving those goals.

Ultimately, integrating performance measurements into daily operations will allow funders access to solid data in order to drive decision making, illustrate progress and establish a culture of continuous learning that will lead to amplified social and economic impact.

What is evaluation?

Evaluation is a systematic set of processes to analyze a program’s performance. It involves collecting and analyzing data about program activities, characteristics, and outcomes. Evaluation can fall within two broad categories: formative and summative.

Learn more
  • Formative evaluations are conducted during program development and implementation; they are useful for directing how to best achieve the goals or improve the program.
  • Summative evaluations are completed once a program is well-established, and they illustrate the extent to which a program is achieving its goals.

The information below, provided by Growth Capital Network, NEI’s chief evaluator, explains when to use formative or summative evaluation methods.

Formative Evaluation Types

  • Needs Assessment
    • Determines what populations need the program, the size of the need, and what activities best meet that need. A needs assessment can determine what populations are not served by the current programs and offer insight into what features the new programs should imbed to meet these needs.
  • Process or Implementation Evaluation
    • Reviews the implementation process of the program and establishes whether the program is operating as planned. This can be performed as a continuous or one-time assessment. The resulting data are used to improve the program.

Summative Evaluation Types

  • Outcome Evaluation
    • Reviews to what extent a program is achieving its desired outcomes (short- and medium-term transformation in participants) that are a direct result of the program.
  • Impact Evaluation
    • Determines the broad, longer-term changes that have occurred as a result of the program. These impacts are the net effects, typically on the individual, their business, or community.

Why evaluate

The case for evaluation is multi-pronged, as evaluation meets the needs of a number of entities, including funders, municipalities, business associations and chambers, policymakers and others.

Demonstrate funding investments make a difference

 

No matter the differences among funding sources, they all share the need to know what works and, especially, what works well. Grants that make a difference bring more value to their communities. They do so by:

  • positioning efforts,
  • informing investment,
  • promoting stronger relationships, and
  • helping funders reach their goals.

To know what is working, foundations must evaluate their grants to

  • assess the quality or impact of funded programs,
  • plan and implement new programs,
  • make future grant decisions, and
  • demonstrate accountability to the public trust.

Some funders use evaluation to determine if their efforts are aligned across their grant programs, initiatives, values, and mission. Effective focus and alignment create an opportunity for more strategic impact, as the synergy of efforts and dollars work together serve the overall mission.

Create understanding that leads to informed investments

Funders use evaluation to make informed investments. Similar to the business sector, past performance is one indicator of future success. Foundations may require their grantees to submit evaluation results in grant proposals or to show results at key milestones in order to receive continued funding.

Promote collaboration

Evaluation can also promote resilient, collaborative relationships. By identifying outcomes in an objective way, evaluation can help funders communicate with their constituents about shared causes and effective initiatives. When evaluation is at its best use, as a learning tool, findings can help funders, stakeholders and grantees have realistic discussions about difficult issues and what it takes to make the change they need.

Strengthen grantee outcomes

From a grantee standpoint, program evaluation is a valuable tool to strengthen the quality of their programs and improve outcomes for the individuals they serve. Program evaluation answers basic questions about a program’s efficacy, and evaluation data can be used to improve programmatic services.

NEInsight  Client data fields

These are the items NEI asked its business support organizations to collect data on the business owners they served; these were also used for initiatives where NEI directly provided awards of dollars and practical assistance to small businesses.

Learn more

Company Name
Number of Founders
Founder1 First Name
Founder1 Last Name
Founder1 Age Range
Founder1 Email
Founder1 Phone
Founder1 Resident of Detroit, Hamtramck, Highland Park
Founder1 Minority Owned
Founder1 Immigrant Owned
Founder1 Woman Owned
Founder2 First Name
Founder2 Last Name
Founder2 Age Range
Founder2 Email
Founder2 Phone
Founder2 Resident of Detroit, Hamtramck, Highland Park
Founder2 Minority Owned
Founder2 Immigrant Owned
Founder2 Woman Owned
Year Company Started**
Year Became Client**
Current Employee Count**
Estimated Number of Employees Residents of Detroit, Hamtramck, Highland Park
Current Annual Revenue ($value only)
Total capital (value only)
Primary Funding Source
Street Address
City
State
Zip
Website URL
Industry Category**
Notes

Evaluation tips

As a broader group of funders are looking to evaluate innovation programs and entrepreneur ecosystems, Growth Capital Network recommends the following tips for thinking about evaluation.

Connect metrics to program activities

 

It’s important to connect the metrics to program activity. Developing a logic model for each program helps to articulate how activity inputs translate into outcomes.

Look beyond basic trends

Beyond basic year-over-year business trend summaries, business dynamics (churn) and job quality (living wages, benefits provided, skill development) are other metrics to consider when reviewing the small business landscape 

NEInsight  Statistical indicators NEI used to measure progress

Growth Capital Network, NEI’s evaluation partner, developed these statistical indicators to measure NEI’s progress. They include program indicators, industry metrics, investment data, and real estate information.
Learn more

Program

  • # Of one-on-one company interactions
  • # Clients Served
  • # Company Clients
  • # Company Clients – Minorities
  • # Company Clients – Women
  • # Company Clients – Immigrants
  • # Company Clients – Detroit based companies
  • # Companies Clients – with secondary degree
  • # Online Users
  • # Online Users – Large Corporations

Company Growth Metrics

  • # Companies Formed
  • # Companies Formed – Minority Owned
  • # Companies Formed – Immigrant Owned
  • # Companies Formed – Detroit based
  • # Companies Formed – Woman Owned
  • # Companies Formed – Women > 45 years old
  • # Companies Formed – Formed by women with college degrees
  • Research Expenditures per Invention (IDF)
  • Portfolio – # of new patents
  • Portfolio Licensing Income

Industry growth – put the companies into buckets

  • # Portfolio – Aerospace
  • # Portfolio – Automotive/Manufacturing
  • # Portfolio – Construction
  • # Portfolio – Creative Arts
  • # Portfolio – Design/Technical Services
  • # Portfolio – Education
  • # Portfolio – Energy/Alternative Energy
  • # Portfolio – Entertainment/Hospitality
  • # Portfolio – Facilities Mgmt.
  • # Portfolio – Food

Investment metrics

  • # Companies Receiving investment
  • Total Investment Capital Received by Clients
  • Total Investment Capital to Minority Owners Total Investment Capital to Women Owners Total Investment Capital to Immigrant Owners
  • Total Capital Investment – Public Dollars
  • Total Capital Investment – Private Dollars

Space utilization

  • # Tenants
  • # Tenants – Physical Space
  • # Tenants – Mailbox
  • # Tenants – Virtual
  • Square Feet Available
  • Square Feet Utilized
  • Square Feet Available – Testing facilities
  • Square feet under management
  • Square Feet Utilized – Testing facilities
  • Square feet developed

Measure economic activity

Understanding how to measure economic inclusion can be challenging. Methodologies that capture relevant data about program impacts related to racial, ethnic or gender diversity — and related to the distress of the places where investments are made — should be used.

Have baseline economic data – from the beginning

This tip speaks for itself and bears repeating. Without a baseline, it’s hard to be sure that progress is happening. It may take discipline on the part of the funders to think about this at the very beginning, but it is well worth the time and investment.

Use baseline data to see the micro-movements

Detailed baseline data allows an initiative to see the micro-movements in the small business community and the neighborhoods that it’s impacting. It’s not enough to look at housing prices or per capita income averages, there needs to be a wide range of patterned data. The information will create a baseline view of the economic ecosystem where the startups or small businesses are germinating. Data from local, state, and federal government sources can provide many details of interest.

Measure economic topography

Using sets of indicators (leading, coincident and lagging) across multiple perspectives allows for a deep understanding of the economic topography of the ecosystem being created, augmented, or scaled. At the baseline, these indicators might just look like a list of numbers. But as time passes, the opportunity to see whether the infusion of capital into a particular section of the city is spurring activity or making an impact will be far more nuanced and interesting.

Leading indicators point toward future events

Leading indicators give a sense of the financial vibrancy of the community. These include important local data such as average weekly hours, initial jobless claims, building permits for new private housing units, manufacturers’ new orders – and national data such as the S&P 500, money supply (M2), interest rate spread, and consumer expectations.

Learn more
Coincident indicators occur in real-time and clarify the state of the economy

Relevant coincident indicators include local real earnings and unemployment, as well as Gross Domestic Product (GDP) on a national level.

Lagging indicators confirm a pattern that is already in progress

Key lagging indicators include the average duration of unemployment, outstanding commercial and industrial loans, change in the Consumer Price Index (CPI), change in labor costs, the ratio of outstanding consumer credit to personal income, and the average prime rate charged by banks.

Understand that circumstances vary by community

Because circumstances and strategies vary widely, the statistics into which funders, communities or nonprofits imbed their own data will vary specific to an initiative.

Choosing statistical measures

Some national statistics may seem too broad or local data too narrow to be useful, but it all plays its part in illustrating the opportunities or challenges within a particular neighborhood.

See an illustration of common barriers

And a community with low wages, depressed housing prices, and chronically high unemployment cannot provide the same volume of friends and family risk capital to a burgeoning business or collateral for a small business loan. See an illustration of that here.

The data and trends provide the context and the contrast for the evaluation and create an opportunity to measure primary impact from a program, as well as potential secondary and tertiary effects.

Find data sources to depict regions and cities, small business ecosystems

The broad indicators in which to place local data come from a variety of places. A comprehensive list of sources are noted in the Why evaluate: A dispatch from Growth Capital Network, NEI’s evaluation partner, but some of the most valuable sources are:

  • US Census Bureau
  • US Bureau of Economic Analysis
  • US Bureau of Labor Statistics

Sources for more narrowly focused data are Burning Glass (labor forces employment) and the University of Michigan (Consumer Sentiment Index). State and city government sites are useful for local stats as are the plethora of reports out in the ecosystem.

Resources for evaluation

Free ebooks by Leap of Reason to keep up with the latest thinking in outcomes evaluation, performance management, and evaluation:

  • Leap of Reason
  • Working Hard and Working Well

For small nonprofits: Small But Mighty (Leap of Reason’s Performance Imperative)

See more

W.K. Kellogg Foundation – Evaluation Toolkit *
The Step-by-Step Guide to Evaluation is designed to guide programs embarking on evaluation.

United Way of America – Measuring Program Outcomes: A Practical Approach *
United Way has developed a step-by-step evaluation manual for health, human service, and youth and family serving agencies.

State of Evaluation: Evaluation Capacity and Practice in the Nonprofit Sector
How nonprofits engage in evaluation, published by Innovation Network.

The State of Evaluation in Colorado’s Nonprofit Sector
Downloadable PDF from The Colorado Trust, CO Nonprofits, Community Resource Center.

How do we build the capacity of nonprofits to evaluate, learn, and improve?
Downloadable PDF on how grantmakers can support nonprofits’ efforts to learn, improve and, ultimately, expand their impact; published by GEO (Grantmakers for Effective Organizations)

DIVE DEEPER: Find a longer list of resources in the Why evaluate: A dispatch from Growth Capital Network, NEI’s evaluation partner

Related case studies and resources

Key Terms

Key Terms

A words-to-know glossary of terms around building a network of inclusive support for entrepreneurs

What’s next

Influencing policy