navigation
--

Evaluation/Performance Measurement

Performance Measurement and Evaluation

We often hear different terms used when it comes to assessing how well we are doing in terms of our strategies, programs and services. Sometimes the term evaluation is common, other times we speak about performance measurement. So what are the differences and similarities between the two? What is appropriate for your work?

Performance measurement refers to the ongoing monitoring and reporting of program accomplishments, particularly progress towards pre-established goals. It focuses on whether a program has achieved its objectives, expressed as measurable performance standards. It is typically conducted by internal program or agency staff (www.gao.gov/special.pubs/gg98026.pdf).

Performance measures may address:

  • The type or level of program activities conducted (process),
  • The direct products and services delivered by a program (outputs), and/or
  • The results of those products and services (outcomes).

Note the similarities with program evaluation in terms of the types of things that can be assessed.

The main benefit of performance measurement is that it can serve as an early warning system to management and as a vehicle for improving accountability to the public because of its ongoing nature.

A "program" may be any activity, strategy, project, function, or policy that has an identifiable purpose or set of objectives.

Evaluation is the systematic assessment of the processes and outcomes of a program with the intent of furthering its development and improvement. Evaluations are conducted periodically or ad hoc to assess how well a program is working and are typically conducted by expert evaluators external to the program or agency for purposes of remaining objective (www.gao.gov/special.pubs/gg98026.pdf).

The main benefit of evaluation is that its in-depth examination of program performance and context allows for an overall assessment of whether the program works and identification of adjustments that may improve its results.

Comparing Performance Measurement and Evaluation

Performance Measurement Evaluation
  • Surface level
  • Describes
  • Captures the "What"
  • "Watch the dashboard"
  • Regular and ongoing
  • In-depth
  • Explains Captures the "What and Why"
  • "Open the hood"
  • As needed

 

In the context of these two definitions, it may be that you need to conduct performance measurement to ensure that you are making progress towards pre-established goals, given that you have likely chosen to implement a program or strategy that has already been extensively evaluated and proven to work (Evidence-Based Practices). So you may not need to "re-prove" evidence of effectiveness long-term. However, you may be required to show the short-term outcomes that theoretically lead to the long-term desired results. Many funding bodies continue to frame reporting on funded work as evaluation. It is recommended that you ensure your evaluation or performance measurement initiatives align with your funding body's requirements.

Performance Measurement - Tips

Performance measures are selected as part of the planning process. Performance measures are ideally attached to each outcome, enabler and strategy in your plan to track your progress against that specific piece.

When selecting measures, ask yourself and your team:

  • Will this measure help us manage?
  • Is the effort of collecting this measure worth the potential benefit?
  • Will the measure be accurate (i.e. no biases/inaccuracies in data)?
  • Will the measure be reported in a timely manner (i.e. will it be available to you when you need it)?
  • Is the measure valid? Does it measure what we intend it to measure?

What types of measures can I use?

Here are some common types of performance measures:

    Quantity:
  • "What and how much have we done?"
  • Tells you how much you have produced (e.g. number of youth serviced or number of workshops offered).
    Quality:
  • "Are our clients satisfied?"
  • Tells you how well an activity or process is performed or output produced (e.g. client satisfaction with youth program).
    Efficiency:
  • "Are we doing things right?"
  • Tells you the ratio of inputs to outputs or processes.
  • This can be in dollar cost per unit of process or output (e.g. cost per youth in program) or timeliness (e.g. number of calls taken per hour).
    Effectiveness:
  • "Are we doing the right things?"
  • Tells you how successful program and strategy outputs are at producing desired outcomes (e.g. % decrease in anger episodes in youth).
    Cost-Effectiveness:
  • Value for your money.
  • Tells you the ratio of inputs to outcomes - what level of the outcome can you achieve for the dollars that you spend?
  • Can this program be taken to scale?

Do you have a good set of measures? Do your measures help you manage?

Use this checklist to help formulate measures:

  • Do your measures have high strategic alignment (i.e. are they connected to your plan)?
  • Do you have a balanced set of measures (i.e. you are measuring more than just quantity)?
  • Do the measures result in social behaviours that will align you to overall success?
  • Do you have measures you don't use or need that could perhaps be eliminated?
  • Are you measuring what's important to your clients and stakeholders?
  • Do you tailor your measures, reporting frequencies, and presentations to the intended audience?
  • Does your performance data help you guide decisions?

Performance Measurement Resources

National Health Services Institute for Innovation and Improvement This website provides examples and walks you through several easy-to-use tools for performance measurement and quality improvement. Although it is heavily focused on acute care settings, the tools are transferable and good to learn from.

Transforming Performance Measurement - Rethinking the Way We Measure and Drive Organizational Success. By Dean R. Spitzer This book focuses on the social aspects of measurement, what can happen when measurement goes wrong, and tips for maximizing the value of organizational performance measurement - a must-read for those involved in implementing performance measurement.

Program Evaluation - Tips

Similar to program planning, using a logic model framework for evaluation can be very helpful for prompting questions and developing outcomes and indicators to address your evaluation questions.

For a guide to conducting program evaluation, check out this resource on Evaluating Programs from the former Health Communications Unit.

This Evaluation Plan Template (MS Word Doc) can help you think through all of the components required to develop an evaluation plan.

Types of Evaluation

Program evaluation can be categorized based on when the evaluation is being conducted and the type of information collected: formative, process, summative or outcome evaluation. Each form is described below:

1. Formative Evaluation

Formative evaluation focuses on programs that are under development. It is used in the planning stages of a program to ensure the program is developed based on stakeholders needs and that programs are using effective and appropriate resources and procedures.

Formative evaluation can include activities such as:

Source: (THCU, 2007)

2. Process Evaluation

Process evaluation focuses on programs that are already underway. It examines the procedures and tasks involved in providing a program. It seeks to answer the question, "What services are actually being delivered and to whom?"

    Process evaluation includes:
  • tracking quantity and description of people who are reached by the program,
  • tracking quantity and types of services provided,
  • descriptions of how services are provided,
  • descriptions of what actually occurs while providing services,
  • quality of services provided, and
  • implementation evaluation.

Source: (THCU, 2007)

3. Summative Evaluation

Summative evaluation focuses on programs that are already underway or completed. It investigates the effects of the program, both intended and unintended. It seeks to answer the questions "Did the program make a difference?"(impact evaluation) and "Did the program meet its stated goals and objectives?"(outcome evaluation).

In its most rigorous form the design of an outcome evaluation can become very complex in order to rule out any other plausible explanations for the results. Outcome evaluation can assess both short term outcomes, immediate changes in individuals or participants (such as participation rates, awareness, knowledge, or behaviour) and long term outcomes (sometimes referred to as impact evaluation) which look at the larger impacts of a program on a community. An outcome evaluation can also analyze the results in relation to the costs of the program.

Summative evaluation can include the following types of analyses:

  • changes in attitudes, knowledge or behaviour,
  • changes in morbidity or mortality rates,
  • number of people participating or served,
  • cost-benefit analysis,
  • cost-effectiveness analysis,
  • changes in policies; and/or
  • impact assessments.

Different types of evaluations can sometimes be referred to by different names. We encourage you not to get too stuck on terminology but to describe your evaluations in a way that is understandable to you and your stakeholders.

Here are a few definitions that may help to distinguish between the different types of summative evaluation.

Impact - Evaluates the impact your program had on the participants or other stakeholders of the project. Impact evaluation goes a little further than outcome. It measures outcomes but also measures what changes occurred as a result of those outcomes.

Outcome - Evaluates what occurred as a result of your program. It determines whether you achieved the programs short-term and/or long term objectives.

Cost-benefit - Evaluates the program in terms of costs. It measures both the program costs and the results (benefits) in monetary terms. This means that the results of the program or benefits must be translated into a dollar value.

Cost-effectiveness - In this type of evaluation only program costs are expressed in monetary terms. Benefits are expressed only in terms of the impacts or outcomes themselves (they are not given a dollar value). Interpretation of this type of analysis requires stakeholders to decide if the benefit received is worth the cost of the program or if there are other less expensive programs that would result in a similar or greater benefit. Source: (THCU, 2007)

Evaluation Resources

Evaluation Training and Education

The following organizations offer training and education about evaluation:

www.peelregion.ca

Home | Contact Us | Search
A-Z Topic List | Privacy & Terms of Use

Smaller Text Larger Text

A-Z List | Accessible Info | Careers | Contact Us