The 5-Step Approach to Evaluation

Designing and Evaluating Behaviour Change Interventions: Summary

Guidance for service providers, funders, planning partnerships and commissioners (Summary version)

Authored by
Catherine Bisset and Leah Gilman

The 5-Step Approach to Evaluation

Designing and Evaluating Behaviour Change Interventions: Summary

Guidance for service providers, funders, planning partnerships and commissioners (Summary version)

Authored by
Catherine Bisset and Leah Gilman

Table of Contents

Foreword

Introduction

Background: The Tricky Business of Assessing Impact in the Real World

The 5-Step Approach - summary

Step 1: Identify your aims

Step 2: Review the evidence

Step 3: Draw a logic model

Step 4: Monitor your Logic Model

Step 5: Evaluate the Logic Model

Helpful Resources

References

Foreword

"I am delighted to introduce this easy-to-use 5-step evaluation guidance which is grounded in tried and tested methods. Not only should it help evaluate services of any size, but also to design more effective services from the outset.

This practical guide provides links to the key evidence on 'what works' and is packed with examples making it a valuable resource for anyone who wants to assess their contribution to outcomes including funders, planning partnerships, service providers and service staff".

Nicola Edge
Head of Justice Analytical Services
Scottish Government

Introduction

This evaluation pack is a summary of the full version, aimed at both service providers and funders who aim to promote behaviour change.

For funders and planning partnerships, it aims to:

- Offer a strategic, evidence-based and outcomes-focused planning tool

- Offer guidance on how to assess evaluations from service providers and therefore direct funding to greatest effect.

- Demonstrate the role you can play in promoting and enabling high quality evaluations from those you fund.

For service providers, it aims to:

- Provide guidance on planning an evidence-based service with a "built in" evaluation process

- Provide guidance and resources for you to effectively assess, understand and demonstrate how well your service is working in relation to your aims.

- Offer an alternative to randomised control trials, using a "logic model" approach to evaluation, which any service provider can use to evaluate any intervention, regardless of size.

- Encourage continual review and improvement of services.

Background: The Tricky Business of Assessing Impact in the Real World

A Scottish Approach to Evaluation

Co-production
Our approach to evaluation enables funders and service providers to work together in pursuit of their shared aims - to improve outcomes for service users and communities. The 5-step approach also engages with service users' views as a resource for evaluation rather than seeing users solely as an object to be measured.

Asset-based
The 5-step approach focuses on ways in which evaluation is possible for services of any size, rather than expecting all services to use an experimental evaluation method which may not be appropriate or possible for smaller, community-based organisations. The 5-step approach allows even the smallest service to demonstrate the contribution they are making to change.

An Improvement Culture
Evaluation enables improvement and even the most successful service can always be developed further. Furthermore, with the 5-step approach, evaluation is an on-going process, not something to be saved for last. This means that services can be continually improved in order to best meet the needs of their users.

How do you know if you are making a real difference to users (making an impact)?

It's not easy to find out if you're making a real difference to people, especially in the chaotic real world. There are 100s of variables which can effect people's attitudes, motivations and behaviour. So how can you tell if your project is making any difference?

Researchers and scientists generally agree that BEST way to determine if your project or service has made a difference is to use a randomised control trial (RCT), sometimes referred to as an "impact evaluation" but these are not easy to do in practice, especially in a complex social setting.

An alternative to RCT's

A "middle ground" approach
Rather than carrying out a small RCT which might be impractical and would only deliver meaningless results, we recommend that small-scale project organisers carry out a 5-step approach to evaluation. This is summarised in the following slides and detailed in the remainder of this pack.

This approach to evaluation is practical for projects of any size but does rely on providers having a clear sense of what they're hoping to achieve and how they're going to get there - a theory of change. For this reason, using the 5-step approach, must begin at the planning stage.

What is evaluation really for?

Although doing evaluation requires the use of techniques and tools, bear in mind that its overall purpose is to help you (re) design services, ask questions, gather evidence, interpret the evidence, communicate important information about your service and take informed decisions. In this sense, the ability to ask relevant questions and clearly communicate the answers at the right time to the right people are key skills in making evaluation useful.

The 5-Step Approach - summary

The 5-Step approach to evaluation

Identify the problem

If your ultimate aim is to change people's behaviour, you need to be clear what it is you are trying to change and why there is currently a need for this to happen.

Review the evidence

What you intend to do should be grounded in the evidence of 'what works' and why. Service providers should review the available evidence in order to plan activities which can be expected to achieve the intended behaviour change. The evidence should guide what you do and help you to understand the process through which it should work.

Draw a logic model

A logic model is a diagram which shows, step-by-step, why the activities you plan should achieve your aims. The logic model forms the basis for evaluating the whole project - you are going to test whether these steps happened as you predicted.

Identify Indicators and monitor your model

Use the logic model to identify indicators (i.e. measurements or observations) that things actually happen as you predicted. You will need to collect data about your project FROM THE START on inputs, activities, users, short, medium and long-term outcomes.

Evaluate logic model

Analyse the data you've collected on your various indictors to evaluate how well your project worked for your various users. Report on whether your data suggests the logic model worked as planned. Be honest about any areas which were less effective. Use this to improve your service.

Step 1: Identify your aims

Identify the Problem

Before it is possible to design an effective service, it is essential that you are clear what behaviour it is that you are aiming to change and why this should be a priority in the context you're intending to work.

An example:

WHAT is the problem? More than 1 in 5 people in Scotland continue to smoke. Long-term declines in rates of smoking have stalled in recent years and smoking is more common in areas of socio-economic deprivation, such as X.

WHY is this a problem? Smoking is a known cause of cancer and heart disease. Rates of smoking are therefore likely to be one cause of health inequalities.

What is your ultimate AIM? Decrease the numbers of people smoking and frequency of smoking in area X

Step 2: Review the evidence

For best results, use a range of evidence

To draw the most robust conclusions about 'what works,' and why, you should take account of evidence produced through a range of methods. For example, quantitative studies (including the results of RCTs) might help you to establish what usually works and for whom. Qualitative work (e.g. interviews with users who 'succeed' and 'fail' and/or with practitioners) might help you to understand the processes through which interventions work or don't work and consider why barriers may exist to achieving your aims.

TIP! If you are short on time and resources, systematic and/or literature reviews are an excellent source of evidence. They often analyse both quantitative and qualitative studies on a particular topic and should do the work of summarising all this evidence for you.

Finding Evidence

When time and resources, are limited, evidence reviews (also called systematic reviews or literature reviews) are a realistic solution - enabling an overview of the evidence in a relatively short time.

Online databases and archives are the most convenient means through which to locate evidence reviews. The following slides provide links to topic-specific databases and individual evidence reviews in health, education, environment and sport behaviour change aims. However, the following databases can be of general help in locating relevant evidence:

Search academic databases:

http://www.mendeley.com/dashboard/

http://scholar.google.co.uk/

Search government archives:

http://www.gov.scot/Publications/Recent

https://www.gov.uk/government/publications

TIP! Try searching for "evidence/literature/systematic review" + your behaviour change aim (i.e. "smoking cessation" or "increase recycling").

Area

A or P*

Topic

Link

Health and Social Care

A

Scottish Government Research

http://www.gov.scot/Topics/Research/by-topic/health-community-care

Cochrane Collaboration

http://summaries.cochrane.org/

NICE (guidance and evidence helpful)

http://www.nice.org.uk/

Health Scotland

http://www.healthscotland.com/resources/publications/search-result.aspx?page=1

Institute for Research and Innovation in Social Sciences (IRISS)

http://www.iriss.org.uk/resources

P

Review of 6 health interventions

http://www.storre.stir.ac.uk/handle/1893/3171#.VEYd1o10zmI

Preventing harmful drinking

http://www.nice.org.uk/guidance/ph24

Smoking cessation services

http://www.nice.org.uk/guidance/ph10

Drug treatment and recovery

http://www.scotland.gov.uk/Publications/2010/08/18112230/0

Using cycling helmets

http://summaries.cochrane.org/CD003985/INJ_campaigns-to-encourage-children-to-wear-cycle-helmets

Education

A

Scottish Government Research

http://www.scotland.gov.uk/Topics/Research/by-topic/education-and-training

Education Endowment Foundation

http://educationendowmentfoundation.org.uk/toolkit/

Joseph Rowntree Foundation

http://www.jrf.org.uk/publications

P

Attainment in writing

https://www.gov.uk/government/publications/the-research-evidence-on-writing

Raising attainment/changing attitudes

http://www.jrf.org.uk/sites/files/jrf/education-attainment-interventions-full.pdf

Crime and Justice

A

Scottish Government Research

http://www.gov.scot/Topics/Research/by-topic/crime-and-justice

Centre for Youth and Criminal Justice

http://www.cycj.org.uk/resources/

P

Reducing reoffending

http://www.gov.scot/Resource/0038/00385880.pdf

Reducing reoffending

https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/243718/evidence-reduce-reoffending.pdf

Sport

A

Scottish Government Research

http://www.gov.scot/Topics/Research/by-topic/sport

P

Examining legacy of major sporting events

http://www.gov.scot/Resource/0044/00449028.pdf

Barriers/enablers to regular exercise

http://www.gov.scot/Publications/2006/09/29134901/0

Environment

A

Scottish Government Research

http://www.gov.scot/Topics/Research/by-topic/environment

P

Reducing climate change

http://www.gov.scot/Resource/Doc/340440/0112767.pdf

All areas

A

Evidence for Policy and Practice Information Coordinating Centre (EPPI)

http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=60

* A = Archive of relevant publications, P = specific publication

A fictitious example:

How the evidence base supports an intervention to promote young women's physical activity

Intervention (what are we doing?)

Evidence (why are we doing this?)

- This project aims to increase physical activity from childhood into adulthood.

- Multiple international systematic reviews, drawing on cross-sectional and longitudinal studies have demonstrated the positive impact of physical activity on physical and mental health (see Scottish Government Literature Review, 2004). Physical activity habits have been shown to become established within childhood.

- The project is targeted at girls in the final year of primary school and first two stages of secondary school

- Statistical evidence shows that women are more likely to do little or no physical activity than men and that this divergence from their male counterparts begins around the age of 11 (Scottish Health Survey, 1998. 2003)

- A choice of team and individual activities will be offered each week. e.g. dance or dodgeball. An emphasis will be made on enjoyment over competition or skill development. There will be no performances or leagues.

- A systematic review of the international literature on promoting physical activity, highlighted a need for greater choice for young people, including non-traditional options. Reviews of quantitative and qualitative research by NICE (2007) demonstrate that competition and fear of having to perform may be barriers to taking part in physical activity, particularly for adolescent girls. However, enjoyment has been shown to be a key factor in overcoming these barriers (NICE 2007, Rees et al. 2006)

- Social media will be used to promote activities and encourage network-building between participants.

- The same reviews by NICE and case-study analysis by the British Heart Foundation (2011) have shown that peer approval and peer participation in physical activity encourages others to join in.

Step 3: Draw a logic model

Show how the project should work:
Clear links between resources, activities and outcomes

What are logic models?/Who can use them?

What are logic models?
Logic models are step-by-step diagrams which simply show:

- What you're hoping to achieve in the long run (long-term outcomes)

- The process (short and medium term outcomes) through which your planned activities can be expected to lead to long-term aims.

- What resources will you need to do this (inputs)

Who can use them?
Anyone who is planning activities with particular aims in mind can benefit from using a logic model. This includes funders and commissioners, who might use them to plan how to assess applications and allocate funds in pursuit of their overall aims, as well as organisations and individuals planning behaviour change projects or services.

A Logic Model Template TO USE

This blank template can be found here:
http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

A project-level logic model

The following logic model shows how a fictitious project, aimed to increase young women's physical activity levels is expected to work. It is based on international evidence about 'what works' to promote active lifestyles, particularly for young women (see the Helptful Resources chapter). It shows clear links between activities and the expected outcomes, based on what research studies tells us.

This model is quite general, "real life" service providers should be a bit more detailed about the evidence they have used to design and deliver the intervention and also describe the content of activities in more detail.

Step 4: Monitor your Logic Model

Identify indicators to monitor progress of the logic model

Use the logic model to identify indicators

Once the logic model is completed, you need to figure out how you will be able to tell if your model works as predicted, or not. To do this, you should:

1. Devise "evaluation questions" - specific questions that you need to answer in order to test whether the model is working as predicted.

2. Identify specific indicators (measures or signals of some kind) that can answer these questions and therefore provide evidence that your model is or isn't working as expected.

See below for an example of how this works in relation to a specific logic model.

Example Indicators for Activities and Outcomes

Data Collection Principles

Now you've identified your indicators, you need to decide on a way of measuring or observing these things. There are lots of different methods you can use to collect this data (see page 14) but some basic principles to observe are:

- Collect data for every stage of your logic model, including resources and activities as well as outputs

- Collect data at a unit level (i.e. about every user of the service) and at an aggregate level (i.e. about the service as a whole). Unit level data can be very useful as it can tell you who the service is working for and who it isn't and you can follow the progress of individuals over time. It can also be combined to give you overall data about your service. But remember, if you only collect aggregate data you will not be able to disaggregate it and therefore collect evidence about particular individuals.

- Follow users through the project. You should collect data about users at the very start, throughout and ideally beyond completion of the project. This will enable you to evidence whether users have changed, in terms of their attitudes, behaviour or knowledge.

- Make use of numbers and stories. Collect qualitative as well as quantitative evidence. Averages and percentages can help you to assess overall trends and patterns in outcomes for service users. Talking to people, hearing about the views and experience of users and stakeholders will help you to explain these patterns.

- Don't reinvent the wheel. Standardised and validated (pre-tested) tools are available to measure such things as self-esteem, wellbeing and employability. Using these will enhance the reliability of your evidence and save you valuable time. Freely available tools are detailed here:

- http://www.clinks.org/sites/default/files/UsingOffShelfToolstoMeasureChange.pdf

- http://www.evaluationsupportscotland.org.uk/resources/tools/

- http://inspiringimpact.org/resources/ (follow link to "List of Measurement Tools and Systems")

- Be realistic and proportionate. Expensive and/or experimental projects should collect greater amounts of data than well-evidenced and established, cheaper projects. You might want to give questionnaires to all users but it would usually be sensible to carry out in-depth interviews with just a smaller sample of your users.

Data Collection Methods

Various methods can be used to collect data in relation to your evaluation questions. Data can be collected from service users, staff or outside agencies. Not all methods will be suitable for all projects. Evaluation Support Scotland have produced excellent guidance on using different approaches.

- Using Interviews and Questionnaires http://www.evaluationsupportscotland.org.uk/resources/129/

- Visual Approaches http://www.evaluationsupportscotland.org.uk/resources/130/

- Using Qualitative Information http://www.evaluationsupportscotland.org.uk/resources/136/

- Using Technology to Evaluate http://www.evaluationsupportscotland.org.uk/resources/131/

More general advice on generating useful evidence can be found in the "Evidence for Success" guide http://www.evaluationsupportscotland.org.uk/resources/270/

TIP! The most rigorous evaluations will be based on data collected using a range of methods

Step 5: Evaluate the Logic Model

Analysing your data to evaluate the project

Once you've collected some or all of your data you can use it to analyse whether or not your model is working as predicted. Analysis is not just a case of describing your data. You need to address the following questions:

1. What does the data tell you?

2. Why are you seeing these results (it could be because of your activities or external factors)?

3. What are you going do about this? How can you improve the outcomes?

Nb. Although you should definitely carry out this process at the end of your project, earlier interim analysis and evaluation is also highly valuable in order to identify problems and improve your service on an on-going basis.

Testing the Logic Model: What does the data tell you?

Did the project work as it should have? The data you've collected will help to tell you whether your model worked as predicted, at each stage of the model. The following are examples of questions you might now be able to answer.

Inputs

- Which aspects of the service were / were not evidence based?

- How much money was spent on activities? Was it sufficient?

- How many staff were employed and at what cost?

- What was staff/user ratio?

- What did the staff do?

- How many staff were trained

- What was the training?

- Were there enough staff to deliver the activities as planned?

- What other resources were required?

Activities and Users

- Who were the target group and was the intended target group reached?

- What was the size of the target group/ their characteristics?

- What were the activities/content?

- How many participants were recruited? How successful were recruitment procedures?

- How many of the target group participated, how many completed and how many dropped out?

- How many sessions were held?

- How long was an average session?

- Did staff have the right skillset to deliver the content?

Outcomes

- How many improved or made progress/did not improve or make progress?

- What were the characteristics of the users who made progress?

- What were the characteristics of the users who did not make progress?

- What type of progress was make e.g. skills, learning?

- Did users achieving short-term outcomes go on to achieve longer-term outcomes?

Explaining Outcomes: Assessing Contribution

Given the complexity of the social world, it is very unlikely that any single project can make a difference to people's behaviour on its own. Where change is evidenced in users (both positive and negative), it is likely that there are multiple causes for this and your project will only be a part of this.

Without using a randomised control trial (which as we have said is often impractical), it is very difficult to really measure the contribution of a single project. However, we can get a broad sense of the relative importance of the project and how it might have contributed to change, in conjunction with other influences

There are two key ways of doing this:

1. Subjective views on contribution

2. Identifying potential outside influences

Subjective Views on Contribution

Users, staff and other stakeholders are valuable source s of evidence in order to assess the relative contribution of your project to observed changes in users, in relation to other influences. You can:

1) Ask users whether they received other forms of support or influences on their behaviour?

2) Ask users to rate the extent to which each form of help contributed to their success, for example, did they say it was the project, their family, friends, another intervention or their own desire to succeed?

3) Ask others who know the users (e.g. family, teachers, social workers) to rate the relative influence of the project on observed changes.

Limitation!

Asking users and staff to judge the influence of a project runs the risk of 'self-serving bias'. This is the well-established tendency for people to take the credit for success and underplay external factors. One way to limit this tendency is to tell staff, users and other participants that you will be asking others to also assess the contribution of the project. Be honest about this limitation in your evaluation reports.

Identifying Potential Outside Influences

By thinking about other potential influences, outside of your project, which might also have influenced behaviour change, you can put your own evidence into context.

Having identified potential influences, you may then be able to exclude or acknowledge whether they actually influenced your own users.

For example, in relation to the project on young women's physical activity, potential influences you might consider are:

- The weather - Unusually good or poor weather might have encouraged participation in the project and/or other kinds of physical activity.

- Local facilities - The opening or closure of sports and leisure facilities might have encouraged or discouraged physical activity.

- Economic conditions - Changes in employment or income levels for families could impact on user participation in the project and outside forms of physical activity (even if free - travel costs may impact).

What can you do to improve?

The crucial next step in the evaluation process is to use your explanations of outcomes in order to improve your model.

- Can you address any issues at the input stage (e.g. issues with staff training or resources)?

- Should you extend activities which appear to have been successful?

- Is it best to stop or redesign activities which the data suggests are ineffective?

- Can you improve the model to better target groups with negative outcomes?

- Can you do anything to address external factors which have negatively impacted? E.g. provide transport

Helpful Resources

General advice on proportionate evaluation for small-scale projects

http://project-oracle.com/standards-of-evidence/

http://www.clinks.org/community/blog-posts/how-can-we-make-evidence-easier#comment-form (see embedded presentation)

Evaluation Plan Worksheets

http://www.uwex.edu/ces/pdande/planning/pdf/EvaluationPlanWorksheet.doc

http://learningstore.uwex.edu/assets/pdfs/G3658-1W.PDF

http://project-oracle.com/uploads/files/2.3_Project_Oracle_-_Evaluation_plan_example_and_template_-_June_2014.pdf

http://www.ces-vol.org.uk/Resources/CharitiesEvaluationServices/Documents/Monitoring%20and%20evaluation%20framework.pdf

References

British Heart Foundation (2011) Translating the evidence: What works for physical activity? Sharing evidence based practice. Loughborough: British Heart Foundation National Centre for Physical Activity and Health.

NICE (2007) Correlates of physical activity in children: A review of quantitative systematic reviews. London: National Institute for Clinical Excellent. Accessed on 26/09/14 at http://www.nice.org.uk/guidance/ph17/resources/promoting-physical-activity-for-children-review-2-quantitative-correlates-2

NICE (2007) The views of children on the barriers and facilitators to participation in physical activity: A review of qualitative studies. London: National Institute for Clinical Excellent. Accessed on 26/09/14 at http://www.nice.org.uk/guidance/ph17/resources/promoting-physical-activity-for-children-consultation-on-the-draft-guidance5

Nutley , S. Powell, A. and Davies, Huw (2013) What counts as good evidence? Provocation paper for the alliance of useful evidence. London: Alliance for Useful Evidence

Petticrew, M. and Roberts, H. (2003) Evidence, hierarchies and typologies: horses for courses. Journal of Epidemiology and Community Health. 57:527-529

Rees, R. Kavanagh, J. Harden, A. Shepherd, J. Brunton, G. Oliver, S. and Oakley, A. (2006) Health Education Research. 21: 6 806-825

Ruiz, J. (2004) A literature review of the evidence for culture, the arts and sports policy. Edinburgh: Scottish Executive

Scottish Executive (2003) The Scottish health survey: Summary of key findings. Edinburgh: Scottish Executive. Accessed on 26/09/2014 at http://www.gov.scot/Resource/Doc/924/0019811.pdf

Shaw, A. McMunn, A. and Field, J. (2000) The Scottish health survey 1998. NHS Scotland. Accessed on 26/09/14 at http://www.sehd.scot.nhs.uk/scottishhealthsurvey/sh8-00.html