The 5-Step Approach to Evaluation

Designing and Evaluating Interventions to Reduce Reoffending: Summary

Guidance for service providers, planning partnerships, funders and commissioners (Summary version)

Authored by
Catherine Bisset
Justice Analytical Services

The 5-Step Approach to Evaluation

Designing and Evaluating Interventions to Reduce Reoffending: Summary

Guidance for service providers, planning partnerships, funders and commissioners (Summary version)

Authored by
Catherine Bisset
Justice Analytical Services

Table of Contents

Foreword

Introduction

Background: The tricky business of measuring impact in a messy world

The 5-Step Approach

Step 1: Identify the problem

Step 2: Review the evidence

Step 3 : Draw a logic model

Step 4: Monitor your logic model

Step 5: Evaluate the logic model

Helpful resources

Foreword

"I am delighted to introduce this easy-to-use 5-step evaluation guidance which is grounded in tried and tested methods. Not only should it help evaluate services of any size, but also to design more effective services from the outset.

This practical guide summarises the key evidence on 'what works' and is packed with examples making it a valuable resource for anyone who wants to assess the contribution they make to reducing reoffending including funders, planning partnerships, service providers and service staff".

Nicola Edge
Head of Justice Analytical Services
Scottish Government

Introduction

This evaluation pack is aimed at both service providers and funders who aim to reduce crime and reoffending.

For funders and planning partnerships, it aims to:

- Offer a strategic, evidence-based and outcomes-focused planning tool

- Offer guidance on how to assess evaluations from service providers and therefore direct funding to greatest effect.

- Demonstrate the role you can play in promoting and enabling high quality evaluations from those you fund.

For service providers, it aims to:

- Provide guidance on planning an evidence-based service with a "built in" evaluation process

- Provide guidance and resources for you to effectively assess, understand and demonstrate how well your service is working in relation to your aims.

- Offer an alternative to randomised control trials, using a "logic model" approach to evaluation, which any service provider can use to evaluate any intervention, regardless of size.

- Encourage continual review and improvement of services.

Background: The tricky business of measuring impact in a messy world

How the 5-step approach came to be

A Scottish approach to evaluation

Co-production
Our approach to evaluation enables funders and service providers to work together in pursuit of their shared aims - to improve outcomes for service users and communities. The 5-step approach also engages with service users' views as a resource for evaluation rather than seeing users solely as an object to be measured. In fact, most complex social outcomes can ONLY be achieved if we make a distinctive, yet joined-up contribution over a sustained period of time.

Asset-based
The 5-step approach focuses on ways in which evaluation is possible for services of any size, rather than expecting all services to use an experimental evaluation method which may not be appropriate or possible for smaller, community-based organisations. The 5-step approach allows even the smallest service to demonstrate the contribution they are making to change.

An Improvement Culture
Evaluation enables improvement and even the most successful service can always be developed further. Furthermore, with the 5-step approach, evaluation is an on-going process, not something to be saved for last. This means that services can be continually improved in order to best meet the needs of their users.

How do you know if you are making a real difference to users (making an impact)?

It's not easy to find out if you're making a real difference to people, especially in the chaotic real world. There are 100s of variables which can effect people's attitudes, motivations and behaviour. So how can you tell if your project is making any difference?

Researchers and scientists generally agree that BEST way to determine if your project or service has made a difference is to use a randomised control trial (RCT), sometimes referred to as an "impact evaluation" but these are not easy to do in practice, especially in a complex social setting.

What is evaluation really for?

Although doing evaluation requires the use of techniques and tools, bear in mind that its overall purpose is to help you (re) design services, ask questions, gather evidence, interpret the evidence, communicate important information about your service and take informed decisions. In this sense, the ability to ask relevant questions and clearly communicate the answers at the right time to the right people are key skills in making evaluation useful.

The 5-Step Approach

The 5-Step approach to evaluation

Identify the problem

If your ultimate aim is to change people's attitudes, emotions or behaviour, you need to be clear and explicit what it is you are trying to change and why there is currently a need for this to happen.

Review the evidence

Interventions should be clearly structured and designed using robust evidence so it is important to be familiar with the results from relevant 'what works' and desistance evidence-base. If the aim of the intervention is more specific, for example to promote recovery from drug addiction or to improve parenting skills then also track down the relevant evidence-base and embed the findings into how the service works.

Draw a logic model

A logic model is a simplified diagram which shows, step-by-step, why the activities you plan should achieve your aims. The logic model forms the basis for evaluating the whole project - you are going to test whether these steps happened as you predicted.

Identify Indicators and monitor your model

Use the logic model to identify indicators (i.e. measurements or observations) that things actually happen as you predicted. You will need to collect data about your project FROM THE START on inputs, activities, users, short, medium and long-term outcomes.

Evaluate logic model

Analyse the data you've collected on your various indictors to evaluate how well your project worked for your various users. Report on whether your data suggests the logic model worked as planned. Be honest about any areas which were less effective. Use this to improve your service.

Step 1: Identify the problem

Identify the Problem

Before it is possible to design an effective service, it is essential that you are absolutely clear what attitudes, emotions or behaviours you are trying to change and why this should be a priority in the context you're intending to work.

An example:

WHAT is the problem? Research studies show that improving the quality of family relationships for female prisoners can prevent reoffending AND reduce the risk of their children becoming involved in crime. However, there are no specialist family interventions in X prison for short term female prisoners.

WHY is this a problem? Poor family relationships decrease the likelihood of desistance from crime which is not only costly to society but can also increase the likelihood that the children of female prisoners develop mental health problems, drop out of school and get involved in crime.

What is your ULTIMATE AIM? Help to reduce the frequency of reconvictions of short term female prisoners by improving their family relationships and reduce the risk of their children becoming involved in crime.

Step 2: Review the evidence

For best results, use a range of evidence

To draw the most robust conclusions about 'what works,' and why, you should take account of evidence produced through a range of methods.

- Quantitative studies (including the results of RCTs and impact evaluations) might help you to establish what usually works and for whom.

- Qualitative work (e.g. interviews with users who 'succeed' and 'fail' and/or with practitioners) might help you to understand the processes through which interventions work or don't work and consider why barriers may exist to achieving your aims.

TIP! If you are short on time and resources, systematic and/or literature reviews are an excellent source of evidence. They often analyse both quantitative and qualitative studies on a particular topic and should do the work of summarising all this evidence for you.

Finding Evidence

When time and resources, are limited, evidence reviews (also called systematic reviews or literature reviews) are a realistic solution - enabling an overview of the evidence in a relatively short time.

Online databases and archives are the most convenient means through which to locate evidence reviews. The following slides provide a summary of the evidence on reducing crime and reoffending and links to full reviews. However, the following databases can be of general help in locating relevant evidence:

Search academic databases:

http://www.mendeley.com/dashboard/

http://scholar.google.co.uk/

Search government archives:

http://www.gov.scot/Publications/Recent

https://www.gov.uk/government/publications

TIP! Try searching for "evidence/literature/systematic review" + your behaviour change aim (i.e. "reoffending", "impulse control", "collective efficacy", "parenting" or "motivation").

Reducing Crime

The following three groups of points summarise the evidence on Reducing Crime.
The full evidence review 'What Works to Reduce Crime' can be found here:
http://www.gov.scot/Publications/2014/10/2518

Tackle the root causes of crime

- Low self control in children is linked to offending

- Parenting programmes are effective in improving self-control

- Social skills training designed to improve emotional intelligence, may help reduce delinquent behaviour

- Offending is linked to abuse, neglect, exposure to domestic violence and parental substance misuse

- Quality of care of children and young people and protection from abuse and neglect are key

- Identification of abuse and neglect at the earliest stage

Address key social factors

- Retain an attachment to school

- Staying at school is a protective factor

- Exclusion could be a significant risk factor

- Behavioural boundary setting is key

- Diversion activities e.g. sport play an important role

- Enable children to realise their potential

- Holistic employment programmes that also provide social and educational support can be effective

- Minimise the impact of criminal justice sanctions on family bonds

- Restrict access to alcohol

- Tackle drug and alcohol abuse (improving social control through effective parenting may play a part)

Deterrence and Changing the Situation

- Detection and punishment used alone are ineffective.

- People are more likely to comply with rules if they are perceived to be fair and legitimate not because they fear punishment.

- Tackling areas of 'concentrated disadvantage' is the most important step to take to reduce crime

- People offend less when communities look after their areas - 'collective efficacy' has been found to be an important factor in reducing crime.

- The certainty of punishment (increasing the likelihood of detection) is more effective as a deterrent than the severity of punishment

- Increasing 'security by design' reduces crime (deadlocks, alarms etc)

Reducing Reoffending: Desired outcomes based on criminogenic needs

'What works' evidence matrix

The following table describes the findings from the international 'what works' evidence on reducing reoffending.

The results are generated by quantitative randomised controlled trials of programmes and interventions.

Links to full evidence reviews can be found in the 'Helpful Resources' section.

There are more evidence summaries in the FULL version of this pack.

Risks to reoffending (evidence-based)

Indicator that the risk is present in an individual

Desired intermediate outcomes

Approaches that work to address the risk

Promising approaches but more evidence needed

Limited social skills, problem-solving skills and poor emotion management

Impulsive, pleasure-seeking, irritable, poor recognition of problems, poor problem-solving skills, poor social skills, lack of awareness of consequences of actions

Skills in problem-solving and perspective taking
Emotion management skills

Structured CBT programmes such as cognitive skills training
Restorative Justice Conferencing

No evidence identified but trained supervisors/mentors could help offenders engage in CBT programmes

Criminal attitudes

Rationalisations for crime, negative attitudes towards the law, negative attitudes to supervision and to society as a whole

Development of pro-social attitudes and a non-criminal identity

Structured CBT programmes such as cognitive skills training and cognitive restructuring techniques

Pro-social modelling, positive supervisor/mentor and staff interactions
Supervisors/mentors challenge anti-social attitudes

Criminal friends

Criminal friends, isolation from pro-social others, easily influenced by criminal associates

Criminal friends replaced by prosocial friends and associates

More evidence needed

Mentoring, circles of support and accountability (for sex offenders)

Lack of positive recreation or leisure activities /anti-social lifestyle

Lack of involvement and satisfaction in prosocial recreational activities. Regular activities encourage offending, recklessness and risk taking behaviours

Participation in pro-social recreational activities, sense of reward form pro-social recreation and sustained involvement in pro-social lifestyle

More evidence needed

No evidence identified but supervisors/mentors could aim to engage offenders in pro-social activities

Drug misuse

Uses drugs, injects drugs, unmotivated to tackle drug misuse, drug use and obtaining drugs a major occupation

Substance use reduced or stopped

CBT programmes, detox, opiate substitution therapy (for acquisitive opiate-addicted offenders) psycho-social support to maintain abstinence,12 step programmes, structured, therapeutic communities for drug misuse.

No evidence identified but supervisors/mentors could help offenders engage with drug programmes

Alcohol misuse

Binge drinking, long term alcohol misuse, violent when intoxicated

Reduced alcohol use or stopped drinking, reduced through disturbances promoted by drinking

More evidence needed

Supervisors/Mentors could help offenders engage with 'promising' programmes which address the interaction between alcohol and violence

Dysfunctional family relationships

Poor family relationships, no current relationship, no previous experience of close relationships, manipulative lifestyle

Conflict reduced, positive relationships, enhanced warmth and caring, reintegration into (non-criminal) social and family groups
Strengthened family ties improving family and intimate relationships, improving parenting behaviours and increasing acceptance into communities and social networks

Therapeutic approaches for young adult offenders that involve the family

No evidence identified but supervisors/mentors could help young offenders engage with therapeutic approaches
Supervisors/mentors could also help offenders engage in 'promising' approaches, namely relationship coaching interventions and they could also facilitate family visits to prison

Unemployment

Poor performance, low satisfaction in work, lack of work-related skills, poor attitude to employment, lack of qualifications

Work skills, good interpersonal relationships at work, reward and satisfaction at work
Long term employment and increased employment skills

Employment-focussed programmes in which offenders can secure real jobs they enjoy.

Gaining work related qualifications, gaining employability skills
Work related support/mentoring

Homelessness

No fixed abode or transient

Finding and keeping suitable housing

More evidence needed

No evidence identified but supervisors/mentors could assist homeless offenders find homes and retain them

Low motivation and/or self-efficacy

Unmotivated to desist and/or the belief that they do not possess the skills to desist from crime

Offenders are highly motivated to engage with supervisors and interventions and offenders are confident they have the skills to desist from crime

Offenders build positive trusting relationships with skilled, empathetic and flexible mentors, collaborative goal-setting

No evidence identified

Effective Practice - Women offenders

- Relationships with others have a stronger influence on women's offending than on men so they are key to desistance: Women desisters say they have strong social support from others and employ strategies for avoiding situations which could lead them back into offending.

- Interventions should be delivered by interpersonally skilled staff who build a consistent and trusting relationship with offenders.

- Interventions are most effective if they start in prison and continue when women are released, address criminogenic and non-criminogenic needs simultaneously and are well targeted and sequenced.

- To reduce reoffending, interventions should help women improve their financial situation, secure suitable and safe housing, establish loving bonds with children, tackle drug abuse in a residential setting and help women form positive relationships.

- Women offenders value help to solve practical problems such as accommodation, childcare and welfare benefits. These short-term needs may have to be addressed before women are ready to engage with interventions or address longer term needs such as education or employment.

- Substance misuse has a stronger relationship with reoffending in women, and women are more likely offend to support others' drug misuse as well as their own. However, some research has shown that recreational and occasional drug use are not strong predictors of reoffending in women, which suggests that intensive interventions should be targeted at drug use that is criminogenic

- Some social conditions that promote desistance in women are outside the control of some formal interventions - maturation, support from family and friends and establishing healthy personal relationships.

Effective Practice - Desistance theory

Some research is beginning to shed light on the process of desistance from crime, and (to a lesser extent) on the potential role of criminal justice social work supervision in facilitating that process. Although there has been relatively little empirical research on the latter subject, a body of theorising has emerged which, follows the idea that probation practice should become 'desistance-focused' seeks to interpret desistance research for practice. Reviewing the available research , these efforts to interpret desistance research for practice tend to stress (albeit to varying degrees) eight central themes:

- Desistance is likely to involve lapses and relapses. There is value, therefore, in criminal justice supervision being realistic about these difficulties and to find ways to manage setbacks and difficulties constructively. It may take considerable time for supervision and support to exercise a positive effect.

- Since desistance is an inherently individualised and subjective process, approaches to criminal justice social work supervision must accommodate and exploit issues of identity and diversity. One-size-fits-all interventions will not work .

- The development and maintenance not just of motivation but also of hope become key tasks for criminal justice social workers .

- Desistance can only be understood within the context of human relationships; not just relationships between workers and offenders (though these matter a great deal) but also between offenders and those who matter to them.

- Although the focus is often on offenders' risks and needs, they also have strengths and resources that they can use to overcome obstacles to desistance - both personal strengths and resources, and strengths and resources in their social networks. Supporting and developing these capacities can be a useful dimension of criminal justice social work.

- Since desistance is in part about discovering self-efficacy or agency, interventions are most likely to be effective where they encourage and respect self-determination; this means working with offenders not on them.

- Interventions based only on developing the capacities and skills of people who have offended (human capital) will not be enough. Probation also needs to work on developing social capital, opportunities to apply these skills, or to practice newly forming identities (such as 'worker' or 'father')

A fictitious example of an evidence-based proposal:

How the evidence base supports an intervention to design a throughcare intervention for short-term prisoners

Intervention (what are we doing?)

Evidence (why are we doing this?)

- This project aims to increase support and interventions for short term prisoners released from prison

- Several international reviews, drawing on randomised controlled trials and qualitative research have demonstrated the positive impact of one-to-support from highly skilled practitoners and needs-led interventions on desistance from crime (see Scottish Government Literature Review, 2011 and 2015). There is also some evidence from impact evaluations that a lack of pre-release planning and poor access to employment, support and accommodation after leaving prison leads to reoffending (Scottish Govt review 2015)

- The project is targeted at male short term prisoners

- Although male prisoners are at a higher risk of being reconvicted than women, they are less likely to take up voluntary throughcare (see Throughcare review 2012).

- Contact by a fully trained throughcare officer will be made 1 month after sentencing. They will spend the first month building a relationship with prisoners before the first of 3 needs assessments are conducted and 'whole person' pre-release plan is developed.

- A systematic review of the international literature on throughcare and resettlement highlighted that needs assessments are higher quality if practitioners give prisoners time to settle into prison, build a trusting relationship and if the needs assessment considers the whole person including family and influences. Research with offenders also shows that trained practioners who use a flexible approach and strong interpersonal skills are able to keep offenders motivated and engaged.

- Practitioners will accompany prisoners through the gate to link them with services and for 3 months after release

- A Canadian review and the international review mentioned above highlighted the need for practitioners to connect prisoners with services once they return to the community. The highest risk of reoffending is 3 months after release from custody (Howard, MoJ 2011)

Step 3 : Draw a logic model

Show how the project should work
Clear links between resources, activities and outcomes

What are logic models? / Who can use them?

What are logic models?
Logic models are step-by-step diagrams which simply show:

- What you're hoping to achieve in the long run (long-term outcomes)

- The process (short and medium term outcomes) through which your planned activities can be expected to lead to long-term aims.

- What resources will you need to do this (inputs)

Who can use them?
Anyone who is planning activities with particular aims in mind can benefit from using a logic model. This includes funders and commissioners, who might use them to plan how to assess applications and allocate funds in pursuit of their overall aims, as well as organisations and individuals planning behaviour change projects or services.

A Logic Model Template TO USE

This blank template can be found here
http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

A project-level logic model

The following simple logic model shows how a project aimed at improving family relationships for people on bail, improve employability and reduce reoffending is expected to work. It is based on international evidence about 'what works' to reduce reoffending and on research which highlights the importance of promoting positive relationships between workers and service users. It shows clear links between activities and the expected outcomes, based on what research studies tells us.

This model is quite general so service providers should be a bit more detailed about the evidence they have used to design and deliver the intervention and also describe the content of activities in more detail.

Another project-level logic model can be found in chapter A parenting skills logic model and there are more examples in the pack entitled 'The 5-step Approach to Evaluation: Designing and Evaluating Behaviour Change Interventions'.

A simple supervised bail logic model

Inputs

Outputs

Outcomes

Activities

Participation

Short‑term

Medium‑term

Long‑term

Bail workers (criminal justice social work/ third sector)

Bail offices

Time

Money

Partners: Judiciary
Defence agents
Police
Procurators Fiscal

Evidence (support)

Suitable candidates for SB identified and given SB

Bail workers meet with bailees 2-4 times per week for an hour

Bail workers assess bailee needs and signpost where appropriate

Bailees and potential bailees

Bailees feel they get on well with bail workers

Bailees feel motivated to attend meetings and comply with conditions

Bailees feel motivated to take up signposted services

Bailees attitudes to their behaviour and aspirations change

Bailees attend meetings

Bailees comply with conditions

Bailees engage with signposted services

Bailees change behaviour and aspirations

Compliance leads to community sentence

Bailees stay in the community and out of prison

Enhanced bailee relationships with family

Reduced reoffending by bailees

Enhanced employability

A parenting skills logic model

Situation: A recent parent survey in primary schools showed that 76% of parents of 3-10 year olds had struggled with dealing with tantrums in the last year. 59% of those parents said they felt 'stressed'.

Source: University of Wisconsin

Step 4: Monitor your logic model

Identify indicators to monitor progress of the logic model

Use the logic model to identify indicators

Once the logic model is completed, you need to figure out how you will be able to tell if your model works as predicted, or not. To do this, you should:

1. Devise "evaluation questions" - specific questions that you need to answer in order to test whether the model is working as predicted. As data collection and analysis can be very resource intensive, only ask what is most important to know.

2. Identify specific indicators (measures or signals of some kind) that can answer these questions and therefore provide evidence that your model is or isn't working as expected.

See the Parenting Skills example for how this works in relation to a specific logic model.

Warning! Measuring outcome indicators for national strategic programmes /reform is not easy. The reality of collecting outcomes data from 1000's of individuals who flow in and out of services and systems across the country can be prohibitively difficult.
The following questions need to be addressed:

- What outcomes data is relevant to measuring performance?

- How is the data going to be collected and how frequently?

- Who is responsible for collecting the data and analysing it?

- Can data be collected and analysed consistently across a range of areas?

- Are outcomes completely within the sphere of influence of the organisation(s) who is being evaluated/performance managed or are outcomes heavily influenced by external factors?

If it is not feasible to collect outcomes data, then collecting information on the delivery of activities and outputs as per the logic model is advised.

Use the logic model to set evaluation questions to identify indicators. This will guide the collection of data:
Parenting skills example

Data Collection Principles

Now you've identified your indicators, you need to decide on a way of measuring or observing these things. There are lots of different methods you can use to collect this data (see page 19) but some basic principles to observe are:

- Collect data for every stage of your logic model, including resources and activities as well as outputs

- Collect data at a unit level (i.e. about every user of the service) and at an aggregate level (i.e. about the service as a whole). Unit level data can be very useful as it can tell you who the service is working for and who it isn't. and you can follow the progress of individuals over time. It can also be combined to give you overall data about your service. But remember, if you only collect aggregate data you will not be able to disaggregate it and therefore collect evidence about particular individuals.

- Follow users through the project. You should collect data about users at the very start, throughout and ideally beyond completion of the project. This will enable you to evidence whether users have changed, in terms of their attitudes, behaviour or knowledge

TIP! Focus on finding indicators that measure the quality of what people do (activities) -unless people deliver a service to a high standard, it is unlikely that outcomes will materialise. Also, if outcomes are hard to measure, focus on quality assurance indictors.

- Make use of numbers and stories. Collect qualitative as well as quantitative evidence. Averages and percentages can help you to assess overall trends and patterns in outcomes for service users. Talking to people, hearing about the views and experience of users and stakeholders will help you to explain these patterns.

- Don't reinvent the wheel. Standardised and validated (pre-tested) tools are available to measure such things as self-esteem, wellbeing and employability. Using these will enhance the reliability of your evidence and save you valuable time. Freely available tools are detailed here:

- http://www.clinks.org/sites/default/files/UsingOffShelfToolstoMeasureChange.pdf

- http://www.evaluationsupportscotland.org.uk/resources/tools/

- http://inspiringimpact.org/resources/ (follow link to "List of Measurement Tools and Systems")

- Be realistic and proportionate. Expensive and/or experimental projects should collect greater amounts of data than well-evidenced and established, cheaper projects. You might want to give questionnaires to all users but it would usually be sensible to carry out in-depth interviews with just a smaller sample of your users.

Data Collection Methods

Various methods can be used to collect data in relation to your evaluation questions. Data can be collected from service users, staff or outside agencies. Not all methods will be suitable for all projects. Evaluation Support Scotland have produced excellent guidance on using different approaches.

- Using Interviews and Questionnaires http://www.evaluationsupportscotland.org.uk/resources/129/

- Visual Approaches http://www.evaluationsupportscotland.org.uk/resources/130/

- Using Qualitative Information http://www.evaluationsupportscotland.org.uk/resources/136/

- Using Technology to Evaluate http://www.evaluationsupportscotland.org.uk/resources/131/

- More general advice on generating useful evidence can be found in the "Evidence for Success" guide http://www.evaluationsupportscotland.org.uk/resources/270/

TIP! The most rigorous evaluations will be based on data collected using a range of methods

Step 5: Evaluate the logic model

Analysing your data to evaluate the project

Once you've collected some or all of your data you can use it to analyse whether or not your model is working as predicted. Analysis is not just a case of describing your data. You need to address the following questions:

1. What does the data tell you?

2. Why are you seeing these results (it could be because of your activities or external factors)?

3. What are you going do about this? How can you improve the outcomes?

Nb. Although you should definitely carry out this process at the end of your project, earlier interim analysis and evaluation is also highly valuable in order to identify problems and improve your service on an on-going basis.

Testing the Logic Model: What does the data tell you?

Did the project work as it should have? The data you've collected will help to tell you whether your model worked as predicted, at each stage of the model. The following are examples of questions you might now be able to answer.

Inputs

- Which aspects of the service were / were not evidence based?

- How much money was spent on activities? Was it sufficient?

- How many staff were employed and at what cost?

- What was staff/user ratio?

- What did the staff do?

- How many staff were trained

- What was the training?

- Were there enough staff to deliver the activities as planned?

- What other resources were required?

Activities and Users

- Who were the target group and was the intended target group reached?

- What was the size of the target group/ their characteristics?

- What were the activities/content?

- How many participants were recruited? How successful were recruitment procedures?

- How many of the target group participated, how many completed and how many dropped out?

- How many sessions were held?

- How long was an average session?

- Did staff have the right skillset to deliver the content?

Outcomes

- How many improved or made progress/did not improve or make progress?

- What were the characteristics of the users who made progress?

- What were the characteristics of the users who did not make progress?

- What type of progress was make e.g. skills, learning?

- Did users achieving short-term outcomes go on to achieve longer-term outcomes?

Explaining Outcomes: Assessing Contribution

Given the complexity of the social world, it is very unlikely that any single project can make a difference to people's behaviour on its own. Where change is evidenced in users (both positive and negative), it is likely that there are multiple causes for this and your project will only be a part of this.

Without using a randomised control trial (which as we have said is often impractical), it is very difficult to really measure the contribution of a single project. However, we can get a broad sense of the relative importance of the project and how it might have contributed to change, in conjunction with other influences

There are two key ways of doing this:

1. Subjective views on contribution

2. Identifying potential outside influences

Subjective Views on Contribution

Users, staff and other stakeholders are valuable source s of evidence in order to assess the relative contribution of your project to observed changes in users, in relation to other influences. You can:

1) Ask users whether they received other forms of support or influences on their behaviour?

2) Ask users to rate the extent to which each form of help contributed to their success, for example, did they say it was the project, their family, friends, another intervention or their own desire to succeed?

3) Ask others who know the users (e.g. family, teachers, social workers) to rate the relative influence of the project on observed changes.

Limitation!

Asking users and staff to judge the influence of a project runs the risk of 'self-serving bias'. This is the well-established tendency for people to take the credit for success and underplay external factors. One way to limit this tendency is to tell staff, users and other participants that you will be asking others to also assess the contribution of the project. Be honest about this limitation in your evaluation reports.

Identifying Potential Outside Influences

By thinking about other potential influences, outside of your project, which might also have influenced behaviour change, you can put your own evidence into context.

Having identified potential influences, you may then be able to exclude or acknowledge whether they actually influenced your own users.

For example, in relation to a project to improve the family relationships of female ex-prisoners in the community, potential influences you might consider are:

- Outstanding warrants - If some of the women were re-arrested on outstanding charges this will have hindered participation

- Child protection issues - Concerns around the safety and well-being of children may have prevented practitioners from working with some families.

- Economic conditions - Changes in income levels for the women could impact on user participation in the project in terms of travel costs

What can you do to improve?

The crucial next step in the evaluation process is to use your explanations of outcomes in order to improve your model.

- Can you address any issues at the input stage (e.g. issues with staff training or resources)?

- Should you extend activities which appear to have been successful?

- Is it best to stop or redesign activities which the data suggests are ineffective?

- Can you improve the model to better target groups with negative outcomes?

- Can you do anything to address external factors which have negatively impacted? E.g. provide transport

Helpful resources

General advice on proportionate evaluation for small-scale projects
http://project-oracle.com/standards-of-evidence/

http://www.clinks.org/community/blog-posts/how-can-we-make-evidence-easier#comment-form (see embedded presentation)

Evaluation Plan Worksheets
http://www.uwex.edu/ces/pdande/planning/pdf/EvaluationPlanWorksheet.doc

http://learningstore.uwex.edu/assets/pdfs/G3658-1W.PDF

http://project-oracle.com/uploads/files/2.3_Project_Oracle_-_Evaluation_plan_example_and_template_-_June_2014.pdf

http://www.ces-vol.org.uk/Resources/CharitiesEvaluationServices/Documents/Monitoring%20and%20evaluation%20framework.pdf

Logic model and theory of change, templates and flowcharts
http://www.uwex.edu/ces/pdande/evaluation/evallogicmodelworksheets.html

http://www.evaluationsupportscotland.org.uk/resources/127/

http://www.clinks.org/sites/default/files/TheoryofChangeGuide.pdf

http://www.clinks.org/sites/default/files/SHIFT%20Hereforshire%20ToC.pdf

Writing an evaluation report.
http://www.uic.edu/depts/crwg/cwitguide/05_EvalGuide_STAGE3.pdf

http://www.evaluationsupportscotland.org.uk/resources/135/

An example of commissioning using key elements of the 5 step approach: Reducing Reoffending Change fund guidance
http://www.gov.scot/Topics/archive/law-order/offender-management/changefund/changefundguidance

What works to Reduce Crime - Scottish Government
http://www.gov.scot/Publications/2014/10/2518

'What works' and desistance theory -full reports
Reducing Reoffending Review- Scottish Government
http://www.gov.scotResource/0038/00385880.pdf

Strengthening Transnational Approaches to Reducing Reoffending - University of Cambridge
http://www.cepprobation.org/uploaded_files/Rep%20STARR%20ENG.pdf

Transforming rehabilitation - A summary of evidence on reducing reoffending - Ministry of Justice
https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/243718/evidence-reduce-reoffending.pdf

Campbell Collaboration Systematic Reviews (e.g. on mentoring and prison-based drug interventions)
http://www.campbellcollaboration.org/lib/?go=monograph

Discovering Desistance - McNeill et al
http://eprints.gla.ac.uk/79860/1/79860.pdf

Key practice skills research
Practitioner skills and attributes
http://strathprints.strath.ac.uk/38070/1/21st_c.pdf

Motivating Offenders to Change
http://www.amazon.co.uk/Motivating-Offenders-Change-Engagement-Psychology/dp/047149755X

The quality of probation supervision - A literature review
https://www.shef.ac.uk/polopoly_fs/1.159010!/file/QualityofProbationSupervision.pdf

Recent key texts
http://psycnet.apa.org/psycinfo/1995-98528-000

Writing an evaluation report.
http://www.uic.edu/depts/crwg/cwitguide/05_EvalGuide_STAGE3.pdf

An example of commissioning using key elements of the 5-step approach: Reducing Reoffending Change fund guidance
http://www.gov.scot/Topics/Justice/public-safety/offendermanagement/changefund/changefundguidance