Evaluations
The increasing emphasis on demonstrating the effectiveness of policies and initiatives (to justify continuation/roll-out and to secure funding), and the focus on continuous improvement, mean that an increasing proportion of social research projects are evaluations.
Evaluations typically answer the following sorts of questions:
What is happening?
- What is the nature of the initiative/service? How is delivered? How is it received?
- How does it fit with other initiatives/services? How does it affect them?
Does it work?
- Why does it work? In what circumstances and for whom?
- Why does it not work?
What should happen next?
- Should it be continued?
- Should it be rolled-out?
- How could it be improved?
Evaluations are often categorised as being ‘formative’ or ‘process’ evaluations (which focus on delivery) or ‘summative’ or ‘impact’ evaluations (which focus on outcomes). But in reality, most evaluations are a bit of both.
As with all our research, the first stage in any evaluation is to identify the key questions which need to be answered.
It is often not possible to measure the impact of specific initiatives in social policy areas – especially the long term outcomes which are often the ultimate aim. This is partly because of the complexity of the social world and the range of other factors which might influence the outcome (randomised control trials are rarely feasible). It’s also because decisions about whether to continue an initiative can rarely wait for the long term impact and, quite rightly, there is a desire for quick feedback on how an initiative can be improved.
‘Theory of change’ and logic models can be used to develop a theory of how and why a initiative might be expected to cause an effect. This will identify the outputs and the short and medium term outcomes which are expected to lead to the longer term outcomes - and these can then guide the development of the specific evaluation questions.
The next stage is gathering and analysing evidence in order to answer the evaluation questions. We typically use a mixed method approach to cover the wide range of evidence that is normally required. Depending on the purpose of the evaluation and the nature of the initiative being evaluated, we might use several of the following methods:
- Qualitative research with participants/service users, staff involved in delivery, managers, key stakeholders
- Quantitative research with participants/service users, staff involved in delivery, managers, key stakeholders
- Observational research
- Literature reviews
- Analysis of monitoring data
- Analysis of financial data
Current Projects
Evaluation of 'Life Begins at 40'
Ipsos MORI has been commissioned by NHS 24 to conduct an evaluation of the Life Begins at 40 Service. This service aims to raise awareness of health and wellbeing among people who have recently turned 40 and offers advice and support on changes they could make to start living a healthier lifestyle. The aim of the research is to investigate the profile of service users, to identify the reasons for response and non-response to the service and to highlight any changes that can be made to improve uptake. The research will comprise a series of depth interviews, conducted across Scotland, with both users and non-users of the service, and quantitative analysis of uptake data.
Ipsos MORI contact: Carolyn Black Tel: 0131 240 3261
Evaluation of 'You First' Pilot Programme
In collaboration with the Centre for Research on Families and Relationships, based at the University of Edinburgh, we have been commissioned by the Scottish Government to undertake the evaluation of the ‘You First’ Pilot Programme - a parenting programme which has been developed by Barnardo’s Scotland to help improve outcomes for parents and their children in the most deprived areas in Scotland. The evaluation comprises both quantitative and qualitative research with parents, programme facilitators, programme managers, health visitors and other stakeholders. Analysis of administrative data containing the demographics of those who attend the programmes will also be undertaken.
Ipsos MORI contact: Jane Eunson. Tel: 0131 226 8676