Evaluating the impact of public services

The evaluation team have developed the 'spotlight toolkit' an online interactive tool and methodology designed for public service evaluation. The team launched the 'spotlight toolkit' to academics in March 2019, showcased the toolkit to attendees at the Catalyst Conference in June 2019 and begin a pilot phase in April 2020. The spotlight tookit (.pdf) has been created to enable any organisation to create their own evaluation process to assess the impact and effectiveness of their programme or services.

Find out how the spotlight toolkit works, what it can do and the benefits it provides.

Aims

The toolkit will produce bespoke outputs, which will include:

• An evaluation plan for a business case or funding bid
• An evaluation framework for a funded project, including timelines, milestones, and deliverables
• Information sharing agreements and protocols to share with partners
• Data collection tools appropriate to your needs, such as surveys, questionnaires, interview protocols, focus group plans, and data sets
• Research ethics protocols.

Contact

Dr Gina Reinhardt (gina.reinhardt@essex.ac.uk)

The Essex Data programme (ED), funded by a grant awarded to Essex Partners to support system wide transformation and delivered with the University of Essex Catalyst Project, is a safe, investigative and predictive modelling tool and data sharing platform. Pseudonymised data is matched from a range of public service organisations to provide insight, and predictive risk profiles, to support targeted early interventions.

The ED programme was the first time that partners from across the public sector in Essex (i.e. Essex Police; Essex County Council; Basildon Borough Council ) had come together to share and match data and channel the power of data to tackle some of the county’s most challenging issues.

During an initial pilot phase (August 2016 – December 2018), the ED programme project ran a series of prototypes using matched datasets to predict what challenges the future holds across four initiatives:

• School readiness
• Escalating domestic abuse
• Violence and vulnerability
• Unnecessary attendance at A&E

The University of Essex Catalyst Evaluation Team worked with Essex Partners to identify key ED programme project aims and develop evidence based methods to evaluate the impact of each aim across the initiatives.

From the results, the Evaluation Team generated recommendations to address the findings and maximise the impact of the programme. The insight generated has informed commissioning and enabled public sector organisations and community groups to co-design and co-deliver early intervention services that help improve people’s lives.

Expert knowledge was shared to provide challenge and support to the working group and to feed into the development of the risk model. Advice was given on the type and use of variables and predictors (using geographically weighted regression).

It was recommended that maximum benefits and stakeholder involvement would be gained by presenting the findings at a neighbourhood level.

The LAC Programme was designed to support local residents by helping them:

• access information
• be heard
• make choices
• identify personal strengths
• find practical ways of fulfilling tasks
• develop and use personal and local networks
• connect with community life
• access support and services.

The Catalyst project evaluated the impact of the Suffolk County Council pilot LAC programme during the period 2016-2018. Evaluation methods involved measuring how the Local Area Coordination programme has been performing against the pre-identified project aims and outcomes, and facilitating discussions for the future of the programme and how it could develop beyond the pilot stage.

The evaluation was conducted by designing a data collection guide prior to intervention delivery, and analysing that completed data collection alongside LAC programme administrative data, testimonials and a survey distributed to a range of stakeholders involved in the programme (GP practitioners, social workers, police staff).

The evaluation showed that the LAC programme:

• reduced demand on other services (health, police, education, council)
• reduced the isolation of individuals
• improved the feeling of belonging for individuals in the community
• improved the wellbeing of individuals 

Aims

A team of researchers from the University of Essex, led by Professor Pam Cox were commissioned by Colchester Borough Council’s Safer Colchester Partnership and the University of Essex Catalyst Project to produce a report that assists in the development of interventions to:

• improve the life chances of marginal young people
• make better use of community assets
• promote inclusive economic growth and prosperity
• reduce anti-social behaviour

The project involved desk research, action research, data analysis and stakeholder engagement workshops.

Findings

Findings were collated in a report that identified some worrying trends. 15% of Colchester's children live in low-income families, and 20% of local children attend schools rated less than good. Further, Essex County Council data used in the report indicates an increase in school exclusions from Colchester's secondary schools over the last three years.

At the time of the report, Colchester was one of Britain's fastest growing towns and had one of the region's largest concentrations of young people. The report calls for Colchester leaders to develop a new strategy to meet their changing needs and to boost the life chances of those among them at risk of being left behind.

Full details: ‘Young Colchester: Life Chances, Assets and Anti-Social Behaviour’ report (.pdf).

Details of the launch event.

The team

• Carlene Cornish and Ruth Weir from University of Essex
• Mel Rundle, Sonia Carr and Kaitlin Trenerry from Colchester Borough Council

This Essex County Council initiative, the Dementia Challenge, was a competition aimed at encouraging local organisations and individual citizens to develop a tool using artificial intelligence that could help both people living with dementia and their carers.

The Catalyst Evaluation team supported the 9 chosen finalists in developing their business case and embedding robust evaluation criteria during their pilot phases, and using those to develop business cases with clear evaluation frameworks.

A finalist reported “The whole six months has been an amazing experience and I need to thank so many people for their support in helping make things happen, from Kakia and Gina at the Catalyst Project.”

Read the interview with the Challenge Dementia Team (.PDF)

In the Parish Safety Volunteers Programme, trained volunteers visited homes in 2016 and provided individuals with bespoke advice on fire prevention and burglary safety. They offered advice on home security methods and mobility aids, and provided support for a number of health and wellbeing requirements.

In the Safe and Well Programme, Fire and Rescue technicians are visiting homes to provide fire safety advice, help install smoke detectors, and assess home safety levels across the county.

Aims

The Evaluation Team has produced:

• A report on Accidental Dwelling Fires in Essex County, including identifying trends, at-risk characteristics, and preferred methods of contact for at-risk groups.
• An evaluation of the Parish Safety Volunteers pilot programme, in which they found that communities receiving PSV visits experienced a significant reduction in fires compared to communities that did not receive visits.
• An event (26 November 2019) for senior management of the Essex Fire and Rescue Service team to provide advice on evaluation methods and models

The Evaluation Team has prepared:

• Analysis of Safe and Well visits, including before-and-after visit surveys and safety assessments
• The team presented their findings (.pptx) at a jointly run workshop on evaluating outcome and impact, delivered at the Local Government Association Annual Fire Conference in March 2019.

Read more on the case study (.pdf)

Researchers from the Catalyst Project Evaluation Team were commissioned by Colchester Borough Homes (CBH) to analyse and evaluate the CBH Community Plan committed to reducing domestic abuse.

This work built on the earlier work carried out by the Catalyst Project for the ‘Young Colchester’ anti-social behaviour (ASB) project commissioned by Colchester Borough Council’s Safer Colchester Partnership and the data analysis carried out with Essex Police and Essex County Council.

The extended work has mapped the geographical areas of antisocial behaviour in Colchester including domestic abuse. This mapping exercise; increased CBH’s understanding of their ASB cases, identified the geographical concentration of ASB cases, identified hotspots in the different categories of ASB (which include, for example, threatening or abusive behaviour, damage to property, drug activity, harassment – see report below). It also identified the risk factors present in areas where victimisation is highest and individual characteristics of those most at risk of ASB.

This project has provided CBH with a baseline from which the CBH Community Plan can develop their interventions and target their resources. The work will not only allow earlier intervention to addressing ASB but improves the capacity and capability within the community, in line with the new CBH Community Plan to empower members of the community to recognise domestic abuse and to report suspicions. Read ‘Colchester Borough Homes: Community Plan Analysis’ for full details of the work carried out by the Catalyst Project.

To improve measures of resilience at national and local levels in the context of emergency planning and management that inform a national disaster management policy.

Aims

To develop a Resilience Index for communities around the world, founded on an associated database that will hold resilience plans and policies from each community that is generated with a predictive agent-based model. The model and Resilience Index will inform national and local disaster management policy makers and practitioners, who will be able to input community characteristics to generate predictions based on various resource allocation scenarios.

Progress

Collection of resilience plans and policies has been under way and currently includes some 240 national disaster management and development plan documents from countries across the world. Collected policies have been converted into a dataset to enable analysis using advanced text mining techniques and machine learning.

In May 2019 the Evaluation Team showcased the Resilience Policy Database and the Resilience Policy Proximity Index. Initially, presentations were given by academics working in the areas of resilience, disaster and emergency management in different contexts (environmental, governance, legislative and regulatory).

These were followed by a presentation (.PDF) of the Resilience Policy Database and the Resilience Policy Proximity Index as well as the model and data that the team has been developing and a demonstration of the data tool and how it can be used.

Essex Partners are leading the delivery of 'The Future of Essex’, a shared vision for the county.

The Essex Partners are piloting 8 vision projects and the Catalyst Evaluation Team will evaluate the impact of the Future of Essex whole system approach and vision on individual project design and implementation .

The team will also evaluate the extent to which collaboration occurs between the separate vision projects.

Read the evaluation report produced by the Catalyst Evaluation Team