The method behind the numbers: how we measure success in delivering change for charities

Posted 02 Nov 2018 | Blog | Evaluation | Pilotlight Programme

Whether it’s through the stories we tell or the yearly Impact Summary we share, we’re constantly reviewing our impact to make sure we’re delivering value for our stakeholders. But how did we decide what success looks like for charities on the Pilotlight Programme? As we’re celebrating 15 years of the Programme this year, we asked our Evaluation Consultant Anna Grey, co-founder of Cornish and Grey, to share the method behind the numbers.

What does success actually look like?

I’ve been working with Pilotlight since 2013, and over that time the way we evaluate the Pilotlight Programme has changed a lot. When I came to the charity reporting was focussed on activities and ‘outputs’, measuring things like the number of hours given by our Pilotlighters, or the number of service users supported by our Charity Partners. These are all great statistics to record if you want to showcase how the Programme operates, but they don’t tell a very full story.

I was keen to understand what success really looked like for the charities that participated. If the Programme’s aim was to make our Charity Partners more efficient, effective and sustainable, I needed to get a better picture of what change was being achieved. To do this, however, I would first need to have a clear understanding of the nature of the support being delivered.

Introducing a new way to track activity

In collaboration with the Project Management team, this brought me in 2015, to propose a new system of tracking Programme activity. In the project meetings themselves our Project Managers would now set and record aims specific to the charities they were working with. Recording these would allow us to identify in more detail what each project was trying to achieve, and what outcomes to look for in the long term. We already knew that strategy and business planning were key areas of focus, but after doing some analysis of these new ‘project aims’ other themes were revealed, most notably, improving governance and leadership.

Alongside setting project aims, we also started to gather feedback from charity leaders to explore what difference (if any) their participation had made to them, both in the short term and in the longer term. Initially, we gathered feedback through in-depth interviews and open-ended questionnaires and made some important discoveries. At the end of the project, leaders talked of increased confidence and improved leadership skills. One and two years on, they had started to implement recommendations and plans made during the programme, for example making changes to their governing bodies, leadership teams or both.

Creating a theory of change

Improved understanding of the Programme’s activity and outcomes enabled us to create a working hypothesis for how Pilotlight contributes to charities working more efficiently and effectively. We devised our theory of change for charities, as illustrated in the outcomes chain below.

How the Pilotlight Programme creates change for charities

Charity-theory-of-change-2018.jpg#asset:6029

This outcomes chain now forms the framework for articulating the change the Programme seeks to deliver and, since implementation, has helped us further monitor its success in delivering that change. Feedback is largely collected through ‘closed’ questions focusing on the outcomes above, and this theory of change generates the numbers you’ll find on the Impact pages and in the yearly Impact Summary we share externally.

Since creating and working with this framework we’ve also started to develop a theory of change for the Pilotlighters who coach and mentor our Charity Partners. Their journey is also becoming one of longer-term change, which can be seen in our latest figures: a third of Pilotlighters who have participated in the Programme for three or more years have become charity trustees. These kinds of numbers are helping us understand the journey our members take with us, and see how they are also impacted by working with the charities. You only have to look at the Pilotlighter Profiles to see that we are creating change for them too.

The future of impact measurement at Pilotlight

Developing the Pilotlighters’ theory of change is just one of the many areas I’m currently looking at. I also report on our Bespoke Programmes for our Business Partners to make sure we’re giving them value and meeting their learning and development objectives. As Pilotlight continues to grow as an organisation I’m looking forward to seeing how our ideas about impact, and how all of our programmes delivering social change, evolve.

.