A public dashboard that tracks what’s working — and what’s notOriginally Published: April 2018
In November 2017, youth employability charity Street League unveiled their new online impact dashboard, an interactive tool allowing funders, staff and the public to track how the charity is performing in real-time.
While other charities have created impact dashboards (such as Citizens Advice Bureau), Street League claims they’re the first to integrate data that’s updated monthly and is backed up by 'auditable' evidence.
Street League uses football, dance and fitness to engage unemployed 16-24 year olds in 10-12 week programs, where they learn skills and gain qualifications and work experience to help them get into work or further training.
The journey of every one of these participants is tracked in the dashboard (profiles are anonymized) from starting point, through the program, to six months after completion. Viewers can track the barriers young people faced initially, where they live, who dropped out and why, the types of jobs they went into and how many stay in work.
The charity claims that the use of real-time data, pulled directly from their participant database, is “a true step forward in transparency”: since nothing is edited, the less positive results are also clear.
“We want to lead by example and be transparent about how hard it can be, rather than painting a rosy, but false, picture,” chief executive Matt Stevenson-Dodd told Social Value UK. He was also keen to get beyond “an over-reliance on stories” and to back them up with hard numbers, as Third Sector magazine reports: “We didn’t just want to tell the stories of a handful of young people we have helped, but all of them."
So how did they do it?
1. Continually questioning, continually improving
The public dashboard is just the latest step in a journey that started back in 2010.
Back then, the charity was simply measuring participation from samples of young people from which they would “draw wild conclusions,” according to Stevenson-Dodd. “We were pretty sure we were doing lots of good stuff, but because we weren’t collecting enough data, we did not know what the good stuff was.”
So Street League worked with support organizations Impetus-PEF and Inspiring Scotland to revisit its theory of change — looking at long term goals and working backwards to define necessary preconditions. The charity also began measuring not only how many people participated, but also what happened next in their lives.
This was better, but it still focused only on “hard” outcomes (getting into employment, education and training); it missed out the bigger picture of individual starting points and barriers to progression. The charity also wanted to be sure it was focusing on those who needed the most support.
So staff began capturing more information, including people’s social, economic, religious and health backgrounds, and from 2013 organized these full data sets in a new database, with help from consultancy firms PwC and KPMG plus continued support from Impetus-PEF. From looking only at hard outcomes, the charity was now measuring its broader impact, including looking at soft outcomes (such as confidence, health and fitness) and — a key indicator of success — focusing on sustained results.
The idea for making all this public came about in 2016, when the board agreed that true transparency meant sharing both good and bad news. That year, Street League's annual report had a chapter headed: 'Last year we weren’t able to help 109 young people.' The next step, a year later, was a real-time, completely unfiltered version of the data, published directly from the internal database to the public platform.
2. Being rigorous about quality and credibility of data
It’s easy to get excited about a colourful new interactive tool — but the dashboard itself wasn’t the most complicated part of the process. Nor was it costly, as it was built in-house using part of the charity’s existing Microsoft Office package. The difficult part, Street League says, was collecting accurate data from thousands of young people in 14 cities.
The steps to quality data included:
- Undertaking a theory of change analysis to review what is measured and why
- Creating a participant database to track every participant’s journey from starting point, through the program, to outcome progression, and then six/nine/twelve months later
- Introducing a four-stage internal audit process to check the evidence behind all outcomes, for example a photocopy of a first payslip or job offer letter. Three members of staff rigorously manage and monitor the data gathered including through monthly spot checks in each region (in 2016, they rejected 48 claims that young people went into work because evidence wasn’t provided)
- Establishing quarterly data audits and development plans for every employee inputting data (managers chase up frontline staff whenever there are gaps in data, providing support or further training if needed)
3. Strong leadership
Stevenson-Dodd has led the charity since 2010, driving increases each year in the number of those achieving positive outcomes. While that initial meeting in 2016 saw trustees divided — “half of the room thought [focusing on those we hadn't been able to support] was a great idea and the other half disagreed,” he writes in a blog for Third Force News — there’s now strong board-level support for the new approach.
“There is certain business critical information that we won’t publish everywhere, but I completely believe that our impact data should be transparent,” says the chief executive, adding that “[the board’s] fears around the change have diminished.”
Head of marketing Sara McCraight told MissionBox that senior managers and trustees are now “really excited” about the dashboard and proudly show it off to partners and funders.
Street League has also called on other nonprofits to follow what they call the “Three Golden Rules” of impact reporting — the principles behind the dashboard: never overclaim; back up all percentages with absolute numbers, and support all outcomes with auditable evidence. Their #CallforClarity campaign, launched in 2016 to encourage uptake of those rules, earned the backing of nearly 150 organizations.
4. Getting staff on board
How do you get 140 staff on board with transparency — especially if it means adding to workload or having to learn to use new tools?
Start with a consistent message from the top, says McCraight, that transparency isn’t an optional extra, but part of everyone’s job. Street League also highlights the need for sufficient training and support systems to ensure there are no barriers; they initially appointed informal ‘ambassadors’ — staff members who were early adopters in each region — to support those around them by answering questions or giving extra training. And McCraight explains that prioritizing internal communication is important, which means rolling out changes and launching any reports internally before going public. A quarterly anonymous Q&A forum, using Padlet, where staff can send questions directly to the chief executive, also helps keep conversations open.
McCraight says her colleagues now feel invested in the dashboard, with “excitement ahead of the actual date we ‘reveal’ our data… staff are excited to see what has changed each month.”
Taking stock and next steps
Street League’s hard work was recognized among Third Sector magazine’s Top 10 digital charity campaigns of 2017.
It has also attracted positive feedback from funders. People’s Postcode Lottery (which has supported Street League’s core costs, including monitoring and evaluation) was quoted by Third Sector saying it is “really helpful” to be able to see progress in such detail, and that it's beneficial “being able to interact with the data and ask our own questions.”
Street League says there’ve been over 4,000 visits to the dashboard in the four months since its launch (as of January 2018), with visitors spending a leisurely average of three minutes on the page (compared with less than one minute, typically, for the rest of their website). In a survey among visitors, three-quarters of people (40 out of 52) said the dashboard had increased their understanding of the charity’s social impact.
So what’s next? After user feedback, Street League have taken steps to anonymize the data further; they have also published a password-protected, more detailed version of the dashboard for the board of trustees.
The next round of improvements, says McCraight, will include finding better ways to explain some nonprofit jargon, reducing page load time and making it more mobile friendly, and adding some sector benchmarks for comparison.