Superu’s annual Evidence to Action conference in mid-June gave me plenty of food for thought. My end-of-conference summary, which is reproduced below, was informed by the speakers – Dr Sarah Morton of What Works Scotland, Karen Field from drummond street services in Melbourne, Prof Stuart McNaughton, the Chief Education Science Advisor, Dr Monique Faleafa of Le Va and MSD’s Regional Commissioner in Northland Eru Lyndon.
All speakers acknowledged the central importance of continuous improvement, and variously highlighted the key elements of curiosity, empowerment, trust, and ambition, recognition of variation, feedback and measurement.
One of the central themes of the conference was that evidence cannot stand apart from its uses. Dr Morton spoke of linkage or exchange mechanisms that helped the translation of evidence into action. She also reminded us of the work of Carol Weiss who recognised that most evidence provides value from its general usability, rather than its influence on any single policy. Some forms of evidence can become privileged, so that other forms are either not gathered or are ignored, despite their potential.
Frameworks for accumulating experience and rethinking desired outcomes were discussed by Dr Morton. This has particular resonance for New Zealand, with our penchant for filing reports before the ink is dry. Associated with this was the ability to unlearn poor practices.
Recognition of systems thinking is critical. The importance of separating single operational incidents in social services from the findings that (rightly) trigger systemic concerns, in order to have more effective political discourse, was recognised.
Prof McNaughton reminded us that system change was in fact very hard and that careful oversight during periods of change was needed. Dr Morton noted that “what works” solutions were often simplistic answers to complex problems, and that differences in context, and the variability of people and systems could make their transfer to new environments rather fraught.
This message was reinforced by Prof McNaughton, who pointed out that much of the value expected of programmes was lost during implementation, through not controlling for the system variability that can come from many sources. If we could not understand why something worked and when, then we could not scale up successfully. Eru Lyndon gave some strong examples of where community connection changed the effectiveness and scope of what was possible.
Prof McNaughton noted that we needed to get better at applying the well-established knowledge we have. He gave the example of how what we know about brain development paths is often ignored in child and youth policy, while Karen Field pointed to eight well recognised points of transition during the life course.
Both Dr Morton and Karen Field talked about their experiences in the UK and Australia, where the political context of social change can be difficult to keep up with, and the balance between evidence and sentiment was made more complex by social media. Karen Field said that this complexity often meant that resourcing preventive measures was compromised in favour of initiatives responding to immediate need. She added that many agency performance measures were about transactions and not about how things work, and that operational goals were not always easily aligned to high level outcomes.
In an ideal world, service improvement would be based on a rich culture of evaluation and continuous improvement, which I suggest is often hard to detect, and not well embedded in New Zealand’s social services. Dr Morton challenged universities to reconsider how they might play a role in discursive thinking and knowledge broking.
Innovative services using IT and social media offer different ways of contact, and some were presented by Dr Monique Faleafa. As these technologies emerge, they provide valuable learning opportunities to recognise and respond to cohort and age differences.
Another dimension of the challenges presented by the dynamics of populations was identified by Karen Field, who talked about the tension of operating place-based programmes, when families can be very mobile. Karen also felt that the older workforce in the NGO sector in Australia was less innovative than it needed to be, and noted that the sector was attracting fewer young people.
As with Superu’s earlier Evidence to Action conferences, the presentations and discussions confirmed the emergence of a broader sense what evidence is and how it and evaluation should be valued as vital elements of free and frank advice in public administration.
Using all the evidence we have to hand – both from systemic reviews, many of which have been shelved without being fully explored, and from qualitative inquiry – will help us to generate a deeper, richer framework and lead to improved social services delivery.