This is part two of Len Cook’s reflections on the 2016-17 year and coincides with the release of the annual report for the Families Commission/Superu.
Superu has been just one of several developments of recent years aimed at lifting the evidence base for social services. These have included the appointment of departmental science advisors working with the Prime Minister’s Chief Science Advisor, the establishment of the Integrated Data Infrastructure by Stats NZ, the Treasury analytics group, the Social Investment Agency, as well as Superu.
The accessibility and form of communicating evaluation studies is especially important to their wider application and influence on improving practice both within government and across community organisations. Superu has demonstrated that a critical mass of experts in the field of evaluation could change the evaluation capability of the social services sector as a whole. Much of our work has been highly applauded by others.
Without such a critical mass of experts, the benefits of much good work done by smaller groups working in isolation almost always stays in-house, remains unpublished and is rarely seen by the many more who could be influenced by it.
Superu has raised expectations outside of government from its demonstration of what can be done here and those expectations are unlikely to die now with the disestablishment of Superu.
I take immense satisfaction from what we achieved in 2016/17. I’d like to thank the Board, Chief Executive and staff for their contribution to what was a successful but challenging year. In particular, I would like to note the Board’s great appreciation for the effectiveness and humanity with which the [former] Chief Executive, Clare Ward, has managed the dis-establishment of Superu.
This is the final annual report of Superu when it has operated at full capacity, and it is timely to identify some reflections on the social services that have emerged as we have done our work. In some ways it can be read as a stock-take of the condition of evaluation across the social services sector. I would add now some personal insights, which point to questions that need more thoughtful consideration as the commitment of governments to evidence-based policy develops.
People are by nature highly variable in their characteristics, and they may describe these in different ways depending on the circumstances or on how the information has been sought. We need to recognise that this natural variability brings a degree of uncertainty in any form of evidence-gathering and modelling, and that the implications of ignoring this uncertainty can include wrongly placed confidence in methods of screening, estimating pathways and the quality of services.
Where particular characteristics of people are important for screening criteria, such as predictive modelling, this variability needs to be monitored across different communities because of the certainty of inaccuracies in determining eligibility. We need to ensure that the certainty of selection errors and their effects are recognised in both the operating rules and autonomy of front line staff.
We know well that many policies have been introduced without the support of a strong evidence base and this will not change quickly, nor will change always be possible. There can be no assurance given to their consumers of the quality of these or any other services until the social services sector can deliver comprehensively on a strong commitment to continuous improvement. This focus is not apparent at present. Consumers of social services have very limited means to hold or withdraw trust. Consequently they have to rely on how services monitor themselves for assurance or await the occasional independent review.
The work of Superu in making available evidence about whānau points to the necessity of a greater understanding of diversity and the contribution of cultural capital, not only between different cultures but also within distinct cultures. Given that these differences are rarely identifiable in the measurement and modelling that underpins policy, their existence needs to be recognised in operational practices which need to be appropriately monitored.
We have yet to understand what can be learned about service delivery in general from the experiences of the Whānau Ora programme. Māori, New Zealand’s Pacific community and Asian New Zealanders are all quite different from Pākehā in their population structure, family forms, family formation and life course, which need monitoring and analysis as they are also changing in different ways.
In the third and final part, to be published on 14 December 2017, Len looks at the benefits and disadvantages of using administrative data to solve social issues, and the need for continuous improvement in the public sector.