Barriers to Implementing the Robodebt Royal Commission: The Audit Office Report on Welfare Timeliness and Accuracy
The Australian National Audit Office report into payment accuracy and timeliness in the welfare system was released last week. It sharply criticises the way accuracy is reported on publicly.
The Australian National Audit Office report into payment accuracy and timeliness in the welfare system was released last week. It reviews the robustness of a central survey in our social security system: The Payment Accuracy Review Programme (PARP). I still insist on calling the survey by its old name: the Random Sample Survey. My vision for it is a deep qualitative survey of real lives in the social security system—from the moment a person needs to have support to when they get it. The language of ‘payment accuracy’ reflects a mindset I will critique in this post.
The report’s headline findings received some media coverage. As the Guardian succinctly put it,
Services Australia has been accused of incorrectly giving itself near full marks in its reporting of payment accuracy to recipients.
The Audit Office sharply criticises the way accuracy is reported on publicly. By excluding errors by recipients, Services Australia graded itself at 98.9% accurate in its decision-making. The Audit Office would prefer recipient errors included— considering such errors are under the institution’s influence, if not its control. That would mean Services Australia was accurate only 81.4% of the time. Under such an approach 13.5% of the errors were overpayments while 5% represented underpayments to welfare recipients. As we will explore, however, that distribution is very much not the whole story.
Services Australia’s approach to error will be familiar to those who followed robodebt. It was this philosophy which drove the infamous statements in 2019 that “99.2% of these decisions are accurate.” By distancing itself from its ‘customers’ confusion, ability to assemble evidence and behaviour, accuracy shrinks to a narrower metric. Once we received information – however inaccurate - from the ‘customer’, did we action it accurately in line with our policy? That emphasis threatens a shift from grounded engagement with people’s vulnerability or life situations, towards a reactive, self-service ‘payments platform.’
The report is really important for a number of reasons, not just the headline recommendations. To me, it underlines key barriers facing a genuine implementation of Robodebt Royal Commission. In this post I want to show how the bureaucratic world of “Key Performance Measures”, too often reflects defensive logics detached from people’s realities and what’s important. In a ‘big data’ system what gets counted is what counts. Services Australia and the Department of Social Services are not seeing and counting what matters – particularly what matters for Recommendation 1 of the Royal Commission.
Robodebt Royal Commission Recommendation 1:
A Trap Baited with Generality
Recommendation 1 of the Robodebt Royal Commission offers a sweeping mission statement for Services Australia’s future. As the only recommendation that applies across the entire social security system, it sets a challenge for all delivery:
Services Australia design its policies and processes with a primary emphasis on the recipients it is meant to serve. That should entail:
avoiding language and conduct which reinforces feelings of stigma and shame associated with the receipt of government support when it is needed
facilitating easy and efficient engagement with options of online, in person and telephone communication which is sensitive to the particular circumstances of the customer cohort […]
explaining processes in clear terms and plain language in communication to customers, and
acting with sensitivity to financial and other forms of stress experienced by the customer cohort and taking all practicable steps to avoid the possibility that interactions with the government might exacerbate those stresses or introduce new ones.
Truth be told, I’d have preferred Commissioner Holmes AC SC set out an exhaustive and exacting laundry list of reforms to implement this recommendation. I’d have loved a last block of hearings on “Reshaping Relationships of Power and Drivers of Exclusion in Australia’s Welfare System”. It is what it is. And things are so much better for that Royal Commission having been.
But a critical task facing us now is to spell out the fullest possible response to Recommendation 1. In one scenario Services Australia responds by abstractly indicating “Of course! In the future… we will always design… .” They need to look at the words and take up the challenge in them. The recommendation is a bit of Rorshach test for the entire institution and whether they feel the imperative to change. It’s not a blob shaped like ‘reminder for down the road’. It’s a vital, immediate challenge for them to take up. Only fourteen victims got to speak at the Royal Commission. This is their recommendation, it’s what their courage called for.
The PARP is part of that picture. It anchors Services Australia’s Annual Report and its commitments shape resource allocation in processing. The report gives us insights into who sits within the 80% “processed in time” cohort and those who sit outside. It tells us the extent, but not the reasons, for some forms of error. And disappears others. Let’s start with the gaps in the survey. That’s where important, often vulnerable people go entirely unmapped.
Not counting what counts
When I first got this thing, I went hunting for something very specific. Remember those victims at the Royal Commission who ended their evidence by saying: “I will never ever again deal with Centrelink…”? The PARP survey does not look for, does not count such people. In 2021, ANU researchers recommended Services Australia and the Department of Social Services take some small steps into the unexplored world of underclaiming and non-take up:
Recommendation 10: Include administrative error which denies a person receipt of a benefit they are entitled to as a payment inaccuracy.
Recommendation 11: Consider including cases in which a recipient does not claim a benefit to which they are entitled, but which they would like to receive, as a payment inaccuracy due to recipient error.
So like a child choosing to play with the box on Christmas Day, I charged straight for Appendix 7 of this report. It’s been two years. We had an entire Royal Commission on how people interact with Centrelink. How they struggle to understand and overcome barriers. The last I saw on these was “in progress”. They still are, but with the additional note…
Services Australia advised that ICT changes are necessary for full rollout of this measure. This may impact on implementation timelines.
We learn Services Australia will ‘assess the operational impact of ANU recommendations that were dependent on ICT changes and report back to the department by 31 October 2023.’ A frustrating delay for what are only baby steps towards probing a broader issue in our welfare system: non-take up.
Around the world, social security systems struggle to measure the degree of non-take up, but they are, increasingly, trying. The concept of stigma in the Robodebt Royal Commission report links directly to last year’s Report of the UN Special Rapporteur on Extreme Poverty and Human Rights. This report—The Non-take-up of Rights in the Context of Social Protection—framed issues of administrative burden and accessibility as key. It has generated a wave of renewed focus on the issue. France—which commissioned a nifty literature review last year—estimates an engrained rate of non-take up amongst families at 20%. The United Kingdom famously found 25% of the non-take up of its new Universal Credit was due to digital exclusion. Just last month the International Social Security Association—an organisation for social security institutions themselves: departments and agencies—did a deep dive on the entire issue. For me, any effective response to the Robodebt Royal Commission has to have this as a key point.
We need to get on the front foot here and get the whole picture. Not a technocratic one centred on defensive KPIs. Even the Audit Office has a discrete and narrow approach. They are not performing a policy evaluation. They focus only on existing ‘in system’ administrative work and how it is performed. But their report notes that Services Australia don’t make clear how limited the PARP is, the extent of its limitations. Footnote 74 points “by way of contrast,” to the disclaimer the UK Department of Work and Pensions prominently shares on its equivalent survey:
When interpreting the statistics, please bear in mind that we only sample cases that are in receipt of benefit. The figures do therefore not include, for example, people who are entitled to benefit but don’t apply, those whose applications are incorrectly rejected, or benefit advances.
Centrelink English: The meaning of ‘error’
Having made clear the inherently narrow framing, let’s look at the meaning of ‘error’. Services Australia’s view is that ‘customer’ error is not something it should be judged upon. It is something they record to be sure. But they are not exploring its causes in sufficient depth. The PARP does not differentiate between recipient fraud, opportunistic non-compliance and inadvertent non-compliance. The Audit Office at page 82 asks how Services Australia can address recipient error if it does not study what it is driving it fully.
Amongst the most fascinating reflections on ‘customer’ error comes at page 76. The Audit Office raises the prospect that one driver could be the rise of the self-service online environment. Services Australia, in narrowly focusing on minimising staff mistakes, and not how recipients are navigating the system online, may be adopting a counterproductive approach:
Services Australia identified in its 2020–21 Annual Performance Statements that payment correctness was expected to improve as it ‘continues to expand on self-service options and automated actions, which reduce the opportunity for staff error’. In 2021–22 Services Australia reported that 69.7 per cent of welfare payment management tasks were self-managed digitally by recipients. Services Australia has not considered that its focus on improving administrative correctness may be having a negative impact on payment accuracy. Figure 3.3 shows there has been a strong negative correlation between Services Australia’s payment correctness and accuracy results since 2013–14.
What looks like an interesting report from October 2022 noted that the current approach lacked fairness and relied on recipient self-disclosure. It described a ‘future state’ vision involving large-scale use of third-party data, prefilling of recipient forms, near real-time adjustment of payments, and minimal ‘touch points’ for recipients. It appears a bit too early to judge what effects Single Touch Payroll pre-filling is having though.
A solid chunk (well over a billion dollars annually) of the overpayment stat in this report is due to ‘generic failure’. Generic failure is pure process fail—the person ‘did not engage’ completely in the review process. But the UK call this ‘cannot review.’ As the ANU researchers explained in 2021:
While in some cases this is a correct classification, in other cases generic failure will not in reality represent an inaccurate payment.
Where a person who is increasing their earnings or exiting social security receives a PARP review notification, they are unlikely to respond. Nearly half of ‘generic failure’ in 2021 was not responding to the letter, another quarter was not submitting all the documentation about yourself. Does this reflect a person on their way out the system? A vulnerable person unable to cope with the idea their benefits are up for review? A person evading the review? All we get on the ANU researcher’s recommendation to nail that down is “generic failure is scheduled to be discussed at BMC meeting on 23 March 2023”.
Those outside the tech, outside the KPIs
Automation and online servicing pop up throughout—it is the prevailing reality of the system. Footnote 111 confirms increasing Straight Through Processing for non-complex cases in Youth Allowance (Student) and Dad and Partner Pay. The timeliness sections of the report point to the political economy of automation— its winners and losers. The rising tide of technology—eg, more processing that effectively takes zero days—has not shifted key performance measures all that much.
What can also emerge is a system of skimming and parking. What can be done reflexively is fast; but what takes a human sits at… the opposite end of the spectrum. The report flags a negative dynamic for complex applications. If a claim dates and can no longer count towards the ‘magical’ 80% that are in time, there’s an incentive to let it sit on the system. Those with complex situations, more likely to be vulnerable or disadvantaged, sit in these less prominent off ramps. That’s the dangerous silence to Key Performance Measures centred on averages and the experience of ‘most cases.’
Another good example of the invisibility automation and online form submission creates is a December 2018 internal brief. This noted that time taken to request additional information is falling because online form submission effectively prevents incomplete applications. Nice for the institution; not so good for the person trying to get into the system. Services Australia has also shifted from measuring from the date the person first enquires about the payment to the point at which a claim is in a complete state for processing. People’s struggles to reach that start line? They sit in the void of the uncounted.
There are other categories of error that Services Australia ‘downplays’—at least in the ANAO’s view. Table 3.10 (see below) of the report is very striking. It reports on administrative errors with no dollar impact. Over 51% of files reviewed during the survey had this type of error, up from 15.5% in 2004–2005. These errors are less dramatic but include things that can become significant later. Struggles to properly update circumstances and personal details for instance. These secondary ‘non claims’ processes are currently not tied to any expectations of timeliness and accuracy. Recommendation 13 asks the Department of Social Services to consider what measures are needed to capture all those placed by the wayside in the current survey and measures.
Historical Unlawfulness and Inaccuracy
A PARP review is anchored in what Services Australia defines as correct – as embodied by its operational blueprints. But as we have seen recently, that’s often wrong, at scale, for years. This inconvenient reality slips into the report’s margins. Details like those in Footnote 25 for instance:
The audit did not include analysis of the implications for welfare payment accuracy of findings outlined in the Commonwealth Ombudsman’s statement titled Lessons in lawfulness (issued on 2 August 2023). The Ombudsman’s findings related to the lawfulness of Services Australia’s income apportionment calculations for welfare payments between 2003 and December 2020. While the audit involved testing of PARP reviews’ conformance with operational policies and procedures, it did not include testing of individual payment calculations for welfare recipients subject to PARP reviews.
So there’s the rub folks! The audit tests conformance of Service Australia with how they themselves see the law. Of course, social security lawyers regularly show up these views as shortcutting law and accurate fact finding for administrative convenience.
The PARP doesn’t capture inaccuracies linked to incorrect internal understandings of the law or to incorrect internal operational blueprint. Those are only errors if a person challenges it and receives a review down the track. Or if Services Australia commits to full own initiative reviews under section 126. The survey operates as a narrow technocratic gauge on the tides of the system. Remember that when the overpayment statistics in this report in the hands of senators in estimates.
Like all true public policy dorks, I enjoy the Audit Office’s work immensely, but it is inherently bounded. The above footnote is exactly why the Audit Office failed to spot the ‘interim manual’ version of robodebt in its 2017 review of DHS compliance. That cohort turned out to be one of the largest financial risks at large in the Commonwealth’s ledger. The cohort was 91% averaged. They received the highest class action interest. The Audit Office’s only commentary was to note that staffing levels on that year of robodebt were above what was predicted!
Like others, ANAO leant a bit on the notion that robodebt began with a computer launch in 2016. It distanced its work on the 2015–2016 interim version when the media controversy hit in 2017. This report states that it does not get into any implications or fallout from the “Online Compliance Intervention” in its analysis. But even apart from refunding debts, there was a mountain of secondary remediation work to get files back to accurate state. I don’t believe it has been performed. But it appears that’s too ‘micro’ for the Audit Office’s analysis of a rolling snapshot survey.
All this auditing takes place within the confines of existing policy settings. And don’t forget that responsibility for overpayments gets a distinctive response in Australia’s system. If an error was 98% Services Australia and 2% you, it’s customer error. A debt will be enforced unless it was ‘solely attributable’ to the agency. To avoid contributing to an error, recipients will have to hunt down and fix any errors in their circumstances. As AAT Senior Member Puplick memorably summed up:
The Parliament has legislated to give a fig leaf of coverage to shield the Department from the embarrassment of exposure to penalty for its administrative errors…
Dry, Boring and… Desperately Important
I hope this post reminds people of the value of reading dry reports closely. Underneath the technicality there’s enormous bureaucratic power and conflict moving through this one. I’m fond of quoting David Foster Wallace’s The Pale King to my students:
Abstruse dullness is actually a much more effective shield than is secrecy. For the great disadvantage of secrecy is that it’s interesting…
This report embodies the mindsets, blueprints and methods that see real lives fall through the cracks. My old social media account used to live for stuff like this! So I want to thank Chris for lending me his platform. After robodebt we all need to stay together to improve engagement with social security policy in this country.