I recently completed a certification on Responsible AI, and part of the course content was of course on keeping the Human in the Loop (HITL). I absolutely understand why the emphasis on this makes sense because if AI is involved in a consequential decision we must make sure a human remains meaningfully involved.
But I kept rolling it through my head, because it just didn’t quite feel right. Because what happens when the AI is working correctly but a bad outcome still happens? Like, what happens when the AI and the human review both correctly apply the same problematic rule?
For example, let’s imagine an organization has developed criteria for determining whether or not someone is a high-risk loan applicant. The AI applies the criteria and recommends declining the application. According to HITL, the human carefully reviews the case, confirms that the criteria have been correctly applied and then approves the decision. Neither the AI nor the human has made an error - both took the criteria and applied it correctly but as it turns out, the candidate was actually a good candidate for other reasons, but didn’t ‘look’ like it on paper.
And that gets me thinking.. The ‘human’ in Human-in-the-Loop isn’t really just a singular human clicking ‘approve’ or ‘deny’ all day. They are actually sitting inside a much larger human system that is made up of policies, historical practices, incentives, targets, norms, data, authority structures and assumptions about what matters in an applicant and what success looks like. But if the AI governance only creates feedback capable of correcting the AI, we may miss the more interesting possibility:
AI-mediated decisions can also give us information about the human system that produced them.
This means that sometimes the model itself needs to change, a particular case warrants an exception, the governing rule needs to be reconsidered, and sometimes the question is even further upstream - should this activity have been standardized and automated in the first place?
And that got me back to a thought that I’ve been considering for some time - what is the right the balance between a system that repeats and scales and a system that is capable of learning?
A learning system needs somewhere for consequential information to go. An affected person (the loan applicant in our narrative above) needs a meaningful way to challenge a decision and to understand the results of that challenge. Patterns across decisions need to be detectable, and evidence needs to be capable of travelling far enough upstream to reach someone with the authority to question not only whether the AI followed the rules, but whether the rules themselves still make sense.
There is established organizational-learning theory behind part of this concept, particularly Chris Argyris and Donald Schön's work on the difference between single-loop and double-loop learning. It made me wonder what happens when we apply that type of thinking into the Human-in-the-Loop governance. So, I pulled those thoughts into a short working paper:
The Human System in the Loop:
From Human Checkpoints to Organizational Learning in Responsible AI
and created a visual model and short explainer alongside it.
Read the working paper and explore the framework →
Final thought:
Keeping a human in the loop isn’t enough if the loop itself can’t question the system that created it.
Prefer to watch? Here’s a short explainer video of the framework:



Kari, your move from the individual reviewer to the larger human system connects closely with a discussion in my doctoral supervision: judgement depends on interacting governance loops, including how information, assumptions and authority move between them.
Three connections may help develop your argument.
Kalpič and Bernus distinguish awareness of knowledge, its externalization, and its formalization (2006, pp.45–46). These are different properties: what an organization has encoded in its rules does not exhaust what people know. Your upstream pathway therefore needs to accommodate relevant knowledge that cannot yet be expressed in the system’s existing categories.
Ashby’s *ultrastability* offers a complementary connection: adaptation requires the capacity to change the organization of behaviour when existing arrangements cannot keep essential variables within viable limits (1960, pp.80–99). Applied cautiously to governance, this raises a further question: whose interests do those protected variables represent?
My own research on Judgement Displacement Risk (JDR) examines a narrower failure mechanism: optimization around proxies can gradually displace the judgement needed to question them, whilst reported performance improves. (!) Both AI and reviewer may continue applying the rules correctly—they might just not know the rules are incomplete because the proxies cannot report what they cannot represent.
Your statement that consequential information needs somewhere to go is therefore pivotal.
I would add: can that information challenge the categories through which its relevance is assessed, and reach someone able to alter the decision before its consequences become irrecoverable? That would help distinguish an available feedback pathway from an operative capacity to learn.
________
References
- Kalpič, B., & Bernus, P. (2006). Business process modeling through the knowledge management perspective. "Journal of Knowledge Management", 10(3), 40–56. https://doi.org/10.1108/13673270610670849
- Ashby, W. R. (1960). "Design for a brain: The origin of adaptive behaviour" (2nd ed.). Chapman & Hall. https://doi.org/10.1007/978-94-015-1320-3
“What happens when the AI and the human review both correctly apply the same problematic rule?”
Your loan example has one more turn in it: the organization may never find out. A declined applicant never gets the loan, so the lender’s own records never show whether they would have repaid. Monitoring can catch a bad approval, because the loan goes bad. A bad decline leaves no trace in the lender’s books. Credit modelers have a family of techniques, called reject inference, for estimating those missing outcomes.
So for the people a rule wrongly screens out, the contest you describe may be the only evidence the lender ever receives. Someone who gives up and borrows elsewhere leaves nothing in the lender’s own records. Where a lender pulls credit bureau data on what rejected applicants did next, that trace exists and this is weaker. Where it doesn’t, the rule keeps looking sound. That’s your point that feedback can reinforce existing assumptions, built into the data itself.
The loop can only question the system with the evidence the system lets exist.