I recently completed a certification on Responsible AI, and part of the course content was of course on keeping the Human in the Loop (HITL).
And I absolutely understand why this makes sense because if AI is involved in a consequential decision we must make sure a human remains meaningfully involved.
But, I kept rolling it through my head, because it just didn’t feel quite right. Because what happens when the AI is working correctly? Like, what happens when the AI and the human review both correctly apply the same problematic rule?
For example, let’s imagine an organization has criteria for determining whether or not someone is a high-risk loan applicant. The AI applies the criteria and recommends declining the application. According to HITL, the human carefully reviews the case, confirms that the criteria have been correctly applied and then approves the decision.
Neither the AI nor the human has made an error - both took the criteria and applied it, but as it turns out, the candidate was a good candidate for other reasons. And that gets me thinking..
The ‘human’ in Human-in-the-Loop isn’t really just one human - they are sitting inside a much larger human system that is made up of policies, historical pratices, incentives, targets, norms, data, authority structures and assumptions about what matters in an applicant and what success looks like.
But if the AI governance only creates feedback capable of correcting the AI, we may miss the more interesting possibility: AI-mediated decisions can also give us information about the human system that produced them…
This means that sometimes the model itself needs to change, sometimes a particular case warrants an exception, sometimes the governing rule needs to be reconsidered, and sometimes the question is even furth upstream - should this activity have been standardized and automated in the first place???
And that got me back to a thought that I’ve been pondering for some time not, considering the balance between a system that repeats and a system that is capable of learning.
A learning system needs somewhere for consequential information to go. An affected person (the loan applicant in our narrative) needs a meaningful way to challenge a decision and to understand the results of that challenge. Patterns across decisions need to be detectable, and evidence needs to be capable of travelling far enough upstream to reach someone with the authority to question not only whether the AI followed the rules, but whether the rules themselves still make sense.
There is an established organizational-learning theory behind part of this, particularly Chris Argyris and Donald Schön's work on the difference between single-loop and double-loop learning. And it makes me wonder what happens when we apply that type of thinking into the Human-in-the-Loop governance.
So, I pulled those thoughts into a short working paper:
The Human System in the Loop:
From Human Checkpoints to Organizational Learning in Responsible AI
and created a visual model and short explainer alongside it.
Read the working paper and explore the framework →
Final thought:
Keeping a human in the loop isn’t enough if the loop itself can’t question the system that created it.
Prefer to watch? Here’s a short explainer video of the framework: