Kari, your move from the individual reviewer to the larger human system connects closely with a discussion in my doctoral supervision: judgement depends on interacting governance loops, including how information, assumptions and authority move between them.
Three connections may help develop your argument.
Kalpič and Bernus distinguish awareness of knowledge, its externalization, and its formalization (2006, pp.45–46). These are different properties: what an organization has encoded in its rules does not exhaust what people know. Your upstream pathway therefore needs to accommodate relevant knowledge that cannot yet be expressed in the system’s existing categories.
Ashby’s *ultrastability* offers a complementary connection: adaptation requires the capacity to change the organization of behaviour when existing arrangements cannot keep essential variables within viable limits (1960, pp.80–99). Applied cautiously to governance, this raises a further question: whose interests do those protected variables represent?
My own research on Judgement Displacement Risk (JDR) examines a narrower failure mechanism: optimization around proxies can gradually displace the judgement needed to question them, whilst reported performance improves. (!) Both AI and reviewer may continue applying the rules correctly—they might just not know the rules are incomplete because the proxies cannot report what they cannot represent.
Your statement that consequential information needs somewhere to go is therefore pivotal.
I would add: can that information challenge the categories through which its relevance is assessed, and reach someone able to alter the decision before its consequences become irrecoverable? That would help distinguish an available feedback pathway from an operative capacity to learn.
________
References
- Kalpič, B., & Bernus, P. (2006). Business process modeling through the knowledge management perspective. "Journal of Knowledge Management", 10(3), 40–56. https://doi.org/10.1108/13673270610670849
As noted on your restack - I’m hoping you will take a look at v2 of this once I incorporate your helpful thoughts - thank you so much for the feedback!
“What happens when the AI and the human review both correctly apply the same problematic rule?”
Your loan example has one more turn in it: the organization may never find out. A declined applicant never gets the loan, so the lender’s own records never show whether they would have repaid. Monitoring can catch a bad approval, because the loan goes bad. A bad decline leaves no trace in the lender’s books. Credit modelers have a family of techniques, called reject inference, for estimating those missing outcomes.
So for the people a rule wrongly screens out, the contest you describe may be the only evidence the lender ever receives. Someone who gives up and borrows elsewhere leaves nothing in the lender’s own records. Where a lender pulls credit bureau data on what rejected applicants did next, that trace exists and this is weaker. Where it doesn’t, the rule keeps looking sound. That’s your point that feedback can reinforce existing assumptions, built into the data itself.
The loop can only question the system with the evidence the system lets exist.
Yes exactly the point - the system itself needs a mechanism to challenge itself both through monitoring in various ways, but through the challenge mechanism itself. But the applicant needs to be informed either way of what the algorithm has detected and why, and be able to challenge it accordingly and then hear the results of that challenge.
Oh yeah, I hear you. We build these beautiful, but somewhat brittle systems to handle scale, and then we lose some humanity in the building unless we build the humanity into it. I think we are just now understanding what it really means to build the humanity into into the system itself…
I kind of feel like we build the systems based on the best knowledge we had at the time, but then we continue to evolve as humans and when we look back at the systems we built that are so beautifully stable now, they begin to look barbaric.. dismantling our creations is incredibly hard - particularly to not throw the baby out with the bathwater.
Kari, your move from the individual reviewer to the larger human system connects closely with a discussion in my doctoral supervision: judgement depends on interacting governance loops, including how information, assumptions and authority move between them.
Three connections may help develop your argument.
Kalpič and Bernus distinguish awareness of knowledge, its externalization, and its formalization (2006, pp.45–46). These are different properties: what an organization has encoded in its rules does not exhaust what people know. Your upstream pathway therefore needs to accommodate relevant knowledge that cannot yet be expressed in the system’s existing categories.
Ashby’s *ultrastability* offers a complementary connection: adaptation requires the capacity to change the organization of behaviour when existing arrangements cannot keep essential variables within viable limits (1960, pp.80–99). Applied cautiously to governance, this raises a further question: whose interests do those protected variables represent?
My own research on Judgement Displacement Risk (JDR) examines a narrower failure mechanism: optimization around proxies can gradually displace the judgement needed to question them, whilst reported performance improves. (!) Both AI and reviewer may continue applying the rules correctly—they might just not know the rules are incomplete because the proxies cannot report what they cannot represent.
Your statement that consequential information needs somewhere to go is therefore pivotal.
I would add: can that information challenge the categories through which its relevance is assessed, and reach someone able to alter the decision before its consequences become irrecoverable? That would help distinguish an available feedback pathway from an operative capacity to learn.
________
References
- Kalpič, B., & Bernus, P. (2006). Business process modeling through the knowledge management perspective. "Journal of Knowledge Management", 10(3), 40–56. https://doi.org/10.1108/13673270610670849
- Ashby, W. R. (1960). "Design for a brain: The origin of adaptive behaviour" (2nd ed.). Chapman & Hall. https://doi.org/10.1007/978-94-015-1320-3
As noted on your restack - I’m hoping you will take a look at v2 of this once I incorporate your helpful thoughts - thank you so much for the feedback!
“What happens when the AI and the human review both correctly apply the same problematic rule?”
Your loan example has one more turn in it: the organization may never find out. A declined applicant never gets the loan, so the lender’s own records never show whether they would have repaid. Monitoring can catch a bad approval, because the loan goes bad. A bad decline leaves no trace in the lender’s books. Credit modelers have a family of techniques, called reject inference, for estimating those missing outcomes.
So for the people a rule wrongly screens out, the contest you describe may be the only evidence the lender ever receives. Someone who gives up and borrows elsewhere leaves nothing in the lender’s own records. Where a lender pulls credit bureau data on what rejected applicants did next, that trace exists and this is weaker. Where it doesn’t, the rule keeps looking sound. That’s your point that feedback can reinforce existing assumptions, built into the data itself.
The loop can only question the system with the evidence the system lets exist.
Yes exactly the point - the system itself needs a mechanism to challenge itself both through monitoring in various ways, but through the challenge mechanism itself. But the applicant needs to be informed either way of what the algorithm has detected and why, and be able to challenge it accordingly and then hear the results of that challenge.
https://youtu.be/JZIVmKOdrBk?is=wc3uoN2ayBzKFUu_
ROFL - oh no.
Oh? What was traumatizing?
Oh yeah, I hear you. We build these beautiful, but somewhat brittle systems to handle scale, and then we lose some humanity in the building unless we build the humanity into it. I think we are just now understanding what it really means to build the humanity into into the system itself…
I kind of feel like we build the systems based on the best knowledge we had at the time, but then we continue to evolve as humans and when we look back at the systems we built that are so beautifully stable now, they begin to look barbaric.. dismantling our creations is incredibly hard - particularly to not throw the baby out with the bathwater.