Discussion about this post

User's avatar
Patrick's avatar

Kari, your move from the individual reviewer to the larger human system connects closely with a discussion in my doctoral supervision: judgement depends on interacting governance loops, including how information, assumptions and authority move between them.

Three connections may help develop your argument.

Kalpič and Bernus distinguish awareness of knowledge, its externalization, and its formalization (2006, pp.45–46). These are different properties: what an organization has encoded in its rules does not exhaust what people know. Your upstream pathway therefore needs to accommodate relevant knowledge that cannot yet be expressed in the system’s existing categories.

Ashby’s *ultrastability* offers a complementary connection: adaptation requires the capacity to change the organization of behaviour when existing arrangements cannot keep essential variables within viable limits (1960, pp.80–99). Applied cautiously to governance, this raises a further question: whose interests do those protected variables represent?

My own research on Judgement Displacement Risk (JDR) examines a narrower failure mechanism: optimization around proxies can gradually displace the judgement needed to question them, whilst reported performance improves. (!) Both AI and reviewer may continue applying the rules correctly—they might just not know the rules are incomplete because the proxies cannot report what they cannot represent.

Your statement that consequential information needs somewhere to go is therefore pivotal.

I would add: can that information challenge the categories through which its relevance is assessed, and reach someone able to alter the decision before its consequences become irrecoverable? That would help distinguish an available feedback pathway from an operative capacity to learn.

________

References

- Kalpič, B., & Bernus, P. (2006). Business process modeling through the knowledge management perspective. "Journal of Knowledge Management", 10(3), 40–56. https://doi.org/10.1108/13673270610670849

- Ashby, W. R. (1960). "Design for a brain: The origin of adaptive behaviour" (2nd ed.). Chapman & Hall. https://doi.org/10.1007/978-94-015-1320-3

Kevin Mears's avatar

“What happens when the AI and the human review both correctly apply the same problematic rule?”

Your loan example has one more turn in it: the organization may never find out. A declined applicant never gets the loan, so the lender’s own records never show whether they would have repaid. Monitoring can catch a bad approval, because the loan goes bad. A bad decline leaves no trace in the lender’s books. Credit modelers have a family of techniques, called reject inference, for estimating those missing outcomes.

So for the people a rule wrongly screens out, the contest you describe may be the only evidence the lender ever receives. Someone who gives up and borrows elsewhere leaves nothing in the lender’s own records. Where a lender pulls credit bureau data on what rejected applicants did next, that trace exists and this is weaker. Where it doesn’t, the rule keeps looking sound. That’s your point that feedback can reinforce existing assumptions, built into the data itself.

The loop can only question the system with the evidence the system lets exist.

9 more comments...

Ready for more?