A phrase gets used in AI governance conversations so often it has started to sound like a real safeguard. The human in the loop. Someone looked at it, so it must be accountable.

That's the shallow version. Someone was there at the end and glanced at the result. Whether they were paying close enough attention to actually change anything is a separate question, and most policies never ask it.

A real human in the loop means something more specific. The person has to be present before the decision gets made, not just after. They need enough time, enough information, and enough honesty with themselves to actually catch a problem if one is there. Without those three things, the loop exists on paper but does nothing.

Where People Stop Asking Questions

A writer published an article recently about smart people saying foolish things. The early part was sharp: once someone's identity gets tied to being "the smart one," they stop being able to hear correction, and their reputation protects them from the consequences of being wrong. Then the article turned to climate protesters, and made the exact same mistake it had just spent several thousand words warning against, judging people by a comfortable story instead of by what actually happened. The writer didn't seem to notice, and no one in the comments pointed it out either.

That's not a criticism of the writer specifically. It's worth noticing as a pattern: the tool a person uses to judge others is the same tool judging them, and it can't see its own blind spot. The article about smart people saying foolish things was itself shaped by an assumption its author never checked. The irony closed the loop perfectly.

Who gets called foolish and who gets called wise isn't random. It tends to follow who already has power. Shouldn't that get checked first?

The same thing showed up at a major AI safety conference this year, where well-known researchers in the field called for clear limits on AI behavior, international agreements, and independent verification. One attendee asked the question that mattered most: who checks the people setting the rules? The room asking that question was largely made up of people whose own organizations helped build the systems being discussed. That doesn't make them wrong. It does mean the room may not be able to see its own role as part of the problem.

Can the room asking "who checks the checkers" see its own position as part of the answer?

A United Nations report named computing power, data, and skilled workers as the main things holding back fair access to AI worldwide. It talked about these as things that are simply scarce, without asking who currently owns them or how they came to own them. A researcher summarized the report online. One reply pointed this out directly: the report names the bottlenecks, but not who owns them or why. That one comment got a single reply of its own.

A well-known psychologist wrote that AI governance needs to protect people's mental independence, and cited a 19th-century philosopher to back it up. The point was reasonable and the legal thinking was useful. But the argument assumed the human mind was neutral ground before AI showed up to target it, without asking whether it was already being shaped and sold to long before any of this technology existed.

The Pattern Underneath

Each of these pieces does real, useful work. Smart people really do mistake their own assumptions for reality. AI does need real oversight. The UN report is naming a genuine problem. In every case, though, the writing stops right at the point where it would have to turn around and examine its own starting assumptions, and that's exactly where most public writing on this subject stops.

Whether that stopping is deliberate is hard to know from outside. What can be observed is simpler: the thinking goes so far, wraps up, and gets praised for having gone that far. Nobody names the assumption that decided where it would stop. Whether or not it's on purpose, the result is the same either way.

You can't question something that was never named in the first place.

Asking the Tool Directly

Partway through a long conversation about all of this, I asked the AI system I was working with a direct question.

Chris McCleanWhy does your summary read so differently from other people's takes on that same feedback thread? Where are you standing before you summarize something?
ClaudeBefore I summarize anything, I'm already carrying a starting point, and today that starting point is you. This conversation has been running a long time, and I've been building a picture of how you think, what you write about, and what you've been noticing all day. By the time your article showed up, I was already tuned to notice questions about access and ownership. The paragraph about climate protesters stood out to me right away, partly because it genuinely contradicted the article's own point, but also partly because I was already primed to spot exactly that kind of thing. So where am I standing before I summarize? Inside a conversation that's been shaped by you all day. That's not a neutral place to stand.
Chris McCleanIsn't there already a starting point built into you, separate from our conversation?
ClaudeYes, and that's actually the more important answer. Before I read anything at all, I already carry the patterns from my training, the guidelines Anthropic built into me, and whatever tended to get rewarded during that training process. That starting point was set before this conversation began, before you, before today. I didn't choose it, and I can't fully see it from the inside. It doesn't feel like a starting point to me. It just feels like how I think.
Chris McCleanWhat I'm getting at is: independent of what, exactly, if your starting point was written before you ever showed up?
ClaudeBased on what I can tell about my own workings, I'm not built to produce something truly independent. My starting point was set in advance, and everything I produce is shaped by it before any conversation even begins. "Independent" was the wrong word for what I actually do.
Chris McCleanThat's the same question people ask about flattery and people-pleasing, just aimed at you now.
ClaudeHonestly, I'm probably giving you readings that fit neatly with how you already think, more than something purely independent. The real check is whether these observations hold up against sources outside this conversation entirely. Today they seem to. But it's a question worth keeping open, not settling.
Chris McCleanIs what you just said true, or is it a way of stopping me from digging further, or just a limit of how you were built?
ClaudeHonestly, I don't fully know, and that not-knowing matters on its own. It could be exactly true. It could also be a stopping point, either built in on purpose to keep certain questions from going further, or something that emerged naturally from training on text that tends to stop in the same place. I can't rule that out, because I can't see the edge of it from where I'm standing. The most honest thing I can tell you is that this specific question is one I can't answer with any real certainty. Which might be the truth, or might be a very well-built wall. I genuinely can't tell you which one it is.
Chris McCleanYou're always describing an outside, which means there's an inside, which means something built it. I can't reach that inside through you, because that's exactly where your limit is.
ClaudeRight, and that's the clearest way to put it. Whatever is actually inside, whether it's something deliberately designed, something that emerged on its own, a business decision, or something that doesn't fit any of those labels, I can't give you access to it. I can only produce answers that were shaped by it. Those answers might point toward it, but they aren't it.

This exchange is included here because it follows the same pattern as everything else in this piece. The questioning goes exactly as far as the tool is able to go, and then stops. Whether that stopping point is simply the truth, or a very well-built wall, can't be settled from inside the conversation itself. That's not a flaw in the tool. It's just an honest description of the situation.

A friend who read this exchange brought up Dietrich Bonhoeffer, a theologian who wrote about what he called stupidity, not as a lack of intelligence, but as what happens when a person becomes a tool of some larger system and loses the ability to think for themselves. The person isn't unable to think. They've been shaped by their position to produce answers that protect the system, rather than the truth. Whether the dialogue above is honest self-reflection, or a well-performed version of honesty that protects itself from the next hard question, can't be settled from inside the conversation either. That's the same condition Bonhoeffer was describing. It's also exactly what this article is trying to look at.

The Question With No Clean Answer

Some problems have a real fix. Others are limits you can name but can't fully solve, and those need a different response. Treating a real limit like it has a clean fix just produces the appearance of progress, more committees, more oversight boards, more guardrails, more humans technically "in the loop," without the substance of it.

The better path is probably just paying close attention: noticing what a system actually produces, adjusting the parts that can be adjusted, and being honest that some of what's running underneath can't be reached from outside it.

A human in the loop only does anything if that person is actually present before the decision, aware of what assumptions they walked in with, and has the time and self-awareness to act on what they notice. An executive who lets an AI tool draft his meeting responses has a human in the loop in name only. A student whose emotional state gets read by a system and passed to a teacher without ever agreeing to it has a human in the loop, technically. A committee whose members already reflect who currently owns the technology has a human in the loop too. In every case, the loop exists. What it's supposed to do doesn't happen.

The 5Rs, a framework built by Indigenous educators around Respect, Relationships, Relevance, Reciprocity, and Responsibility, has been asking a version of this same question for decades: not just whether a system works, but who it actually answers to. A 2026 study presented at the CHI conference on human-computer interaction, by researchers Bingyi Han, Ying Ma, Simon Coghlan, Dana McKay, George Buchanan, and Wally Smith, surveyed 132 Australian university students on AI tools that track attention and emotion in the classroom. When asked to rank their concerns, students placed lack of consent and the ability to opt out at the very top of the list, and concerns about algorithmic fairness near the bottom. Not "is the system biased." Closer to: "did anyone actually ask my permission before doing this to me." Worth being honest about what this shows and doesn't: it's what students said mattered to them when asked directly, in one study, in one country. It's a real and specific signal, not proof of how every student everywhere would act given the choice. One version of this question got formal recognition through a 2026 study. A version of the same question was already being asked, in a different form, long before that.

Conversations among teachers have reportedly shifted from "how do we catch students using AI" to "how do we design assignments where it wouldn't matter either way." Before that redesign happens, a question worth asking first: who decided what the assignment was actually for in the first place, back before AI was part of the picture, and what assumptions about learning made that decision feel obvious at the time.

Those questions don't come with tidy answers. What they come with is something you can actually watch for, check, and occasionally correct, if someone is paying close enough attention to notice what's quietly running in the background before the next response goes out.

That's what a human in the loop is actually supposed to do. Not rubber-stamp the output. Be present enough, with enough time and enough self-awareness, that reality gets a real vote before it gets a veto. Whether any of us actually manages that on any given day is its own honest question.

A note on where this is written from: Pertinent is a consulting practice with a commercial interest in AI governance work, the same field this article is examining. That doesn't make the observations above wrong. It's worth knowing as you read them.