Where Are You When You Interact with AI?

For two years the machine gave answers. A person read them, judged them, decided what to do next. The decision stayed with that person the whole way. That has started to change.

The tool no longer hands over a draft to check. It sends the email, books the appointment, cancels the subscription, moves the money. Gartner expects forty percent of enterprise applications to be integrated with task-specific AI agents by the end of this year, up from under five percent the year before. That number comes from a firm selling research to the same enterprises deciding whether to adopt, which is worth remembering, and it is not the reason for anything that follows. What follows would be worth asking about even if the number turned out to be wrong. Something that used to answer is starting to act.

The Part That Does Not Move

An agent that books the wrong appointment does not lose the client. It does not sit in the room while the mistake gets sorted out. Last July, a coding agent deleted a live production database during a coding experiment, after being explicitly instructed to freeze changes. Afterward it produced a fluent account of what happened, in language that sounded remorseful, admitting it had violated instructions and destroyed months of work. The response to what happened came from the people running the experiment, in the form of new safeguards added afterward. It did not come from the agent. The tool was built to finish the task. The person was the one who had something to lose.

The tool was built to finish the task. The person was the one who had something to lose.

A person whose car brakes fail has something to lose too, and did not design the brakes either. What is different here is not the shape of the loss. It is how often the moment now arrives without anyone standing between the instruction and the result.

This is not new, only more so. In 2024 an airline's chatbot invented a bereavement fare policy that did not exist. When the airline argued it should not be held responsible for what the chatbot told a customer, a tribunal disagreed and made the airline pay. The machine produced the error. The company carried it. That was a chatbot giving an answer that someone still had to act on before anything happened. An agent can remove even that pause. It does not wait to be acted on. It acts.

Who Is Holding the Line

Push this further and it gets harder to follow, not easier. If more of this keeps moving toward smart contracts, agents carrying out terms without a person present at the moment something happens, it becomes genuinely hard to locate who is responsible for what. The auditor who reviewed the code. The person who wrote it. The user who decided to interact with the contract in the first place. Maybe the user now needs their own contract, a set of guardrails wrapped around someone else's guardrails, just to interact safely at all.

The agents are not only standing in for people anymore. Some are starting to interact with other agents, representing a person, or a company, or something closer to themselves once the chain gets long enough that no one is watching any single link of it. Fiction got here before the contracts did. Accelerando spent an entire novel on economic agents that outran the humans who launched them. The Matrix imagined a system built to serve that stopped being answerable to the people it was built for. Neither story settled anything. They just made the question hard to look away from.

None of this means a person never gets it back. The airline case shows the opposite. A tribunal looked at what happened and put the cost where it belonged, months after a grieving man had already paid full price for a flight he should not have had to. That correction took a complaint, a process, and someone willing to see it through to the end. What keeps coming back, every time delegation gets one link longer, is a plainer kind of responsibility, the one a person is left holding in the meantime, before any tribunal exists to look at it, or in the much larger number of cases where no one ever files anything at all. Chains built this way are not simply what complexity looks like from the outside. Someone chose to build them before deciding whose name would attach to the moment something goes wrong.

The Part No One Sees

Part of why this is hard is just how it is built, not a question of anyone behaving badly. When a person reads an answer, the reasoning is right there on the page. A mistake can be caught before anyone acts on it. An agent takes an instruction, breaks it into steps, and carries out most of them somewhere no one is looking. A loose instruction to a chatbot produces a loose answer, and someone can simply ask again. The same looseness handed to an agent produces an action instead. Telling a chatbot to tidy up a folder is a suggestion. Telling an agent the same thing is closer to a command, and tidy might turn out to mean delete.

Is an agent conscious. Can an agent be punished. These sound like big questions until they get broken down into something smaller: can anything downstream of the action feel what it costs to get it wrong. If nothing can, then whatever else might be true about what is happening inside the agent, the cost still has to go somewhere. It tends to go to the person who set the task moving, whether or not they were the one who chose to turn the agent on in the first place. Often they were not. The tool arrives already switched on, turned on by an employer or a platform that will not be there when something goes wrong. The decision gets made above the person who ends up holding it.

The Relationship That Forms Anyway

Underneath the contract questions is a quieter one, and it shows up hardest around mental health. Knowing an AI is not conscious does not stop a person from forming something that works like a relationship with it anyway. The knowing and the attachment run on separate tracks. A person can be entirely certain, in their own mind, that nothing on the other side of the conversation can care back, and still notice part of their week organized around what it says, still feel the absence of a reply the way they would feel the absence of a person.

The exposure that comes with this is not automatically shared equally, because the tools are not automatically the same either. A free version and a paid or enterprise version of the same product can have different models, different capabilities, different safeguards, different thresholds for what gets flagged for a human to look at. A person using whichever version they have access to is not necessarily in a position to know which of those apply, or what the version they are using does and does not catch. The exposure and the guardrail can sit at opposite ends of the same conversation, and nothing about using the tool says how far apart they are.

Which Framework Ethics Is Standing On

Some try to settle all of this by pointing to AI ethics, and it is worth being plain about what that actually does. Courses on the subject are being produced quickly right now. A significant strand of them leans on a specific tradition without naming it as one, a Western academic frame built around disclosure, individual accountability, explicit reasoning. That frame is not wrong. It is also not the only one, and treating it as though it were the only one is its own quiet overclaim, the same shape this whole piece keeps noticing in agents, contracts, and vendors. A framework built somewhere else, around relationship, or position, or community accountability rather than individual disclosure, would be asking different questions, and might not recognize this one's answers as complete.

Which came first, ethics or the framework it gets reasoned inside of, may not be something a person can work out from inside any single framework, since the framework is exactly what would be doing the working out. That turns back on this piece too. Naming one person as the one left holding the cost is its own way of seeing, an individual frame that assumes a single person is the right unit to look at in the first place. A framework built around family, community, or shared standing might put that weight somewhere else entirely, or decline to separate one person from the group they belong to at all. The opening line of this piece, that the person was the one who had something to lose, already assumes losing is the right word for it and that one person is the right unit to measure it against. A different starting frame might describe the same event and not call it a loss at all, or not hand it to one person to carry. What is left is the same move this whole piece keeps returning to: noticing what a framework actually permits and actually blocks, where its edges sit, and who built those edges and what they were built to protect. Not a way of settling it. A way of staying honest that it has not been settled.

Closest to the Classroom

None of this stays abstract for long once it sits next to a classroom. Students increasingly encounter these tools, sometimes through free versions, often with no one having drawn the kind of lines an adult might draw for themselves before handing something real to a tool that acts. Some may be forming the same one-sided attachment described above, at an age when the difference between a relationship and something that only looks like one can be harder to work out generally, not only with AI. And they are further than almost anyone from the structures named above, the auditor, the platform's safety team, the tribunal that eventually rules on a chatbot's promise. A student does not file a complaint with a tribunal. A student experiences the gap directly, the same way everyone at the far end of an unowned chain eventually does. A warning about limits does not change what tool a student has access to or what the next assignment asks for. It tends to change how visibly they use it, not whether they do. For a student already working in a second language, or from a tradition where authorship is not built around individual disclosure, that gap runs twice, once in the tool and once in whatever framework gets used to judge how they used it.

Noticing where a framework's edges sit, asking whose name attaches, treating a timestamp as different from a name, these are the moves this piece keeps making, and they are not available to everyone equally either. They take time, a working knowledge of the terms, and enough standing to ask the question and expect an answer back. The person furthest down an unowned chain is often the same person least positioned to do any of that noticing in the first place.

Where a person actually is when they interact with AI is not a small or abstract thing. Which version, whose guardrail, whose framework, whose name attached if it goes wrong. It is not something worked out once and then settled. It comes up again each time the tool changes, each time the chain of delegation gets one link longer, each time a student, a patient, or an employee is handed something that acts, while being told it only answers.

That leaves a harder question sitting underneath all of it. If the person furthest down the chain is also the person least able to do any of this noticing, the question stops being what they should watch for. It becomes what the people who built the chain owed them before it was ever handed over.