Governance
5 min reading · 866 words
The Error Budget at an Eldercare Service
A field argument about risk and harm where systems meet an older person and family navigating care without making a simulation stand in for a relationship.
At an Eldercare Service, a system may arrive with the promise of speed, clarity, or availability. The person on the other side is an older person and family navigating care without making a simulation stand in for a relationship. What reaches them may be a reminder, care summary, or support suggestion, but it is never only that. It is also a message about whether the institution sees them as a case to be processed or a person entitled to an answer. The stakes are dignity, safety, continuity, and real human support.
That is why “The Error Budget” belongs here. Every system has error; the ethical question is who is expected to absorb it and whether the distribution is defensible. The most compelling version of the argument does not demand that every useful tool be removed. It asks what the tool is permitted to decide, what the institution remains obliged to do, and whether a person can tell the difference between a product’s performance and a responsible public or commercial relationship.
The relationship beneath the task
A technical description can make the setting sound smaller than it is. Someone may say that the system merely assists with a workflow, drafts a response, or ranks a queue. But workflows decide who waits, who is asked for more proof, whose time is treated as cheap, and whose story becomes legible to an institution. When the output travels through the care provider and its technology partners, the product inherits the setting’s existing asymmetries. It should not be allowed to deepen them while calling the result neutral.
There is a legitimate reason to use assistance in difficult settings: people are overworked, information is scattered, and routine tasks can consume time that ought to go toward care, investigation, or judgment. The relevant standard is therefore not purity. It is whether the system gives more capacity to the people with duties of care and accountability, or whether it shifts the burden onto the person least able to absorb an error.
What the system cannot carry
Average performance can sound reassuring while the same people repeatedly carry the cost of a false flag, missed need, or unavailable remedy. At an Eldercare Service, the missing element is often not another generated sentence. It is a person or office willing to own the decision, look at the exception, and say what will happen next. A model can make a pattern available. It cannot by itself owe fairness, keep a promise, or repair the cost of being wrong. Those remain responsibilities of people acting through institutions.
The distinction matters especially when the interface is persuasive. A calm tone, an apparent memory, a citation-shaped answer, or a confident recommendation can make the service feel more authoritative than its actual evidence warrants. The person receiving the output should not have to reverse-engineer the product in order to know whether it is a suggestion, a record, a decision, or a dead end.
The record a person should be able to find
For this use to be defensible, the care provider and its technology partners should be able to show what the system saw, what rule or prompt shaped the output, who reviewed it, and what changed because of it. That record is not bureaucratic decoration. It is the difference between a system that can be investigated and one that can only be narrated after the fact. It also protects workers inside an Eldercare Service: without a clear record and real authority, they can be made responsible for a product they were not empowered to question.
- What exact task is the system performing in this setting?
- What information shaped the output, and what relevant information may be absent?
- Who adopted the output as a reason to act or withhold action?
- What does an error cost a person here, and who is most likely to bear it?
- Can the person reach a human being with enough authority to investigate and change the result?
- Can the care provider and its technology partners pause or change the system when the record shows that it is causing harm?
A better operating rule
Specify the harms, monitor who bears them, and stop treating preventable human loss as a routine operating expense. In this setting, that means treating the system as part of a relationship of power, not as a floating intelligence that happens to offer advice. It means disclosing when automation is involved, preserving meaningful alternatives, and refusing to make a person’s dignity contingent on their ability to perform fluency, patience, or technical literacy for a machine.
The test is not whether the answer sounds human. The test is whether a human being can still obtain a responsible answer.
The argument is deliberately practical. It leaves open hard questions about minds, intelligence, and future systems while refusing to let those questions eclipse the people already present. If the care provider and its technology partners wants the benefit of automation, it must retain the burden of explanation, correction, and care. That is the line between assistance and abdication.