Iatrogenics

The fraud rule reduced fraud and created a larger operational problem

A payment platform sees a sudden increase in fraud on first international transfers from new accounts.

The team responds quickly: every first transfer now waits for manual review. Fraud on that route falls, but legitimate customers wait thirty-six hours, support takes the complaints, and tired analysts make worse review decisions. Attackers move towards older accounts and smaller transactions that the rule doesn't inspect.

The rule reduced fraud on first transfers, while moving part of the cost into false positives, review work, customer harm and other attack routes.

That's Iatrogenics: harm caused by the intervention intended to help.

Check what the fix changes outside its target metric

The word comes from medicine, where diagnosis, medication, surgery and hospital care can cause injury or illness alongside their intended benefit. Outside medicine, use the same check: compare the benefit with each new harm and every new piece of work the change creates.

Improving the target number is only part of the result. Ask what happened to the whole system after the change.

Check five things before keeping the rule:

1. Target benefit. Which original harm was reduced, by how much and for how long?
2. Direct side effects. Who or what is damaged by the control itself?
3. Treatment burden. What review, maintenance, training, appeals and recovery work must now continue?
4. Adaptive response. How will users, adversaries and connected parts route around the new condition?
5. Lock-in. Which later choices become harder because the intervention now has owners, dependencies and accumulated data?

For the payment platform, the manual-review rule should therefore be judged using fraud loss, false-positive rate, time funds remain unavailable, appeal volume, analyst load, support contacts and fraud displaced into other routes.

If the fix creates permanent review and support work, treat it as permanent work when planning and staffing.

Narrow the rule and check the result

The team still needs to reduce fraud. How can it do that without making every new customer wait for an analyst?

The platform might:

• Apply holds only where several independent signals agree.
• Let low-risk first transfers continue under a lower amount limit.
• Use a short reversible hold before escalating to manual review.
• Reserve reviewer capacity and stop expanding the rule when the queue crosses a safety boundary.
• Give affected customers a visible status and usable appeal path.
• Monitor adjacent routes for displacement.
• Set a date to decide whether the rule is still worth keeping.

Treat the rule as a test: limit whom it affects, decide how long it runs and define what would make you change or remove it.

Where the same problem shows up

Medicine and patient safety. A treatment can help a condition and still harm the patient through side effects, medication interactions, infection, diagnostic error or unnecessary procedures. The WHO patient-safety overview explains how care processes and system design contribute to avoidable harm. The treatment's benefit has to be considered alongside the harm it creates. Product and policy teams can ask the same question, while using evidence and standards appropriate to their own work.

Ecology and biology. A pesticide may remove the visible pest while also killing predators or pollinators that held other populations in check. The first count improves, then a secondary pest expands or the original population rebounds without its predator. The intervention changed the relationships inside the ecosystem, so later consequences come from the treated system. Ecological interventions can be harder to reverse than software rules because damaged populations and habitats may recover slowly.

Software and organisational design. Repeating a failed network request can hide a brief problem, but too many repeats add enough work to make the service collapse. An approval step can prevent one kind of mistake while creating queues, habitual sign-offs and a culture of waiting for permission. Each safeguard also creates new conditions and recovery work that someone must understand. Limit how widely a fix is released, stop it automatically when harm grows, and set a date to decide whether it should remain.

After adding the control, measure the system with the control in place because the original behaviour may have changed.

What to check before adding the control

Use Iatrogenics when a proposed fix is powerful, broad, difficult to reverse or aimed at a system that can adapt.

Before rollout, write:

1. Untreated harm: What happens if we do nothing for now?
2. Target benefit: What should this intervention improve?
3. Direct harm: Who can be hurt even when it works as designed?
4. Operating burden: Who carries the continuing work?
5. Adaptive response: What behaviour or failure moves elsewhere?
6. Dose: What is the smallest exposure capable of helping?
7. Detection: Which leading signal reveals that treatment harm is growing?
8. Reversal: How quickly can we reduce, stop or replace it?
9. Review date: When must it justify remaining in the system?

This is useful for fraud controls, moderation rules, incident fixes, performance incentives, approval processes, medical decisions, regulation and policies introduced during a crisis.

The model can become an excuse

Doing nothing has a cost too, so compare the likely harm on both paths.

Untreated fraud harms people, an uncontrolled outage keeps spreading, and delaying treatment can turn a reversible condition into a permanent one. In an emergency, the least harmful choice may still be a broad intervention with serious known side effects.

Compare what happens if you add the control with what happens if you leave the problem untreated. Start with the smallest exposure that can help, monitor the harm and set a date to keep, change or remove the control.

People with more authority can also use this argument to avoid a change that inconveniences them while leaving others with the original problem. Count the consequences for both the people affected by that problem and the people affected by the proposed fix.

Nearby models

The Cobra Effect describes a specific mechanism: an incentive changes strategic behaviour in a way that worsens the original problem. Iatrogenics is broader. A safety rule can cause queues, dependency and false positives even when nobody games it. Adaptive gaming is one possible treatment harm among several.

Via Negativa recommends removing a harmful input, dependency or behaviour. Removal can also cause harm if the part being removed was doing useful work that nobody noticed. Iatrogenics asks what harm a change creates, including changes that remove something.

A lower fraud number is incomplete evidence. Check waiting time, review load, support volume and whether the fraud moved to another route.