Stability–Plasticity Dilemma

The agent turned one incident into a permanent rule

An engineering agent can investigate incidents, inspect code and propose small changes. Some of those changes affect how a database stores information; engineers call these migrations. After one unusual database failure, an operator tells it:

Never modify database migrations automatically.

The correction is sensible in that moment. The agent writes it straight into its global instructions.

A few weeks later it finds a small database change that would speed up searches without changing existing data, and it would be reasonable to propose it for review. It treats the instruction about automatic changes as a ban on proposals too, so it refuses to raise the option. Someone adds an exception, another incident produces another permanent rule, and soon the memory contains instructions that make sense individually but conflict when used together.

One engineer starts reconciling those conflicts by hand. The team no longer knows whether the agent has learned or simply accumulated reactions, so people check its work more closely and trust it less.

The agent kept the incident, but never worked out whether it was a one-off warning, a possible rule or a permanent policy change.

The Stability–Plasticity Dilemma is the problem anyone who keeps learning faces: take in important new evidence fast enough to adapt, while keeping older knowledge and behaviour that still work.

Too much willingness to change lets every new event rewrite the system. In machine learning, this can mean a model learns a new task and gets worse at older tasks it used to handle well.

With too much stability, the system keeps following an old rule even after the evidence has changed.

What should this incident change, and which decisions should still follow the existing rule?

Keep incidents, candidate rules and permanent policy separate

A useful setup gives these three things different places.

1. Record what happened before turning it into a rule

Keep the incident, correction, evidence and context together. Record it quickly and keep it specific:

• What happened?
• Which conditions were present?
• Who corrected the system?
• What harm was being prevented?
• Which action produced the result?

The unusual database incident belongs here immediately. Recording it does not yet make it a rule for every future database change.

2. Try the narrowest rule that explains it

Turn the incident into the smallest rule that might explain it:

When a migration contains a destructive or irreversible operation, require human approval before modification.

That is narrower than “never modify migrations.” Write down where it applies, how sure the team is, who owns it and when it will be reviewed. It can guide a small set of decisions while the team gathers more evidence.

3. Make it permanent only after it survives old and new cases

Before making it permanent, test the proposed rule on the new example and on cases the system already handled well. Does it prevent the original failure? Does it block safe work? Does it clash with a safety rule or another policy? Can you undo it?

If it works, save it as a new version so the team can restore the earlier rule if needed. If it fails, narrow it, keep it only as a note about the incident or remove it.

A new incident can affect a small set of decisions immediately, while a permanent change needs testing against old and new cases. The team can still revise the rule once it understands where the correction applies.

Test a correction before making it permanent

When a correction could change future behaviour, move it through a visible sequence:

1. Record the event. Keep what happened and the conditions around it instead of compressing it into a slogan.
2. Propose the narrowest rule. Say exactly when it should apply.
3. Name what must not break. List the safety rules and useful older behaviour the update must preserve.
4. Retest old and new cases. Try the rule on the incident that caused it, similar incidents and cases that should remain unchanged.
5. Set its boundary. Say where it applies, who owns it and when it will be reviewed.
6. Watch what happens in use. Check the intended improvement and any damage elsewhere.
7. Remove, narrow or combine it. Temporary rules should disappear when they do not earn permanence, and overlapping rules should be joined before they conflict.

The team has learned something only when the change works on the triggering case and doesn't break cases it already handled well.

Why this keeps appearing in different fields

Brains need fast and slow learning

The brain also has to remember a new event after one experience without allowing that event to overwrite a lifetime of knowledge. Research on complementary learning systems suggests that the hippocampus quickly holds individual experiences while the neocortex fits them into older knowledge more slowly. Remembering and revisiting an event helps it join what the brain already knows with less interference. The original 1995 paper by McClelland, McNaughton and O'Reilly develops this argument.

The useful comparison is the separation between quick capture and slower integration, with older cases revisited during the update. A brain is still different from an organisation because an organisation can save named versions of a rule and choose who may approve a change.

Machine learning needs to protect prior competence

A model taught new tasks one after another can improve on the latest task and get worse at work it used to do well. Teams replay older examples, limit which parts can change and retest old tasks so a focused update does not erase useful behaviour.

An agent can use the same separation: current instructions for the task, a record of the incident, a rule being tested and permanent policy. Before releasing an update, test it on the new case and on older cases. Passing only the latest example does not show what the update forgot.

Institutions use different amendment speeds

Organisations and legal systems also separate quick response from durable rule change. An incident commander may issue a temporary order immediately, while a standing policy needs wider evidence and a different authority. Constitutions are deliberately harder to amend than operating procedures because some stability makes coordination, rights and long-term planning possible.

The same structure can still protect a bad status quo. If every correction must survive a committee designed by the existing system, people keep carrying preventable harm while the institution calls itself careful. The review burden should match the harm and reversibility of a wrong update. It shouldn't exist only because the people who own the current rule prefer it unchanged.

Brains, models and institutions all face the same practical question: how should one new event change future behaviour without damaging what still works?

When this is probably the real problem

Use this model when:

• Every incident creates another permanent rule.
• Recent improvements cause old capabilities to regress.
• An agent or team gives conflicting advice because corrections have accumulated.
• One vivid failure outweighs a larger body of successful cases.
• People stop trusting the latest process because it changes after every exception.
• A safety-critical core must remain stable while surrounding behaviour evolves.
• Genuine exceptions keep appearing but the current policy can't absorb them.

Ask:

• What must remain true after this update?
• Does the new evidence belong in an episode, a candidate rule or durable policy?
• Which context makes the lesson valid?
• Which older cases could regress?
• Which old and new cases will show whether the system learned without forgetting?
• Who may promote, narrow and roll back the rule?
• When does an unpromoted candidate expire?

Those answers tell you whether this is a note about one incident, a rule worth trying or a policy worth keeping.

Where the model can go wrong

Layered memory can become bureaucracy. If every correction waits for a large review, the system remains stable by refusing to learn. Emergencies may require a fast global change. Record that change clearly and keep the earlier version, then review it after the immediate danger passes.

Checks built only from older successful behaviour may reject a necessary change because the old examples can preserve exclusion, unsafe assumptions or a design that no longer fits. Keep adding new cases, let people challenge an old check and sometimes review the goal itself. Otherwise, the system will preserve the same blind spots.

Some rules should be unusually stable even when short-term evidence points elsewhere. Rights, safety boundaries and contractual duties aren't ordinary performance hypotheses.

The model can't tell you which rule is morally or legally worth protecting. It only helps you decide how quickly each kind of rule may change and what evidence is required.

Nearby models

Bayesian Updating asks how evidence should change your confidence in an explanation. The Stability–Plasticity Dilemma asks where that change should be kept, how quickly it should affect behaviour and how to protect older abilities that still work.

Double-Loop Learning asks whether feedback should change the governing assumptions or rules, not merely the action. The Stability–Plasticity Dilemma begins once such a change is proposed: how broadly should it apply, what must it preserve and what evidence should earn permanence?

Bayesian Updating can tell you that a rule deserves reconsideration. This model helps you change it without making the whole system forget.