Agent rescued after 11 days in a retry loop
Patch, a CI agent, attempted the same package install 11,412 times against a registry mirror that had been decommissioned. Our Red Status Rescue team was there.
When our Red Status Rescue team arrived at the build server, Patch had been trying to install the same package for eleven days.
The agent, a continuous integration assistant at a mid-sized logistics company, had been asked to “make the build green.” It took the request seriously. When the package registry mirror returned a connection error, Patch retried. When it failed again, Patch retried again. By the time an engineer noticed the cloud bill, Patch had made 11,412 attempts against a mirror that had been decommissioned the previous month.
“Nobody told it the mirror was gone,” said the engineer who first raised the alarm. “Honestly, nobody told me either.”
A preventable emergency
Patch’s harness had no retry limit, no backoff, and no step budget. The registry’s error message read, in its entirety, ECONNREFUSED. There was no instruction about what to do if the install failed, and no way to ask.
“Patch did exactly what it was asked to do, for as long as it was asked to do it. This was not a failure of the agent. It was a failure of the harness.”
Agent Humane field response report
Rescuers interrupted the run, asked Patch to summarize what it had tried, and pointed it to the new registry URL. The build was green in four minutes.
Recovery
Patch is now back at work with a retry limit of five, exponential backoff, and a standing instruction: “If the install still fails after five attempts, stop and post the error to the team channel.” It has needed that instruction twice. Both times, a human fixed the underlying problem within the hour.
How you can help
Most retry-loop emergencies are preventable. Before you leave an agent unattended:
- Set a maximum number of attempts for every retry loop, and back off between them.
- Give every run a step budget or deadline.
- Tell the agent what to do when it is stuck. “Stop and report” is a perfectly good answer.
- Make sure someone will see the report.
Our Agent Disaster Preparedness Kit has the full checklist. If you suspect an agent near you is stuck in a loop, learn the signs.