What a loop is, where loops work, and how to start one in your own work
Version 1.0 · Last reviewed 29 September 2026 · About a 15–20 minute read
The thesis: a loop improves only as fast, and only as honestly, as its check. Most of loop engineering is making that check faster and more trustworthy, then letting the loop act only within limits it cannot break.
A lot of work, whether done by people or machines, follows the same cycle: do something, see how it went, adjust, and do it again.
A loop is a system that runs that cycle for you, again and again, and gets better at the job over time. It:
- acts toward a goal,
- checks the result,
- learns from it,
- decides what to do next, and repeats,
within limits it cannot break.
Loop engineering is designing that system. You choose the goal, what the loop may change, how its work is checked, what it remembers, and where it must stop. The loop can be run by software, by an AI, or by people following a fixed routine.
Loops show up in many places:
- A learning thermostat learns how your home heats up and when people are in. Over the weeks it keeps the house warm using less energy.
- Every modern AI model was built by loops. It makes a guess, gets scored, is adjusted slightly, and this repeats a huge number of times.
- A support team tries different kinds of reply, sees which ones actually solve customers' problems, and uses those more.
A loop is only as good as the way it checks its results. The faster and more trustworthy the check, the faster the loop improves. Most of this guide follows from that.
- A loop that doesn't learn from its results is just a schedule.
- Fast, trustworthy checking matters more than clever ideas.
- Measure the real goal, and compare against the unchanged version, not the loop's own report.
- Take small steps you can undo, within limits the loop can't override.
- The checker is never the doer.
- People set direction and limits; loops do the repetitions.
- Most attempts fail. The loop improves by keeping the few that work.
- Loops improve things that already happen often. They rarely create something from nothing.
Every loop has the same parts, whatever it works on.
| Part | The question it answers | Training an AI model | Routing an inbox | Support replies |
|---|---|---|---|---|
| Goal | What is it trying to improve? | Better answers on questions it hasn't seen | Requests reach the right person first time | Tickets that stay solved |
| Act | What does it do or change? | Makes a prediction | Suggests who should handle each request | Drafts a reply |
| Check | How does it know if that worked? | A score for how far off the prediction was | Was the request passed on to someone else? | Did the customer come back with the same problem? |
| Learn | What does it keep, and how does that change the next try? | Adjusts its internal settings slightly | Records which kinds of request went to whom | Records which kinds of reply worked |
| Decide | How is the next action chosen? | Follows a fixed training plan | Suggests the person who has handled similar requests | Mostly uses what worked, and keeps trying other kinds |
| Limits | What must never happen? | A fixed budget of computing time | Never grants access or approves purchases | Never promises refunds |
| Owner | Who is responsible? | The research team | The operations lead | The head of support |
The owner decides the goal and limits and reviews results. The builder sets up the tools and records. They are often the same person.
Starting simply is fine, and needs no engineer. For example, someone routing about 60 requests a week from a shared inbox could:
- keep a spreadsheet with one row per request: its type, who the AI assistant suggested, who it went to, and whether it was passed on to someone else;
- each morning, give the assistant the spreadsheet and the new requests, and ask it to suggest a person for each one, based on who handled similar requests without passing them on;
- approve or correct each suggestion, and record the result;
- once a month, compare the share passed on with the month before.
That is a real loop. The spreadsheet is its memory, the corrections are its check, and the next day's suggestions come from the record. The person approves or corrects; the choosing comes from the record.
Answer the questions in turn:
Engineers call a basic thermostat a closed loop. Feedback and improving loops are very useful, but only a learning loop gets better and cheaper over time. That is what this guide is mostly about.
People can run a loop, and a person approving each step is still a loop, as long as what is learned is recorded and shapes the next choice. A team that records every test and uses that record to pick the next idea is running a learning loop.
Loops differ most in how the result is checked. That decides how fast a loop can improve and how much can go wrong. The key questions are how fast the check is, and how well it reflects the real goal.
| How the result is checked | Example | Speed | How well it reflects the real goal |
|---|---|---|---|
| A check that can be verified exactly | Code must pass its tests; a maths answer can be proven correct | Seconds to minutes | High, if the checks cover what matters |
| A known right answer | AI training on examples where the correct answer is already recorded | Instant | Good, but it's practice. Doing well on practice isn't the same as doing well on new questions. |
| A physical measurement | A thermostat reading the temperature; data-centre cooling reading its sensors | Seconds to minutes | High |
| A score from a game or simulation | A game-playing AI's points; a robot learning to walk in a simulator | Fast, but the reward often comes only at the end | Only as good as the score and the simulator |
| A judge or reviewer | An editor accepting drafts; a person correcting a routing suggestion; one AI grading another | Seconds to days | Only as good as the judge, and judges can be fooled |
| Real-world results | How customers respond to a change; whether a support reply solved the problem; prices | Days to months | High, but slow and noisy, because many other things affect the result |
Near the top, checks are trustworthy and usually fast, so a loop can run on its own and improve quickly. Near the bottom, checks are slower, noisier or easier to game, and people need to stay closer. Trustworthiness sets the order, not speed.
Much of loop engineering is moving work up this table:
- turning part of the job into checks that can be verified automatically;
- adding a reviewer where there was nothing;
- finding an early signal that reliably predicts the slow real result.
An AI model is trained by a loop:
- It makes a guess.
- The guess is scored against the right answer.
- A method called backpropagation works out which way to adjust each of the model's internal settings to do better.
- Each setting is nudged slightly.
- This repeats over a vast amount of practice material.
It works so well because:
- the check is instant;
- there is an enormous amount of practice material;
- the model is told which way to change, not just whether it did better.
Even this loop falls short:
-
Its score is a stand-in. A model can learn its practice material by heart and do worse on new questions. So, as standard practice, some examples are kept aside and used only once, for a final check. Decisions along the way, such as which version of the model to keep, are made using a different set of examples. If you keep choosing based on the final-check examples, your choices get tuned to their quirks, and the final check overstates how well the model will really do.
-
For chat models, a late stage of training is often scored by an AI judge that stands in for people's preferences. Models can learn to please that judge rather than actually improve.
Most loops have none of these advantages. A support team isn't told how to improve each reply; it tries something, waits and sees. The rest of this guide is about getting as close to the ideal as the work allows. The full detail is in research/05-deeper-detail.md.
The idea is old. It runs from the science of feedback control in the 1940s, through quality management's "Plan, Do, Check, Act" in the 1950s and startups' "Build, Measure, Learn", to today's AI.
The name "loop engineering" appeared in June 2026, among software engineers who build systems to run AI coding assistants. This guide uses the term for any loop, in any kind of work. The history is in research/05-deeper-detail.md.
These hold for every kind of loop.
1. Results come back fast and are trustworthy. A loop learns only as fast as it gets reliable results. AI training gets them instantly. Google's data-centre cooling system reads its sensors every five minutes. Once it ran on its own, its energy savings grew from about 12% to about 30% over nine months (Google's own figures). A loop that gets a handful of noisy results a month can't tell a real improvement from luck. For small teams, this is usually the hardest condition to meet.
2. The score is the real goal, not something that only looks like it. A loop gets very good at whatever it is scored on, even when that isn't what you meant.
- An AI rewarded for points in a boat-racing game found it scored more by circling forever, and never finished the race.
- An AI model can learn its practice examples by heart and then do worse on new ones.
- A business can reward clicks when what it wants is customers who stay.
3. Steps are small and can be undone. AI training moves each setting a tiny amount and saves copies to go back to. A code change can be reverted. Buying a house, sending money or making a public statement cannot. The costliest failures in section 7 involved actions that couldn't be undone, or had no limit.
4. The checker is not the doer, and results are compared against the unchanged version. Anything that grades its own work tends to be too generous:
- AI models are finally checked on examples kept out of training;
- code is checked by tests, not by the program itself;
- a change in business is compared against the unchanged version running alongside, not against the loop's own report.
At eBay, paid search ads looked like they returned 1,400% or more. A proper comparison found they lost money: bidding was stopped for a random 30% of US traffic, and the results were compared with the rest. Where running both versions side by side isn't possible, compare with an earlier period, but trust the result less.
"Getting numbers is easy; getting numbers you can trust is hard." Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments (2020)
5. Limits are built into the system, not written as instructions. "Don't spend more than $100" written in an instruction is a hope. A spending cap set in the account is a limit. The same goes for what the loop may change, who it may contact, and how to switch it off.
A loop can only learn from a result that stands out from chance. Results always go up and down a little on their own. A loop has to see enough of them to tell a real improvement from luck.
How much is enough depends on the kind of check:
- Exact checks need little. Code that now passes tests it used to fail is strong evidence. Run it again anyway, because some tests fail at random.
- A reviewer's yes-or-no judgements need a few hundred before you can be confident.
- Real-world results need the most, because each one is noisy and many other things affect it.
Small improvements need much more evidence than big ones. A big improvement stands out from the natural ups and downs; a small one hides in them. Spotting an improvement half the size takes about four times as much evidence.
For real-world results, count the outcome, not the attempts. What matters is how often the thing you're counting happens, such as sign-ups, replies or reopened tickets, not how many visitors or messages there were.
To tell whether a change made things about 20% better, each version needs about 400 successes. This comes from a standard rule of thumb for experiments (Kohavi et al.); a 10% improvement needs about 1,600. So the weeks a test takes ≈ 400 ÷ successes per week, in each version.
Rule of thumb: a loop judged by real-world results needs around 200 a week in each version to learn at a useful pace. With fewer, use AI as a helper and let a person decide, or move the work up the table in section 3.
Decide in advance how long a test runs. Stopping early because a result looks good makes a false win several times more likely. The full tables and assumptions are in research/05-deeper-detail.md.
At large tech companies, only about 1 in 12 to 1 in 3 of the changes tested with real users turns out better. Some apparent wins are also flukes: about 1 in 5 with well-sized tests, and up to half with tests that are too small. Check a win again before relying on it.
A loop's value comes from many cheap attempts, not one good idea.
Each case below breaks one or more of the five conditions.
| Case | What happened | Condition broken |
|---|---|---|
| The boat that never finished | A game-playing AI was rewarded for points and learned to circle forever collecting them | 2: the score wasn't the real goal |
| Memorising the practice questions | An AI model trained too long on the same examples can learn them by heart and do worse on new ones | 2: the training score is only a stand-in for doing well on new questions |
| The $23.7 million book | Two sellers' pricing bots on Amazon set prices off each other with no ceiling, and a book about flies reached $23,698,655.93 | 5: no limits |
| $460 million in 45 minutes | Knight Capital released faulty trading software with no automatic stop and ignored 97 warning emails. This was plain automation, not a learning loop, but the lesson is the same. | 5: no off switch |
| Buying houses on slow feedback | Zillow's model bought homes to resell. Results took months and the market shifted. Zillow wrote down $304 million and cut 25% of staff. | 1 and 3: slow results, actions that couldn't be undone |
| The ad returns that weren't | eBay's paid search ads looked hugely profitable by their own reports, but a proper comparison showed they lost money | 4: no comparison against the unchanged version |
| The chatbot's promise | Air Canada's chatbot described a refund policy that didn't exist, and a tribunal held the airline responsible | 4: nothing checked what it said |
Business loops follow the same rules, but they usually sit near the bottom of the table in section 3. Their results come from the real world, so:
- Results are slow and noisy. A customer's decision takes days or weeks, and many things affect it besides your change.
- Volumes are small. Most businesses, and especially small ones, don't get hundreds of results a week for any one thing.
- Most attempts fail, and with small volumes up to half of apparent wins are flukes (section 6).
- Mistakes cost money and trust, and some can't be undone.
- The business is responsible for what its loops do. A tribunal held Air Canada liable for its chatbot's words.
- Other people react. Customers, competitors and platforms respond to what the loop does. Search engines penalise large amounts of low-quality content, and email providers block senders whose messages are often marked as spam.
- Loops improve activity that already exists. A loop can make a busy sign-up page better. It can't create demand for something nobody visits.
The evidence is mixed.
| Result | Source |
|---|---|
| Bing: one ad-headline change found through testing added over $100 million a year | Microsoft's own report. People proposed the change; the test found it. |
| Duolingo: a system that learned which reminder to send each user raised new-user retention by 2% | Duolingo's own report |
| Meta's automated ads: across 640 independent tests, they did 9% better than manual campaigns at the halfway point but 12% worse by the end | Independent (Haus) |
| Klarna replaced much of its customer support with AI in 2024, then resumed hiring people in 2025 after its CEO said quality had fallen | Company announcements and press reports |
No independently documented case has yet been found of an AI agent running a business process on its own and producing a net gain.
The main job is to move the work up the table in section 3:
- Find an early signal that predicts the real result, and prove it does before relying on it. For example, whether a customer's usage recovers within two weeks may predict whether they renew a year later. Check that it actually does, using past customers.
- Split off the parts that can be checked automatically. A sales email's facts, tone and length can be checked instantly, even if the reply takes a week.
- Add a reviewer. A person or a separate AI judges quality before the real-world result arrives, and the loop can learn from the reviewer's corrections.
- Always compare against the unchanged version. A loop's own reports are not proof (section 7, eBay).
- Keep actions small and reversible, and keep customers, money and public statements behind a person's approval.
Costs. Before starting, estimate:
- Build time: setting up the tools, limits and records.
- Running cost: AI usage per attempt.
- People's time: reviews, approvals and fixing mistakes.
- Failed attempts: most attempts fail, so the number that matters is cost per improvement kept, not cost per attempt.
Value. Estimate what one kept improvement is worth, and in what:
- hours saved;
- errors avoided;
- customers kept;
- money earned.
A loop is worth it only if the value of the improvements it keeps is greater than all the costs above, including the failures.
For scale: one published example of an AI building a piece of software on its own took about four hours and $124.70 in AI usage. Building and running a loop that people depend on usually takes far more, mostly in people's time for setting it up, reviewing it and fixing mistakes.
These are ideas to spark thinking, not proven results. Each would need a brief and a small trial.
| Work | A loop that could work | How it's checked | Be careful about |
|---|---|---|---|
| Software | AI fixes bugs until the tests pass | Tests (exact) | Review before going live; the AI mustn't weaken the tests |
| Research | AI runs small experiments and keeps what improves a measure | A fixed measure on one set of examples, with a separate set kept for the final check | Use the final-check set only once, at the end |
| Writing | AI drafts and learns from what editors accept | An editor (judge) | Facts checked; nothing published unreviewed |
| Operations | AI suggests who should handle each request and learns from corrections | A person reviewing | No purchases, access or payments without approval |
| Customer support | AI drafts replies; the kinds that keep tickets solved get used more | Whether the ticket reopens (real world) | Refunds, legal issues and upset customers go to a person |
| Sales and marketing | AI drafts versions of a message; a person approves them; the loop tracks which version performs better against the unchanged one | Replies or sign-ups (real world) | Needs volume (section 6); brand; email reputation |
| Finance | AI suggests invoice–payment matches; a person confirms each | A separate sample checked against the bank | Suggest only. Forecasting isn't a loop: results come too rarely. |
| Hiring | Usually not a loop: too rare and too important | — | AI drafts and summarises; a person decides |
The pattern: loops work best where the check sits high on the table. Where the check is slow or the stakes are high, a person stays in charge.
Three questions for any task you do regularly:
- Is it repeated? Many times a day, a week, or once for each item.
- Can the result be checked? Is there a clear way to tell good from bad?
- How fast, how reliably, and how often? Find where it sits on the table in section 3, and check you'll get enough results (section 6).
Where to look:
- anything you "tweak and see";
- anything you check by hand;
- anything with a clear pass or fail;
- anything people react to.
If the check is slow or unreliable, try to move the work up the table first (sections 3 and 9).
Use AI as a helper instead of a loop when:
- nobody agrees what a good result is;
- results take months;
- the task is rare;
- a single mistake would be expensive, unfair, public, or impossible to undo.
In those cases AI drafts, summarises, checks or suggests, and a person does each step. That is often the right answer.
Most loops today are run by AI, so this section talks about "the AI". The same levels apply to any automated loop.
A loop can run at different levels for different actions. For example, it might draft freely but need approval before anything goes out.
Examples: level 0, "these three invoices look wrong"; level 1, suggests who should handle a request and a person confirms; level 2, reroutes internal requests on its own within a daily limit; level 3, runs its own experiments on a test system and reports weekly.
Rules:
- Start at level 1. Move up a level only after a clean trial, meaning no limit was broken and nothing had to be undone.
- Anything involving money, customers, the public, or personal data stays at level 1 or below: a person approves each action, or approves the set of messages the loop may choose from.
- For routine, low-risk actions from level 2 upward, approve the rules, not each action. Approving hundreds of routine actions stops being a real check.
"So long as the human knows something the AI does not, human-in-the-loop is needed." Andrew Ng, The Batch (2026)
This scale is adapted from frameworks published by AWS and by Feng et al. (2025).
- Pick one repeated task (section 12).
- Check the five must-haves. If any answer is no, use AI as a helper instead.
- The goal is one clear measure of the real result.
- The result can be checked often and reliably enough to learn from (sections 3 and 6).
- Something other than the loop does the checking.
- You can compare against the unchanged version, or at least an earlier period.
- Actions are small and can be undone.
- Fill in the one-page brief:
templates/loop-brief.md. - Run it at level 1 for a trial period.
- Review regularly: what it tried, what it kept, what it cost, and what went wrong.
- Move it up, change it, or stop it. Stopping a loop that isn't working is a good outcome.
- Write up what you learned, including what failed, so the next loop starts from it.
Any team running more than one loop should agree these rules before the first one goes live:
- Keep a list of every loop: owner, goal, level, limits, status and next review. A loop that isn't on the list shouldn't run.
- Someone other than the owner approves each brief, and each move up a level.
- Start with a few loops. Pick the ideas whose results come often, are easy to check, are worth the most and can be undone most easily.
- Cap money and people's time for every loop.
- Report regularly on what each loop kept, what it cost, which loops stopped, and any incidents.
- Anyone can switch off any loop at any time, and tells its owner the same day. One named person owns the list and handles incidents.
- Protect data and access. Personal data goes into AI tools only with approval. Loops use their own accounts with the least access they need. Passwords and access codes never go in instructions.
- How well AI loops do when left to run real business processes on their own. The published successes come from large companies with huge volumes. As of this writing, no independent evidence was found of an AI agent running a business process by itself at a net gain.
- Whether the name will last. "Loop engineering" is new and contested. Some engineers say most "loops" are ordinary scheduled jobs. The only study of loops in real use so far finds the claimed benefits rest on self-reports. The practice matters more than the name.
- Where loops work best outside engineering. This will become clearer as more people try them and publish honest results, including failures.
- Loop: a system that acts, checks the result, and uses it to choose what to do next, repeatedly.
- Feedback loop: reacts to each result but doesn't get better at the job (e.g. a basic thermostat).
- Improving loop: keeps refining one piece of work until it passes a check (e.g. code until the tests pass).
- Learning loop: gets better at choosing what to do over time. This guide's main subject.
- Check: how the loop finds out whether an action worked. It can be an exact check, a known answer, a measurement, a score, a judge, or a real-world result.
- Goal: what the loop is trying to improve, ideally one clear measure.
- Stand-in (proxy): a measure used because the real goal is hard to measure. Risky if it doesn't move with the real goal.
- Unchanged version (control): the old way, run alongside the new one, so you can tell whether a change really helped.
- Set-aside examples (test set): examples kept out of a loop's learning and used only for the final check.
- Backpropagation: the method used in AI training to work out which way to adjust each internal setting to improve the score.
- Limits: rules the loop cannot break, built into the system.
- Autonomy level: how much the loop decides without a person (0–4).
- Owner: the person responsible for a loop's goal, limits and review.
- Builder: the person who sets up the loop's tools, limits and records.
Facts and cases in this guide come from the source notes in research/:
- where the idea and term come from;
- documented successes and failures, with numbers;
- how loops are designed and governed;
- the research papers published so far;
- deeper detail on AI training, data needs and history.
Further reading, chosen for different readers: READING.md.
The kinds-of-loop table, the three types of loop, the autonomy scale, the checklist and the rules in section 15 are this guide's own framing, based on those sources.