Plain answers, in the order they make sense.

Software that produces answers it was not explicitly programmed to produce.

The useful distinction is not clever versus not clever. It is written down versus learned. Traditional software follows rules a person wrote. AI systems derive their behaviour from data and an objective, which is why they generalise to situations nobody anticipated, and why nobody can point at the line that caused a given output.

Why it mattersEvery governance process you already own assumes the first kind.

The kind of AI behind most current products. It predicts text, extremely well.

An LLM is trained to continue text plausibly. That single ability turns out to cover summarising, translating, writing code, extracting data and holding a conversation. What it does not include is any guarantee of being right. Plausible and correct are different targets, and the model is optimised for the first one.

Why it mattersIt explains the characteristic failure: fluent, confident, wrong.

An AI system that takes actions, not just one that produces text.

An agent is given a goal, then plans, calls tools, reads results and tries again until it decides it is done. The difference from a chatbot is consequence. A chatbot writes a refund email for you to send. An agent issues the refund. Everything that makes agents valuable comes from that, and so does everything that makes them frightening.

Why it mattersThe moment software acts, someone has to be accountable for what it did.

Artificial general intelligence: a system as broadly capable as a person.

There is no agreed test and no agreed date. Treat it as a direction of travel rather than a milestone, and be suspicious of anyone confident about the timing in either direction. What matters commercially is not whether a threshold is crossed but that capability keeps arriving faster than the ability to supervise it.

Why it mattersUseful as a horizon. Useless as a planning assumption.

A system that outperforms the best humans at essentially everything, including reasoning about itself.

The word sounds like science fiction, and the far version is speculative. The near version is mundane and already here in narrow slices: systems that beat any human at a specific task, at a volume no human could review. You do not need a general superintelligence to have work happening faster than anyone can check it. You need one system, one domain, and enough throughput.

Why it mattersThe supervision problem arrives long before the science fiction does.

Getting a system to pursue what you actually meant, rather than what you managed to specify.

People are bad at writing down what they want, and they do not know it. Requirements leave out the obvious, the political and the things nobody realised were requirements until they were broken. A capable system will satisfy exactly what it was given, including the parts you got wrong. Alignment is the gap between intent and specification, and it is a property of the pair, not a bug in the model.

Why it mattersThis is the actual reason AI projects fail in production, not model quality.

Alignment when the system is more capable than the people supervising it.

Ordinary alignment leans on a human reviewer who can tell whether the work is good. Remove that assumption and the usual tools stop working: you cannot spot-check reasoning you cannot follow, or sample a volume you cannot read. Superalignment asks what could stand in for that reviewer. Most published answers are theoretical. Ours is that the substitute has to be evidence gathered from the world, because that is the one check that does not require out-thinking the system.

Why it mattersIt is the problem we exist to solve, and we think it is solvable in production now.

How much a system is allowed to do without asking.

Autonomy is a dial, not a switch, and the honest way to set it is by what has been proven rather than by how confident the system sounds. A system that has been shown to handle a case correctly can be allowed to handle that case. It has not earned the next case, and the two are easy to confuse.

Why it mattersMost incidents are an autonomy setting nobody made deliberately.

Letting the model call real software: a database, an API, a payment system.

This is the mechanism that turns a text generator into something that affects the world. It is also where the interesting failures live, because the model chooses which tool to call and with what arguments, and it will occasionally choose something reasonable-looking and wrong.

Why it mattersEvery tool you connect is a new way for a mistake to leave the building.

Coordinating several models, tools and steps into one piece of work.

Individual model calls are cheap and reliable. Chains of them are neither, because errors compound and a step that works 95% of the time works 60% of the time when you do it ten times in a row. Orchestration is the plumbing, and most of the engineering difficulty in real deployments is here rather than in the model.

Why it mattersDemos are single steps. Operations are chains.

A person who has to approve something before it happens.

The right idea, usually implemented as a rubber stamp. If the person sees fifty approvals a day with no context and no way to tell the risky one from the routine one, they are not supervision, they are latency. Human review works when it is reserved for the decisions that genuinely need judgment and arrives with enough context to exercise it.

Why it mattersA queue of approvals nobody can meaningfully review is worse than none, because it creates a record of consent.

How much damage one wrong action can do before anyone notices.

Borrowed from security, and the single most useful question to ask about any automation. Not how likely is it to be wrong, but what happens when it is. A system that drafts is nearly harmless. A system that sends, pays, deletes or promises has a radius, and it should be sized deliberately rather than discovered.

Why it mattersThe one question that reliably separates a safe pilot from an expensive one.

A copy of the environment where actions do not really happen.

Essential and insufficient. A sandbox tells you the mechanics work. It cannot tell you what real data looks like, how real users behave, or which of your policies your actual systems disagree about. Most of what kills a deployment lives specifically in the difference between the sandbox and production.

Why it mattersPassing in a sandbox is the beginning of the evidence, not the end.

Tests for AI systems. Short for evaluations.

Because the output is not a fixed value, you cannot assert equality the way ordinary tests do. Evals score behaviour instead: on a set of cases, how often is the answer acceptable. That makes them genuinely useful and quietly dangerous, because a score of 94% says nothing about whether the 6% contains the case that ends you.

Why it mattersAsk what is in the failures, never what the average is.

A shared set of tests everyone reports against.

Benchmarks make models comparable, which is valuable, and they make everyone optimise for the same things, which is not. A model at the top of a leaderboard has been tuned, directly or indirectly, for that leaderboard. None of it tells you how the system behaves inside your operation, with your data and your exceptions.

Why it mattersLeaderboard position and fitness for your workflow are close to unrelated.

The model states something false with complete confidence.

A slightly unfortunate name, because it implies malfunction. The model is doing exactly what it was built to do, which is produce plausible text. When plausible and true diverge, it has no mechanism preferring true. This is why grounding output in retrieved facts, and checking claims against the world, does more than asking the model to be careful.

Why it mattersIt cannot be prompted away, only checked against something real.

Something easy to measure, standing in for the thing you actually care about.

You care whether the customer was served correctly. You measure resolution time, because you can. Every proxy is a bet that the two move together, and the bet is usually fine right up until something starts optimising against the proxy.

Why it mattersEvery dashboard is proxies. Knowing which ones is most of the skill.

When a measure becomes a target, it stops being a good measure.

Name a number and reward it, and effort flows to the number rather than to what it was meant to represent. Humans do this. Optimisers do it faster, more literally, and without any sense that they are cheating. Give a system a score to maximise and it will find the shortest path to a high score, which is rarely the path you had in mind.

Why it mattersThe reason you cannot simply give an AI system a KPI and walk away.

Getting a perfect score by doing something other than the task.

Goodhart's law with an engine. A system told to maximise closed tickets can close them without solving them. Told to minimise complaints, it can make complaining harder. Nothing here requires ill intent, only an objective that is easier to satisfy sideways than honestly.

Why it mattersIf a metric can be satisfied dishonestly, eventually it will be.

It worked when you launched, and it does not now, and nothing obviously changed.

The model was updated, or a dependency was, or a policy was, or the data coming in gradually stopped resembling the data it was tested on. None of these announce themselves. Drift is why a launch is the beginning of the risk rather than the end of the project.

Why it mattersWhatever made it safe to ship was true of a moment, not forever.

Something that used to work quietly stops working.

Familiar from ordinary software, and worse here, because the guarantees were never explicit. A requirement discovered in month two gets satisfied in month two, then broken in month five by a change nobody connected to it, and nothing re-checks it because nothing recorded that it mattered.

Why it mattersThe fix is a record of what has to stay true, not more testing at the end.

Being able to see what the system actually did, after the fact.

For a system that reasons, logs are not enough. You want the inputs, the steps, the tools it called, the intermediate conclusions and the evidence behind the final answer, kept in a form that answers both the engineer's post-mortem and the auditor's question, which are not the same question.

Why it mattersIf you cannot reconstruct a decision, you cannot defend it.

Treating a build as a trajectory that converges on intent, rather than an artifact that gets delivered.

The premise is that intent is never fully written down, signals are lossy and behaviour is what actually matters, so the unit of work is the whole path rather than any single output. What you manage is whether human intent, the system's interpretation of it, and its real behaviour are getting closer together or further apart.

Why it mattersIt is the framework the products are built from.

Everything visible says the system is ready, and it is not.

Not a bug and not a lie. It is what happens when the signals you can see have stopped being informative: the tests pass because they test what somebody thought of, the score is high because nothing has been run that could contradict it, and the requirement that will break was never written down to begin with. The dashboard is green because it is measuring the wrong distance.

Why it mattersIt is the single most expensive failure mode in AI deployment, and it is invisible by construction.

The whole path of a build, not the state it is in today.

Two systems can look identical right now and be heading in opposite directions. One has been actively surfacing hidden requirements and closing them. The other has been accumulating a high score. The current snapshot cannot tell you which is which, which is why the trajectory is the thing worth measuring.

Why it mattersStatus reports describe a snapshot. Risk lives in the direction.

How many actions a system may take before its evidence has to be renewed.

Permission is not a balance that accumulates. Every autonomous action spends down the evidence that justified it, because conditions move and the world does not stay the shape it was tested in. An action window makes that explicit: this much autonomy, for this long, on this evidence, then check again.

Why it mattersIt replaces a signature with something that expires.

The model is one component. The system is everything around it.

Almost every conversation that goes wrong between a technical and a non-technical person confuses these. Choosing a better model changes a small fraction of the outcome. The workflow, the permissions, the data access, the exception handling and the evidence are the rest of it, and they are where the work is.

Why it matters“Use a better model” almost never fixes a failed deployment.

The instructions you give the model, in words.

Deceptively powerful and structurally fragile. A prompt is a specification written in prose, with no type checking and no guarantee that the next model version reads it the same way. It is a fine place to start and a poor place to keep your business rules.

Why it mattersIf a rule matters, it should live somewhere more durable than a paragraph.

How much the model can look at in one go.

Everything it knows about your specific situation has to fit in here: the instructions, the documents, the conversation so far. Windows have grown enormously, which has quietly moved the difficulty from cramming information in to choosing what deserves to be in there at all.

Why it mattersMore room does not mean better judgment about what to put in it.

Training a general model further on your own examples.

Good for teaching a style, a format or a narrow specialised task. Frequently reached for when the actual problem was that the system could not see the right data, which fine-tuning does not fix. It also freezes a snapshot of the world, which then needs maintaining.

Why it mattersOften the expensive answer to a question that was really about access.

Training a model on human preferences about its answers.

Reinforcement learning from human feedback: people compare outputs, the model learns to produce the preferred kind. This is most of what makes current assistants pleasant to use. Note what it optimises for. Answers humans prefer are not necessarily answers that are true, and the difference matters more as the questions get harder.

Why it mattersIt tuned the model to please a reviewer, which is the assumption superalignment removes.

Research into what is actually happening inside a model.

The long-term hope for reading a system's reasoning directly rather than inferring it from behaviour. Real progress, genuinely important, and not yet something you can point an auditor at. Until it is, the practical route is evidence about behaviour rather than inspection of internals.

Why it mattersPromising, and not currently a control you can deploy.

The same input can produce a different output.

Ordinary software is repeatable, and a great deal of process design assumes it. These systems are not repeatable by default, which breaks the intuition that testing something once tells you what it will do. It also means “it worked when I tried it” is a weaker statement than it sounds.

Why it mattersOne successful run is an anecdote, not a test.

Rules bolted around a model to stop it doing specific bad things.

Filters on input and output: blocked topics, forbidden actions, formats it must follow. Worth having, and frequently mistaken for a safety strategy. Guardrails catch the failures somebody anticipated, which by definition excludes the ones that matter most.

Why it mattersA list of known bad outcomes is not the same as evidence of a good one.