From Threat Modeling to a General Principle

For a long time, I thought I was simply changing fields. First emergency medicine, then public health, and later cybersecurity. The problems looked different, the language changed, and so did the tools, so I treated each transition as a move into a new domain with its own rules. Only much later did I realise that I had been circling around the same question all along: why do some systems become more resilient, while others, despite increasingly sophisticated controls and increasingly impressive reports, gradually lose the ability to achieve the purpose they were designed for? This realisation did not arrive as a theory, and I certainly did not begin my career looking for some universal principle connecting medicine, organisations, technology and politics. It came slowly, more as the recognition of something I already knew from practice. At first, I assumed I was simply encountering similar organisational problems in different places. Different industries have their own pathologies, different technologies create different risks, and different organisations make different mistakes. Over time, however, I began to suspect that beneath these differences there was something more persistent. Not the same problems and certainly not the same tools, but perhaps the same meta-rule.

Emergency medicine taught me to work with protocols. In an emergency, reducing complexity to a sequence of decisions is not merely useful; sometimes it is the only way to act effectively. You cannot keep every physiological dependency in your head while someone is deteriorating in front of you. You need a representation of the situation that is simple enough to act upon, and you need to know when to trust it. Public health taught me something different and, in retrospect, complementary: interventions do not happen in isolation. They happen inside systems, and systems respond. People adapt to incentives, prohibitions, warnings and controls. An intervention intended to eliminate one risky behaviour can change something elsewhere and create consequences that were never part of the original plan. This is one of the reasons harm reduction made such an impression on me. Its starting point is uncomfortable but pragmatic: risky behaviour will continue to happen. If that is true, preventing every instance of it cannot be our only objective. We also need to reduce the damage when prevention fails. That idea eventually followed me into cybersecurity more naturally than I expected. People click links because links are how the Internet works. They write passwords down because human memory has limits. They install unfamiliar applications because they are trying to solve problems and get their work done. Simply telling them to stop does nothing to address the reasons those behaviours exist. My earlier work in public health had already taught me to be suspicious of interventions that measure success primarily by the disappearance of a visible behaviour while paying much less attention to what happens elsewhere in the system.

I saw the consequences particularly clearly while working with phishing simulations. On paper, such programmes can look excellent. Click rates fall, training completion rises and dashboards begin to glow green. Management sees progress, the security team can demonstrate activity, and everyone has a number that appears to move in the right direction. Yet this does not necessarily mean that the organisation has become more resilient. Employees can learn to perform well in a simulation without becoming better at dealing with real attacks. They can learn that the objective is not to exercise judgement but to avoid being the person who clicked. They can become suspicious of the security team rather than more capable of recognising phishing, or start treating every simulation as a small internal trap whose purpose is to catch them doing something wrong. Research on embedded phishing training has also found that commonly deployed approaches can be ineffective and may produce negative side effects, including increased susceptibility to phishing. What bothered me was not simply that a training method might fail. Training methods fail all the time. What bothered me was that the programme could become increasingly successful according to its own representation of success while becoming less successful at the thing that representation was supposed to stand for. The organisation could improve the number and lose the purpose behind the number.

That was when I began to suspect that improving our representation of reality and improving reality itself might be two very different things.

By a system, I mean a collection of interconnected elements whose interactions shape the behaviour of the whole. An organisation is a system, but so is a government, an economy, a social network, a security architecture, a team or even a person. What makes these systems difficult is not simply the number of components but the relationships between them. Change one element and something elsewhere may change with it, sometimes immediately and visibly, sometimes indirectly and only after a delay. We cannot hold all of that complexity in our heads, so we construct representations of it. We build models, reports, dashboards, diagrams, procedures, risk matrices, rankings, KPIs, security scores and economic indicators. We do this not because we are foolish enough to believe reality is simple, but because abstraction is the price of being able to think and act at all. There is no practical way to manage an organisation of thousands of people, a national economy or a large cloud environment while continuously accounting for every relationship inside it. In that sense, we rarely manage systems directly. We manage their representations, and there is nothing inherently wrong with this. In fact, we cannot do otherwise. The problem begins when we forget that a representation is only a representation. A dashboard starts to stand in for the organisation, a KPI for the objective, a security score for security. The map does not merely help us navigate the territory; slowly, almost imperceptibly, we begin making decisions as though improving the map were the same thing as improving the territory.

This is also why KPIs are so powerful. Executives operate under enormous informational pressure. They cannot investigate every process, customer interaction, employee problem, dependency or vulnerability before making a decision, and I find the casual criticism of executive simplification slightly unfair for exactly this reason. Of course they want a simple answer. Of course they want the explanation that can fit on one slide. Of course they want someone to explain the problem in five minutes rather than five hours. Their job requires them to move constantly between domains in which other people have spent entire careers developing expertise. They need compression and prioritisation. They need complicated realities reduced to forms that can fit on a screen and still support a decision. The desire for simplification is not evidence of stupidity or laziness; it is rational, often necessary, and probably unavoidable. The difficulty begins when the simplification stops serving the decision and starts defining success. A number created to describe an outcome becomes something teams are expected to improve. Incentives align around it, budgets follow it, careers become attached to it, and eventually the organisation becomes very good at producing whatever makes the dashboard turn green. Nobody needs to conspire for this to happen. People simply respond rationally to what the system rewards. Whether the organisation has become equally good at achieving the underlying objective is a separate question, and often a much harder one to answer.

Only relatively late did I encounter Goodhart’s Law: when a measure becomes a target, it ceases to be a good measure. It immediately felt familiar. Someone had given a name to part of what I had been seeing for years. Yet I also felt that the interesting part extended beyond the measure itself. What concerned me was not simply that a metric could become corrupted once people learned how to satisfy it. The more interesting change seemed to happen around the metric. Once incentives, decisions and resources became attached to a representation, the representation started participating in the behaviour it had originally been created to describe. The system adapted to the description of itself.

Facebook is an unusually visible example of what I mean. Its early proposition was remarkably human: connect people. It gave us a way to maintain relationships across distance, follow the lives of people who mattered to us and gather around subjects we genuinely cared about. The value of that experience was obvious to users but difficult to express as a number. Meaningful connection is a terrible KPI. Engagement is much better. Engagement can be measured continuously through clicks, comments, shares, reactions and time spent on the platform, and once those signals become important representations of success, the system inevitably begins learning what produces them. Meaningful relationships happen to be one source of engagement, but they are hardly the only one. Outrage produces engagement. Fear produces engagement. Conflict, curiosity, amusement and sensationalism do too. A political argument between strangers can generate more measurable activity than an ordinary update from someone I actually care about. A short absurd video can hold my attention more effectively than the life of an old friend. Misinformation does not need to be true if being provocative makes it engaging. None of this requires the platform to become malicious. In some ways, the opposite explanation is more interesting: the system can produce these outcomes precisely because it becomes extraordinarily effective at doing what it has been asked to do. Engagement improves while other properties of the system, harder to measure and easier to overlook, begin changing around it.

The advertising system creates another version of the same tension. An advertising platform needs to know whether an advertisement works, so it measures attention, clicks and conversions. Fraudsters care about exactly the same observable signals. A deepfake investment scam is designed to capture attention and produce a click. So is a phishing campaign. The platform and the fraudster can therefore become very good at optimising the same observable representation while pursuing entirely different goals. From the narrow perspective of the metric, a fraudulent advertisement may perform beautifully. From the perspective of the larger system, every successful scam consumes something far harder to represent on a dashboard: trust. At first the effect may appear external to the metric. Revenue still arrives, the advertisement converts, engagement remains healthy. But trust is part of the system too. Once enough of it is consumed, users become less willing to believe what they see, less willing to interact and eventually less willing to remain in an environment they no longer consider useful or trustworthy. A locally successful optimisation can therefore degrade the conditions that made the optimisation valuable in the first place. This is the kind of feedback that interests me: not a simple case of somebody choosing the wrong KPI, but a system gradually reorganising around one observable dimension while the hidden dimensions accumulate consequences elsewhere.

I see an even clearer version of this problem in threat modelling, perhaps because this is the area in which I have spent enough time to distrust elegant abstractions. Security organisations need metrics, and for perfectly sensible reasons. We count vulnerabilities, measure remediation times, track incidents and phishing attempts, and compress all of this information into dashboards and scores. Tools such as Azure Secure Score are useful precisely because no executive, and probably no security engineer either, can continuously reason about every dependency in a large environment. The difficulty is that an attacker has no obligation to respect the way we have chosen to describe that environment. They do not care whether our Secure Score improved this quarter. They do not care whether we patched ten thousand vulnerabilities instead of eight thousand, or how many malicious messages our filters rejected yesterday. They are interested in whether a path to their objective still exists. A minor configuration weakness may mean very little by itself, as may an overly permissive identity, an exposed service, a forgotten trust relationship or an employee making an ordinary mistake. What matters is whether those elements can be connected. Once they can, a collection of individually unremarkable weaknesses becomes an attack path.

This creates an odd situation in which defenders and attackers can effectively be playing different games on the same field. The organisation looks at the number of problems removed, while the attacker looks at the relationships between the problems that remain. Both perspectives are rational, but they are answering different questions. Artificial intelligence is making the distinction more visible because automated systems are becoming better at discovering vulnerabilities and helping construct increasingly complex attack chains. The understandable defensive response is to accelerate remediation: find more vulnerabilities, prioritise them faster, patch them sooner and automate as much of the process as possible. There is nothing inherently wrong with any of this. Faster remediation is generally preferable to slower remediation. What worries me is the assumption that increasing the velocity of remediation necessarily increases resilience at the same rate. It is entirely possible to become extraordinarily efficient at removing large numbers of isolated weaknesses while leaving the relationships that form the most important attack paths almost untouched. The attacker is still asking a much simpler question: is there a path to the objective?

This is one reason I keep returning to threat modelling. Its value, for me, is not that it gives us another diagram or another security methodology. It forces us to question the representations we already have. Instead of treating vulnerabilities as an inventory of independent objects, it asks us to look at relationships, dependencies, trust boundaries and possible paths through the system. It brings our view of the environment slightly closer to the way an adversary sees it. The same thought leads naturally to TTPs — tactics, techniques and procedures. If our defences genuinely change the conditions under which an adversary operates, we should eventually force that adversary to change their way of operating as well. Yet familiar TTPs often survive while domains, payloads, infrastructure and individual tools are replaced around them. This is what I find compelling about the Pyramid of Pain: it shifts attention away from the volume of defensive activity and towards the cost imposed on the other side. Blocking an IP address may be useful, but it is cheap for an attacker to replace. Forcing an adversary to rethink techniques and procedures on which an operation depends is something else entirely. And this leaves me with an uncomfortable question. If our security programmes are improving year after year, why do we so rarely force attackers to fundamentally change how they operate? Perhaps in some cases we are improving the dashboard more than we are changing the game.

This is close to what I sometimes call bullshit security. I do not mean fake security, nor do I mean that the people producing these programmes are incompetent or dishonest. Quite often the opposite is true. Highly competent people can spend enormous amounts of sincere effort producing evidence that security is improving while the relationships responsible for the most important risks remain largely unchanged. More vulnerabilities are scanned, more tickets are closed, more controls are deployed, mean remediation time falls and reports continue to glow green. Each individual activity may be sensible. The problem appears only when activity itself becomes our representation of resilience and we stop asking whether the adversary’s options have meaningfully changed. Confidence can then grow faster than resilience, not because anyone has lied, but because the system has become extremely good at proving progress in the language it uses to describe itself.

Once I started seeing this distinction, I found it difficult to keep it confined to cybersecurity. Organisations need ways to understand whether people and teams are doing useful work, so they construct representations: performance ratings, utilisation, attendance, objectives completed, tickets closed, sales targets reached. Each of these tells us something, but none is identical to the thing we actually care about. Once the representation acquires consequences, people quite reasonably respond to it. If tickets closed determine performance, closing tickets becomes important. If office attendance represents commitment, being visible acquires value. If quarterly revenue dominates the evaluation of a company, decisions that improve quarterly revenue gain a gravitational pull even when their effects elsewhere are harder to justify. This does not mean the metric is useless or that the behaviour is irrational. In fact, that is precisely what makes the mechanism interesting. People can make individually rational decisions and still produce a system-level outcome that nobody intended. A company can reduce headcount and improve its financial ratios while simultaneously losing expertise, institutional memory and the capacity to innovate. The decision may still be correct; sometimes organisations genuinely need to become smaller. What interests me is what happens when one observable dimension of success becomes so dominant that consequences outside it become progressively easier to discount. An organisation can become more efficient according to its representation while becoming more fragile as a system.

The representation does not even need to be numerical. Status is also a representation. Prestige, titles, privileges, luxury and visible displays of wealth can all function as proxies for things that are much harder to observe directly: competence, contribution, influence or social importance. Once status itself becomes the objective, however, resources can begin flowing towards the production of its symbols rather than towards whatever those symbols were originally supposed to signify. History offers plenty of societies in which elites invested enormous resources in visible markers of position while tensions accumulated elsewhere in the system. I would not claim that status competition explains the decline of the Maya, political crises in ancient Egypt, the upheavals of European societies or their revolutions. Those events had ecological, military, economic, institutional and cultural causes that cannot honestly be compressed into a single mechanism, and doing so would commit exactly the kind of reduction I am arguing against. What I find interesting is something narrower: the recurring possibility that a representation of success becomes increasingly important while the conditions that originally gave that representation meaning deteriorate underneath it. The palace becomes grander while the state becomes more fragile. The title becomes more important while confidence in the institution declines. Visible status continues to grow while the social structure required to sustain it becomes increasingly unstable. For a surprisingly long time, a system can continue producing symbols of success even as its capacity to sustain them weakens.

Artificial intelligence has made the same question feel immediate again. Organisations are under enormous pressure to use AI. Executives are asked about AI strategies, employees are encouraged to integrate AI into their workflows, processes are redesigned around automation and large amounts of capital are being invested on the assumption that these systems will produce substantial productivity gains. Naturally, organisations want to know whether any of this is working, and once again they need a representation. How many employees use AI? How many processes contain it? How many prompts are submitted? How many tasks have been automated? How many tokens are consumed? All of these numbers tell us something about utilisation. None, by itself, tells us very much about productivity. An employee generating twice as many tokens has unquestionably consumed twice as many tokens. Whether they have created twice as much value is another question entirely. A department can automate more processes without making better decisions. An organisation can reduce the number of people required to perform a task while quietly increasing errors, weakening institutional knowledge or shifting costs somewhere else in the system.

This is why token maxing interests me. It is not really an argument about tokens. It is another example of what happens when something easy to observe becomes a convenient representation of something much harder to understand. Productivity is difficult to measure; AI utilisation is easy to measure. Once the two become associated strongly enough, increasing utilisation begins to feel like increasing productivity, and the organisation starts adapting accordingly. More AI becomes better AI, more automation becomes greater efficiency, lower headcount becomes higher productivity, and lower cost becomes organisational improvement. Any of those statements can be true locally, but none necessarily describes what has happened to the system as a whole. A company can improve revenue while degrading the product that generates it. It can reduce labour costs while weakening its ability to innovate. It can automate interactions while degrading customer relationships. At a larger scale, many firms can simultaneously pursue perfectly rational local efficiencies whose combined effects alter employment, demand and the wider economy on which those same firms depend. I am not arguing that automation is bad, that revenue should not matter, or that organisations should stop measuring things. That would simply replace one crude representation with another. What interests me is the distance between local improvement and systemic improvement. The spreadsheet may be correct while the model behind it is wrong.

This is where my observations eventually led me beyond Goodhart. Goodhart’s Law tells us something important about measures: once a measure becomes a target, its usefulness as a measure deteriorates. What I keep wondering is whether this is one visible case of a broader phenomenon. Perhaps, in a complex system, making one observable representation the dominant target of optimisation changes the behaviour of the system around it. The change does not stop at the variable being measured. It propagates through incentives, dependencies and feedback loops, including parts of the system that the representation does not capture at all. As those relationships become more complex and feedback becomes stronger, local improvement can increasingly diverge from systemic improvement. The representation may continue to improve even after it has begun pulling the system away from the purpose it was originally meant to describe.

I am not claiming that this is a new law, and I am not even sure it deserves to be called a rule. It is simply the most concise description I currently have for something I have repeatedly encountered. There is also nothing philosophically new about beginning with the distance between reality and our representation of it. Kant was there long before me, and Wittgenstein approached the relationship between the world and what can be said about it from another direction. Goodhart gave me a wonderfully practical formulation of one consequence. I am standing on very old ground. What is personal is the path by which I arrived there: emergency medicine taught me to trust protocols while remembering that the patient in front of me was not the protocol; public health taught me that interventions change the systems into which they are introduced, and that preventing risky behaviour is not always possible or even the most useful objective; cybersecurity taught me that adversaries do not attack our dashboards, they attack our systems; threat modelling taught me to pay attention to the distance between the two. Somewhere along that path, I began to wonder whether that distance is more important than I had previously understood.

We need representations. Without them, complex systems would be impossible to reason about. The answer cannot be to abandon metrics, models, scores or abstractions and somehow embrace reality directly. There is no direct managerial access to an organisation of fifty thousand people, just as there is no single view from which we can comprehend an economy, a society or a cloud environment in its entirety. Simplification is not our mistake. It is a condition of being able to make decisions at all. The problem is forgetting that we have simplified. Once incentives become attached to a representation, the system begins responding to that representation. People adapt, organisations reorganise, algorithms learn, resources move and behaviours change. If the pressure is strong enough, the representation can gradually stop describing the system we thought we were managing and begin participating in the creation of a different one. Perhaps that is part of what happens when phishing click rates become more important than phishing resilience, when engagement becomes more important than connection, when security scores become more important than attack paths, when quarterly revenue eclipses organisational resilience, when status becomes more important than the contribution it was supposed to represent, or when AI utilisation becomes confused with productivity.

I want to keep the word perhaps, because I am not trying to close the argument. I am trying to find out where it breaks. The examples that interest me most now are not the ones that confirm this way of looking at systems. I want the ones that contradict it. I want to know where a representation can become a target without meaningfully distorting the system around it, which systems tolerate that pressure, which do not, and what distinguishes one from another. Most of all, I want to know where this way of thinking stops explaining reality. Perhaps I have spent enough time looking at systems that fail that I have developed a habit of seeing the same failure everywhere. Perhaps I am connecting things that should remain separate. Perhaps the similarities are more aesthetic than causal.

Or perhaps there really is something here.

I would like to know which.