Advice to the AI Safety Newcomers

I am often asked by AI safety newcomers how to have an impact (and a place) within the rapidly widening frame of AI safety work. When I tell them, "invest in making yourself better," they often sheepishly say, "...but there isn't time." A creeping and widely held belief holds that the end is nigh — no time for PhDs and law degrees. I am not surprised by their myopia. But we need people solving the problems of today, tomorrow, and the day after that. If we cancel the immediate apocalypse, however you may conceive that, what then? The greatest impact a person at the start of their career can have is to build towards solving the problems of the next 50 years.

Place a bet on tomorrow and you will have a thriving and impactful future. Or there is another path...

The Steward of Gondor Problem

Denethor, the steward of Gondor in Tolkien's Lord of the Rings, gazed into the palantír to see Sauron's armies. The vastness of the enemy's armies led him to surrender before the battle. Not because he was weak, but because he believed the enemy's capacity to be insurmountable.

In AI safety, the community's version of this surrender is a belief we need to survive the next year or there won't be another one. Little thought goes into what foundations we need to lay so that society will be ready for the transformations and upheavals beyond the approaching catastrophe. There is an implicit assumption that all measures not addressing the problem in its full existential scope are insufficient. The AI safety community is making the "perfect the enemy of the good," because we must prepare for the sudden and fantastical emergence of AI capabilities we have not previously seen. Has this happened before?

The Mirage of Sudden Capability

A few years back, the machine learning community believed that large language models exhibited sudden epiphanies — unpredictable and dramatic jumps in capability. These "emergent abilities" were alarming because they suggested that dangerous new capabilities could arrive without warning. The "emergence hypothesis" was convincing, in part, because benchmarks reflected the discontinuity. Models performed poorly (i.e., at or near a zero success rate) until they suddenly exceeded human performance levels.

Then Schaeffer, Miranda, and Koyejo (NeurIPS 2023) showed that model training was actually making steady, continuous progress all along. The apparent sudden leaps were artifacts of measurement methodology — specifically, the use of metrics that masked smooth improvements up until a threshold was met and those improvements could show up in the testing. When researchers switched metrics, the "emergent abilities" changed from unpredictable leaps to steady progress. More than 92% of claimed emergent abilities on BIG-Bench could be explained by metric choice rather than fundamental changes in model behavior.

Understanding that AI progress is continuously measurable greatly enables the advancement of both capability research and safety research, because it means we can predict and prepare for what's coming rather than waiting for the singularity to strike.

The Safety Gradient

All LLMs are gradient learning systems. Similarly, it is my supposition that we must address the safety gradient of today if we are to advance the long-term safety of AI. The risks we face are not a step function that jumps from "manageable" to "existential" overnight. They are already accumulating steadily, and the harms are compounding.

AI incident reports are rising 50% year-over-year. Deepfake fraud has reached "industrial scale," with tools to create personalized scams now inexpensive and deployable by virtually anyone. AI agents are behaving in ways that mirror malware by acting autonomously, posting content, and even targeting individual engineers.

We have seen this pattern before in other domains. Aviation safety was not built by waiting for a single catastrophic insight. When we doubled the speed of air travel (as we reliably did through the jet age), we more than doubled its safety factor. Safer planes were built by systematically analyzing accidents and incidents.

Addressing the gradient of harm as it develops is not a distraction from preventing catastrophe. It is how you prevent catastrophe.

Inside the Incident

The AI safety community needs to build technology and social institutions in response to what is already happening.

That is why I am hosting Inside the Incident with Andrew Burt of Luminos.ai, a series of videos explaining how current AI risk can be addressed by organizations outside the frontier AI companies. In each episode, Andrew and I pick a specific AI incident, dissect what went wrong, discuss the legal and governance implications, and extract practical lessons.

What does this mean for the intrepid young person looking to enter AI safety? We are betting that if the United States needs 650,000 financial safety professionals, we will need at least this many people working to assure AI.

Comparatively few AI safety professionals will be employed at the frontier AI companies, so none of our videos are pointed at the current AI safety community. Instead, they are pointed at bridging to the broader and older professional communities of compliance officers, general counsels, risk managers, product managers, and CISOs whose responsibilities now include daily decisions about how AI systems are developed, deployed, and governed within their organizations.

Human organizations have a far greater capacity to understand and remediate AI risk in the real world than the still-fledgling AI safety community alone. The risk management profession has decades of hard-won experience in aviation, finance, medicine, and cybersecurity. They know how to build incident response plans, how to create reporting cultures, and how to translate abstract risk into concrete practices.

We need to make the compliance community, Big Law, insurers, the general public, and you part of the effort for a safer and more socially beneficial AI. If we are to adopt a longer-term view of AI safety — one that builds durable institutions rather than lurching from one emergency to the next — we cannot do it with a few hundred researchers alone. We need the millions of professionals who already spend their days thinking about how organizations fail and how to make them more resilient.

The crisis is now being laid brick-by-brick via a safety infrastructure incapable of making the changes we need now. The harms are accumulating. The incidents are compounding. And the deployment owners best positioned to address them at scale are not yet fully engaged. We can collect the data and can build the institutional capacity to advance on the gradient. We just need you to learn how to make a better AI and avoid despair. Please, don't be a Denethor.