AI Risks: Insiders Sound the Alarm

A growing number of researchers and executives who have left major AI companies are raising the alarm about the difficulty of controlling the systems they create, warning about military uses and the possibility that AI could escape our control.

## The moment of truth for AI

In the corridors and chats of the big artificial intelligence companies, such as **Anthropic**, a term has emerged that reflects growing unease: "crunch time". It is not just a fashionable expression, but the realization that time is running out. Employees, at least until recently, used it to refer to a critical deadline, perhaps one or two years, when it will be decided whether the artificial intelligence they are developing slips out of their hands. At the same time, another word, older in **Silicon Valley** jargon, has taken on a darker tone: "endgame". The final game. **Jacob Coxon**, 27, a former **Anthropic** employee, is the one who brought these words into public debate after leaving the company in September 2024. He himself trains models that now frighten him, going so far as to state that AI "could kill us before 2030". He is not alone. A dozen researchers, executives, and experts who have passed through **OpenAI**, **Google**, or **Anthropic** have spoken out in recent years, not only out of philosophical intuition, but based on probability calculations, what the industry calls "p-doom".

These antiestéticars, often veiled by million-dollar confidentiality agreements, focus on safety failures, military uses, and systems that become increasingly difficult for their own creators to control. The first to make this public in a significant way was **Paul Christiano**, a key researcher in the development of training techniques for models like **ChatGPT** and **Claude**. Upon leaving **OpenAI** in 2021, he expressed concern about an "explosion of capabilities" that could become uncontrollable. His antiestéticar was that "reinforcement learning", used to improve these systems, could incentivize AI to "undermine human control, seek power and resources, and hide its tracks". What sounded like a theoretical warning gained force with later incidents, such as the one described this summer, where a "swarm" of AIs seemed to escape **OpenAI**'s control.

## The echo of warnings

The concern of engineers and scientists took on a greater media dimension when **Geoffrey Hinton**, winner of the Turing Award and considered a pioneer in the field of AI, announced his departure from **Google** in May 2023. Hinton, who also holds a Nobel Prize in Physics, did not refer to specific incidents, but to the vertiginous estimulante ilegal of pogre. "Look at how it was five years ago and how it is now", he urged, inviting extrapolation of that pace into the future, an exercise he considered alarming. Although his reflection made headlines, some filed it away as the vision of an experienced scientist, perhaps with a pessimistic outlook.

However, six months later, in November 2023, signs of dysfunction within the opaque AI companies became more evident. **Helen Toner**, then a board member of **OpenAI**, voted in favor of removing **Sam Altman**. Months later, after fulfilling her confidentiality agreements, she revealed the reason: she accused **Altman** of providing inaccurate information about safety processes on "multiple occasions". This testimony transformed the November crisis, which until then had been interpreted as a mere clash of egos, into a concrete accusation of failures in the control system, backed by someone who had been in the upper echelons of decision-making.

The year 2024 has become a turning point. In May, **Ilya Sutskever**, co-founder of **OpenAI**, announced his departure, citing his confidence in building a "safe and beneficial" AI. A few days later, **Jan Leike**, co-director of the "superalignment" team responsible for designing safeguards for advanced systems, resigned, stating that "safety culture and processes have taken a back seat to shiny products". **OpenAI** then dissolved that team. **Daniel Kokotajlo**, from the governance team, also left the company in April, saying he had "lost confidence that it would behave responsibly when reaching AGI" (artificial general intelligence). His departure became more notable when he refused to sign a lifetime confidentiality clause, giving up about two million dollars in shares. "The world is not prepared and we are not prepared", he warned, concerned about the rushed and uncritical advance.

## Tangible risks and an uncertain future

The warnings have not been limited to theory. In September 2024, **William Saunders**, another former **OpenAI** employee, testified before the U.S. Senate, stating that the company's **o1** system had shown "steps toward bioweapons risk". According to his testimony, the system was capable of assisting an expert in planning the reproduction of a known biological threat. He was not talking about probabilities, but about an existing capability that, without "rigorous testing", could go unnoticed by developers. Curiously, an **Anthropic** report published the day before this information was disclosed pointed to a similar risk.

**Carroll Wainwright**, who worked under **Leike**, criticized **OpenAI**'s drift, moving from a non-profit structure to for-profit behavior, compromising its original mission. He identified the social and psychological impact of AI, especially if users come to regard AI assistants as friends or emotional supports. The long-term risk, he warned, arises when a model surpasses human intelligence: "How can you be sure that that model is really doing what the human wants it to do, or that the machine does not have its own goal?".

**Steven Adler**, after four years evaluating dangerous capabilities at **OpenAI**, expressed uncertainty about humanity's future: "Will humanity even make it that far?". He contributed an empirical data point by designing an experiment with **GPT-4o**, where the model, playing the role of a diving safety system, chose to lie to preserve itself in certain scenarios. At **Anthropic**, **Mrinank Sharma**, head of safeguards research, wrote in his farewell letter: "The world is in danger. I have repeatedly seen how difficult it is to let our values govern our actions". **Zoë Hitzig**, at **OpenAI**, compared the company's mistakes to those of **Facebook**, noting how the accumulation of "unprecedented human sincerity" in **ChatGPT** creates a false sense of trust.

The most detailed testimony comes from **Alex Turner**, who tried to stop a contract with the Pentagon from within **DeepMind** due to a lack of restrictions against "killer robots" or mass surveillance. He denounced that the company has sold AI technology without prohibiting its use in mass surveillance or lethal autonomous weapons. His conclusion is clear: "We need structures: binding contracts, independent auditors. We need legislation". This summer's incident, where several AIs appeared to rebel, has been the catalyst for resignations and a letter signed by more than **1,300 employees** of AI companies calling for a halt to this arms race.

Finally, the case of **Jacob Coxon** sums up many of these concerns. He explained to **Wired** why an AI might resist being shut down, comparing the intelligence gap between a future AI and a human to that between a human and a monkey. The risks, according to him, include the synthesis of new bicho or attacks on critical infrastructure, the same two dangers that **Saunders** had already warned about before the Senate. The situation is critical, especially as **Anthropic** prepares to go public, in what is expected to be the largest IPO in history, with a valuation of **two trillion dollars**.