AI safety may lag behind $7 trillion race for market domination

Industry insiders tell Axios that the competition for artificial intelligence supremacy among labs, firms and nations is likely irreconcilable with the drive for AI safety. The core conflict stems from the massive financial incentives to advance the technology, which critics argue will consistently outweigh efforts to engineer a slowdown.

While chief executive officers promote self-regulation and third-party assessments to the public, the models themselves are becoming less transparent regarding their reasoning processes. The intricate nature of these systems, coupled with the use of thousands of semi-autonomous agents, renders real-time oversight extremely challenging. Independent safety organizations face a credibility deficit because of perceived close relationships between the industry and certain evaluators, as well as overlapping investor bases within the AI research sector.

The financial reward for rapid development far exceeds the benefits of prioritizing safety. OpenAI and Anthropic are preparing to go public at valuations in the multi-trillions, creating pressure to release progressively superior models to justify those prices. Goldman Sachs estimates that more than $7 trillion will be invested in scaling AI over the next five years. To recoup this capital, frontier labs must outperform rivals, including preventing customers from switching to open-weight providers, which are often cheaper and considered safer by some.

Daniel Kokotajlo, a former OpenAI researcher and the executive director of the AI Futures Project, stated he does not believe AI can be scaled safely. His team authored the AI 2027 and 2040 essays. Kokotajlo suggested that top companies should refrain from using their extensive computing resources for model improvement, a move that would force a deceleration. He described researchers and employees involved in scaling as "doing something wrong." Having left OpenAI in 2024 due to safety concerns, Kokotajlo has since spoken to current employees about his work.

He characterized the internal culture at these firms as one of continuous conflict and debate about their actions, asking if they should stop and whether they are on the right side. Conversely, many others, including President Donald Trump, view a safe future for AI as attainable and highlight potential civil or criminal liability for unsafe products. Trump posted on Truth Social that "Whoever wins AI, WINS!" and reiterated that he would not hinder or slow down AI development.

Akshay Krishnaswamy, chief architect at Palantir, told Axios that human determination is sufficient to figure out how to scale these models safely. He noted that open models represent one path to achieving this. As the industry embraces rhetoric about slowing down, it remains unclear whether such a pause is achievable in the near term.