OpenAI and Anthropic have each signed unprecedented agreements that give the U.S. government early access to the two companies' most spectacular new AI models to conduct security testing before they are released to the public.
According to a press release from the National Institute of Standards and Technology (NIST), the agreement creates a “formal collaboration on AI safety research, testing, and assessment with both Anthropic and OpenAI” and the U.S. Artificial Intelligence Safety Institute.
The deal will give the US AI Safety Institute “access to important new models from each company before and after their release.” This will ensure that public safety does not depend solely on how the companies “assess capabilities and security risks and methods to mitigate those risks,” NIST said, but also on joint research with the US government.
The US AI Safety Institute will also collaborate with the UK AI Safety Institute to study models to detect potential safety risks. Both groups will provide feedback to OpenAI and Anthropic “on potential safety improvements to their models”.
NIST said the agreements also build on voluntary AI security commitments AI companies have made to the Biden administration to evaluate risk detection models.
Elizabeth Kelly, director of the US AI Safety Institute, called the agreements “an important milestone” to “help shape the future of AI responsibly.”
Anthropic co-founder: AI safety ‘critical’ for innovation
The announcement comes as California is on the verge of passing one of the nation's first AI safety laws that will regulate the development and deployment of AI in the state.
Among the most controversial aspects of the law is the requirement that AI companies build in a “kill switch” to prevent their models from posing “novel threats to public safety,” especially if the model operates “with limited human oversight, interference, or monitoring.”
Critics say the bill overlooks existing security risks posed by AI – such as deepfakes and election fraud – in favor of doomsday prevention. This could hinder AI innovation and provide little security today. They have called on California Governor Gavin Newsom to block the bill if it lands on his desk, but it is still unclear whether Newsom intends to sign it.
Reuters reported that Anthropic was among the AI companies that cautiously supported California's controversial AI bill, arguing that the potential benefits of the regulation likely outweigh the costs after a late round of changes.
The company's CEO, Dario Amodei, explained in a letter to Newsom last week why Anthropic now supports the bill, Reuters reported. He wrote that while Anthropic isn't sure which aspects of the bill “seem concerning or ambiguous,” Anthropic's “initial concerns that the bill could potentially hamper innovation due to the rapidly evolving nature of the field have been greatly reduced by the recent changes to the bill.”
OpenAI has notably joined critics of California's AI safety law and has been criticized by whistleblowers for lobbying against it.
In a letter to the bill's co-signer, California Senator Scott Wiener, Jason Kwon, chief strategy officer at OpenAI, suggested that “the federal government should take a leadership role in regulating breakthrough AI models to address the implications for national security and competitiveness.”
The ChatGPT maker, which signed a contract with the US AI Safety Institute, seems to agree with this line of thinking. As Kwon told Reuters: “We believe the institute has a critical role to play in defining US leadership in the responsible development of artificial intelligence and hope our collaboration will provide a framework for the rest of the world to build upon.”
While some critics fear California's AI safety law will hamper innovation, Anthropic co-founder Jack Clark told Reuters today that “safe, trustworthy AI is critical to the technology's positive impact.” He confirmed that Anthropic's “collaboration with the US AI Safety Institute” will leverage the government's “extensive expertise to rigorously test Anthropic's models before their widespread adoption.”
In the NIST press release, Kelly agreed that “security is critical to driving breakthrough technological innovation.”
By working directly with OpenAI and Anthropic, the U.S.-based AI Safety Institute also plans to conduct its own research to “advance the science of AI safety,” Kelly said.