A California bill that would establish first-in-the-nation safeguards for the world’s largest artificial intelligence systems passed a landmark vote Wednesday. The proposal, which aims to reduce potential risks created by AI, would require companies to test their models and publicly disclose their safety protocols to prevent the models from being manipulated to, for example, destroy the state’s power grid or help build chemical weapons — scenarios that experts say could be possible in the future with such rapid advances in the industry.
The measure passed the state Assembly on Wednesday and now faces a final vote in the state Senate, where it has already passed once, before heading to the governor’s desk for his signature, though he has not indicated his position on it. Gov. Gavin Newsom then has until the end of September to decide whether to sign it into law, veto it or allow it to become law without his signature. He declined to weigh in on the measure earlier this summer but had warned against overregulating artificial intelligence.
Supporters said the bill would establish some of the first much-needed safety standards for large-scale AI models in the United States. The bill targets systems that require more than $100 million in data to train. No current AI model has met that threshold.
The proposal, authored by Democratic Sen. Scott Wiener, faced fierce opposition from venture capital firms and tech companies including OpenAI, Google and Meta, the parent company of Facebook and Instagram. They say safety standards should be set by the federal government and that the California legislation targets developers rather than those who use and exploit AI systems to cause harm.
Wiener said his legislation took a “soft” approach.
“Innovation and safety can go hand in hand, and California is leading the way,” he said in a statement after the vote.
Wiener’s proposal is among dozens of artificial intelligence bills that California lawmakers have proposed this year to build public trust, combat algorithmic discrimination and ban fake news involving elections or pornography. At a time when artificial intelligence increasingly affects Americans’ daily lives, state lawmakers have sought to strike a balance between reining in the technology and its potential risks without stifling the burgeoning local industry.
California, home to 35 of the world’s top 50 AI companies, has been an early adopter of AI technologies and could soon deploy generative AI tools to address road congestion and traffic safety, among other things.
Elon Musk, owner of X (formerly Twitter) and founder of xAI, backed the proposal this week, though he said it was a “tough decision.” X operates its own chatbot and image generator, Grok, which has fewer protections than other prominent AI models.
“For over 20 years, I have advocated for regulating AI, just as we regulate any product or technology that poses a potential risk to the public,” Musk tweeted.
A group of several California House members also opposed the bill, with former House Speaker Nancy Pelosi calling it “well-intentioned but ill-informed.”
The Chamber of Progress, a left-leaning industry group funded by Silicon Valley, said the bill is “based on science fiction fantasies about what AI could be like.”
“This bill has more in common with Blade Runner or Terminator than the real world,” technology policy director Todd O’Boyle said in a statement after Wednesday’s vote. “We shouldn’t be crippling California’s core economic sector for a theoretical scenario.”
The legislation also has the support of Anthropic, an artificial intelligence startup backed by Amazon and Google, after Wiener tweaked the bill earlier this month to include some of the company’s suggestions. The current bill removed a provision on the penalty for perjury, limited the state attorney general’s power to sue offenders and reduced the responsibilities of a new AI regulatory agency.
Anthropic said in a letter to Newsom that the bill is crucial to preventing catastrophic misuse of powerful artificial intelligence systems and that “its benefits likely outweigh its costs.”
He also earlier this week critics for dismissing the potential catastrophic risks of powerful AI models as unrealistic: “If they genuinely believe the risks are false, then the bill should present no problem.”