Controversial California bill to prevent AI disasters heads to Newsom despite pushback from Big Tech

California lawmakers passed a sweeping artificial intelligence bill, SB 1047, that seeks to establish safeguards for tech firms developing large AI models. If signed into law, SB 1047 would dramatically alter the way top AI firms operate, leading to pushback from prominent players like OpenAI, Meta ( META ), and Google ( GOOG , GOOGL ).

Governor Gavin Newsom has not indicated whether he plans to veto SB 1047.

In an interview with Yahoo Finance's Brian Sozzi on Wednesday , Salesforce ( CRM ) CEO Marc Benioff weighed in on the importance of regulation. "Looking at AI safety it has to be the highest of everybody’s interest to make sure that these AI models do not get out of control," Benioff said.

"And I think that these kinds of ideas are incredibly interesting and need to be fully evaluated. And it’s worthwhile."

Democratic state Senator Scott Wiener, who co-authored the bill , says the bill codifies voluntary safety evaluations that large AI labs, such as Meta, have already committed to. The bill requires developers spending over $100 million to build an AI model to perform safety testing before releasing it. They are required to mitigate the risks if testing finds a model has the potential to cause "catastrophic harm."

Supporters of the bill, including the 'godfathers of AI,' Yoshua Bengio and Geoffrey Hinton, say the regulation is crucial to addressing potential harms posed by AI models. They say the government has not done enough to address the threat.

“SB 1047 is a much-needed step towards advancing both safety and long-term innovation in the AI ecosystem, especially incentivizing research and development in AI safety," Bengio, a professor of computer science at the University of Montreal, told Yahoo Finance.

Tesla ( TSLA ) chief Elon Musk, who also owns xAI, voiced his support this week on X, but said it was a "tough call."

"I think California should probably pass the SB 1047 AI safety bill. For over 20 years, I have been an advocate for AI regulation, just as we regulate any product/technology that is a potential risk to the public," he wrote.

Musk has previously urged AI labs to pause the development of systems that can compete with human-level intelligence, citing potential risks to society. However, he's also been highly critical of state officials in California.

SB 1047 is co-sponsored by the Center for AI Safety Action Fund, Economic Security California Action, and Encode Justice. “It is long past due that we hold Big Tech accountable for their outsized influence and unchecked power,” Teri Olle, director of Economic Security California, said on the assembly floor.

“We need legislative solutions like SB 1047 to flip the script and ensure the public can decide how critical technology benefits society as a whole.”

A poll released this week by the Artificial Intelligence Policy Institute, a nonprofit that advocates for safe AI development, found that 7 out of 10 Californians support the bill and over 60% of respondents said they would partially blame Gov. Newsom for future AI-related incidents if he vetoes it.

Wiener's longtime ally, San Francisco Mayor London Breed, announced that she's against the bill. “I share your desire that AI develops in a safe and sustainable way, and I agree with the overall intention of your legislation,” Breed wrote in a letter to Wiener. “With additional time and collaboration, I am confident we can find a solution that addresses many of the concerns raised in recent months, while still enabling this emerging field to grow in a safe and sustainable way.”

Breed joins a growing list of California Democrats in Congress who oppose the measure, including former House Speaker Nancy Pelosi and Reps. Ro Khanna, Zoe Lofgren, Anna Eshoo, Scott Peters, Tony Cárdenas, Ami Bera, Nanette Diaz Barragan, and Lou Correa.

The bill was amended several times to assuage the fears of AI opponents, including Amazon-backed ( AMZN ) Anthropic. Some of the major amendments include removing the state regulatory body formerly called the Frontier Model Division and removing criminal penalties, while civil penalties remain.

Additionally, if an open-source developer spends less than $10 million fine-tuning an existing model, the liability would rest with the model's original developer. The attorney general's ability to seek civil penalties has been limited to situations where "a harm has occurred or there is an imminent threat to public safety."

Governor Newsom has until Sept. 30 to sign or veto the bill. When asked whether he's confident it will become law, Wiener said, "The governor has made statements that are very consistent with my own thinking — that regulation could be appropriate here and we want to make sure we’re fostering innovation as well.”

Yasmin Khorram is a Senior Reporter at Yahoo Finance. Follow Yasmin on Twitter/X @YasminKhorram and on LinkedIn . Send newsworthy tips to Yasmin: [email protected]