As AI systems become more powerful and pervasive, questions of regulation and ethics have moved from academic philosophy to urgent policy debates. San Francisco, where most advanced AI is developed, finds itself at the center of arguments about who should control AI, how it should be governed, and whether regulation will protect the public or stifle innovation.
The Light-Touch Era
For most of Silicon Valley's history, tech companies operated with minimal regulation. The internet was governed by a "hands-off" approach based on the belief that innovation moved faster than regulation, and government involvement would do more harm than good.
Section 230 of the Communications Decency Act (1996) exemplified this approach. It protected online platforms from liability for user-generated content, allowing social media companies to grow without the legal risks traditional publishers faced.
This regulatory environment helped create the tech giants. Facebook grew to billions of users without significant regulatory constraint. Google became a verb. Amazon dominated e-commerce. Apple created a mobile ecosystem.
But by the 2010s, concerns mounted: monopolistic practices, privacy violations, manipulation of elections, amplification of harmful content, and more. The light-touch era was ending.
The AI Ethics Awakening
AI-specific ethical concerns emerged in parallel. Several high-profile incidents raised alarm:
- 2016: Microsoft's Tay chatbot turned racist within 24 hours of exposure to Twitter
- 2018: Amazon's hiring AI showed bias against women
- 2019: Facial recognition systems showed racial bias
- 2020: GPT-3 demonstrated concerning capabilities for generating misinformation
- 2023: ChatGPT raised questions about academic integrity, job displacement, and information quality
These incidents, combined with growing awareness of AI's potential impact, created momentum for AI-specific regulation and ethical frameworks.
The EU Takes the Lead
Europe, historically more regulatory than the United States, moved first. The General Data Protection Regulation (GDPR), implemented in 2018, gave individuals rights over their data and imposed significant requirements on companies. Tech companies complained about compliance costs, but GDPR became a global standard.
The EU's AI Act, proposed in 2021 and finalized in 2024, took a "risk-based" approach:
- Unacceptable risk: Social scoring, manipulation, some facial recognition—banned outright
- High risk: AI in hiring, credit scoring, law enforcement—strict requirements
- Limited risk: Transparency requirements (e.g., disclosing when content is AI-generated)
- Minimal risk: Most other AI applications—no specific requirements
The Act included provisions specific to "general purpose AI" (like ChatGPT) and "foundation models," requiring transparency about training data, safety testing, and more.
The U.S. Approach: Fragmented and Hesitant
The United States lacked comprehensive federal AI regulation as of 2024. Instead, a patchwork emerged:
- Executive Orders: Biden's October 2023 AI executive order required safety testing, established AI principles, but created few enforceable requirements
- State-level regulation: California, Colorado, and others passed AI-specific laws
- Sector-specific rules: FDA guidance for AI in medicine, FTC enforcement on deceptive practices
- Voluntary commitments: Companies making public pledges about responsible AI
This fragmented approach reflected competing pressures: concerns about AI risks versus fears that regulation would hand China or Europe competitive advantages. Silicon Valley lobbied heavily against regulation that might constrain innovation.
The Self-Regulation Debate
Many in Silicon Valley argue that industry self-regulation is preferable to government intervention. Their arguments:
- Technology moves too fast for regulation to keep up
- Regulators lack technical expertise to write effective rules
- Self-regulation allows flexibility and experimentation
- Market competition naturally punishes harmful practices
Critics counter that self-regulation has failed repeatedly. Social media companies promised to address harmful content but often acted only when forced. Companies have strong incentives to prioritize growth and profit over safety. Without regulatory requirements, competitive pressure pushes toward risk-taking.
The debate intensified with AI. OpenAI initially structured as a non-profit specifically to avoid profit incentives that might compromise safety. But the transition to a capped-profit model showed the difficulty of maintaining values when competing with deep-pocketed rivals.
The Ethics Researchers' Revolt
Several high-profile departures of AI ethics researchers raised questions about whether tech companies were serious about ethics:
- 2020: Timnit Gebru fired/resigned from Google after conflicts over AI ethics research
- 2021: Margaret Mitchell, Gebru's colleague, also departed
- 2023: Multiple AI safety researchers left OpenAI citing concerns about the company's direction
These incidents suggested that ethics research within companies had limited influence when conclusions conflicted with business objectives. Can companies be trusted to regulate themselves when their own researchers face retaliation for raising concerns?
The Existential Risk Debate
A different regulatory question emerged: not whether AI might be biased or harmful, but whether it might pose existential risk to humanity. This view, associated with researchers like Stuart Russell, Yoshua Bengio, Geoffrey Hinton, and organizations like the Center for AI Safety, argues that:
- AGI development is accelerating faster than safety research
- Misaligned superintelligent AI could pose catastrophic risks
- Current approaches to AI safety are inadequate
- Regulation should slow development until safety is assured
This perspective led to open letters calling for pauses in AI development and stricter oversight. In 2023, over 1,000 researchers and tech leaders signed such a letter.
Critics dismissed this as science fiction or, more cynically, as a tactic by established companies to block competitors. The debate became contentious and political, with profound disagreements about AI's actual risks.
California's Pivotal Role
As the state where most AI development occurs, California's regulatory choices matter enormously. The California Privacy Rights Act (CPRA), similar to GDPR, set high standards. Proposed AI-specific regulations faced intense lobbying from both tech companies and civil rights organizations (with different concerns).
The challenge: regulate too aggressively and companies might relocate (though where they'd go that offers comparable ecosystem advantages isn't clear). Regulate too lightly and problems will inevitably emerge.
The Global Regulatory Race
Beyond the U.S. and EU:
- China: Heavy government oversight but focused on maintaining political control rather than safety or bias concerns
- UK: Post-Brexit, positioning as a pro-innovation alternative to EU regulation
- Canada: Proposed AI and Data Act balancing innovation with safety
- Singapore: Light-touch regulation trying to become an AI hub
This regulatory fragmentation creates challenges for global companies but also opportunities for regulatory arbitrage—locating different operations in jurisdictions with favorable rules.
The Liability Question
A crucial unsettled question: who's liable when AI causes harm? If an AI system makes a discriminatory hiring decision, provides dangerous medical advice, or causes an autonomous vehicle accident, who's responsible?
- The company that developed the AI?
- The company that deployed it?
- The individuals who trained it?
- The users who relied on it?
Without clear liability rules, both innovation and accountability suffer. Companies may be hesitant to deploy beneficial AI, while those harmed by AI systems lack clear paths to compensation.
"We're trying to regulate tomorrow's technology with today's laws, written for yesterday's problems. It's not working."
Toward Effective AI Governance
What would effective AI regulation look like? Proposals include:
- Licensing: Require safety testing before deploying powerful AI systems
- Transparency: Mandate disclosure of training data, capabilities, limitations
- Auditing: Independent third-party assessment of AI systems
- Liability: Clear rules about responsibility for AI-caused harm
- Safety research funding: Public investment in AI safety and alignment research
The challenge is designing regulation that's effective without being onerous, that protects without stifling, and that can adapt as technology evolves. It requires regulators to understand technology deeply while maintaining independence from industry capture.
The Window Is Closing
The regulatory window may be narrow. Once AI systems are deeply embedded in society—once millions of people depend on them, once economic value is created, once powerful interests are established—regulation becomes much harder. The decisions made in the next few years, largely in or influenced by San Francisco, will shape how AI is governed for decades.
Whether those decisions will be adequate to the challenge remains very much an open question.
Author's Commentary: On Karen Hao's "Empires of AI"
In the discourse surrounding AI regulation and ethics, few voices have been as consistently illuminating as that of Karen Hao, whose investigative journalism and analytical work on the political economy of AI cuts through the industry's carefully constructed narratives.
Hao's concept of "Empires of AI" provides a critical lens missing from much of mainstream AI policy discussion. Rather than treating AI regulation as merely a technical problem requiring technical solutions, she examines how AI development is fundamentally shaped by—and serves to reinforce—existing power structures. The major AI labs aren't just building technology; they're building empires with their own territories, resources, and subjects.
What makes Hao's work particularly relevant to this chapter is her analysis of how regulatory capture occurs before regulation even exists. Silicon Valley companies don't wait for regulation to appear and then lobby against it—they shape the very conversation about what regulation should address. By emphasizing existential risks and AGI timelines, they deflect attention from present harms: labor exploitation in data annotation, environmental costs of training runs, concentration of power in a handful of companies, and the extraction of value from communities whose data trains these systems.
Her reporting on the working conditions of data annotators in Kenya and other Global South countries reveals an uncomfortable truth that regulation debates often ignore: AI's supply chain depends on invisible labor, poorly compensated and psychologically damaging, far from the gleaming offices of San Francisco. Any serious ethical framework for AI must account for these global inequities.
Hao's skepticism toward "AI ethics" as practiced by major tech companies proved prescient. Her coverage of Timnit Gebru's firing from Google—and the broader pattern of ethics researchers being marginalized or forced out—exposed how ethics washing often substitutes for genuine accountability. Companies create ethics boards and hire ethics researchers not to constrain their actions but to provide legitimacy while continuing business as usual.
This framing transforms the debate. Instead of asking whether regulation will "stifle innovation," we should ask: innovation toward what end? Whose innovation? Who benefits and who bears the costs? These are political questions, not technical ones.
Hao's work on "Empires of AI" also highlights how regulatory fragmentation—the patchwork approach described earlier in this chapter—serves incumbent interests. Large companies can afford compliance teams for multiple jurisdictions; smaller competitors and non-profits cannot. Complexity becomes a moat.
For readers of this book, Hao's journalism serves as a necessary corrective to triumphalist narratives about San Francisco's AI leadership. Yes, this is where the technology is being built. But that concentration of power raises profound questions about democratic governance. Can a technology that will reshape society be governed democratically when its development is controlled by a small number of companies in a single city, operating in a country that has historically resisted technology regulation?
The empires being built in San Francisco are not just economic or technological—they are political. And empires, historically, have rarely regulated themselves in the public interest. That's the challenge facing anyone serious about AI governance: how to impose democratic accountability on institutions that have accumulated power specifically by avoiding such accountability.
Hao's work reminds us that the most important questions about AI aren't technical—they're about power, justice, and who gets to decide the future. These are the questions that effective regulation must address, even if—especially if—they make Silicon Valley uncomfortable.