AI Crackdown in Silicon Valley: Developers Left Speechless by Sweeping New Regulations

The proposed AI policy highlights several key concerns regarding the potential risks of advanced AI systems:

1. Existential and Catastrophic Risks: The policy defines a "major security risk" as including any existential or global catastrophic risks that could harm everyone on the planet, such as AI systems establishing self-replicating autonomous agents or permanently escaping human control.

2. Ability Benchmarks vs. Compute Benchmarks: The policy suggests using compute-based benchmarks (e.g., FLOPS) to categorize AI systems into tiers of concern. However, the proposal acknowledges that this approach may not accurately reflect the true capabilities of AI systems, as efficiency improvements can lead to more capable models with less compute.

3. Stopping Early Training: The policy proposes that medium-concern AI systems must undergo regular performance testing, and training must be stopped if the system exhibits unexpectedly high capabilities. This could slow down the development of advanced AI systems.

4. Determining Extremely High-Concern AI: The policy requires the development of standards to identify AI systems that could assist with weapons development, destabilize global power, or pose other catastrophic risks. Defining these standards within a 12-month timeframe may be challenging given the rapid pace of AI advancement.

5. Hardware Monitoring: The policy includes a requirement to report the purchase, sale, and use of high-performance hardware (e.g., GPUs) used for AI development, potentially leading to increased government oversight and restrictions on access to such hardware.

6. Foreseeability of Harm: The policy states that developers cannot use the "surprise" of an AI system's unreliability as a valid defense, as they should have known that frontier AI systems pose a wide variety of severe risks, some of which may not be detectable in advance.

7. Emergency Powers: The policy grants the president and an administrator the ability to declare a state of emergency and impose sweeping powers, including the destruction of AI hardware, deletion of model weights, and physical seizure of AI laboratories, in response to perceived major security risks.

8. Whistleblower Protection: The policy provides protection for whistleblowers who report or refuse to participate in practices forbidden by the AI Act, even if their beliefs about the violations are ultimately incorrect.

Overall, the proposed policy reflects the growing concerns about the potential risks of advanced AI systems and the desire to establish a regulatory framework to mitigate these risks. However, the policy also raises questions about the feasibility and potential unintended consequences of such a comprehensive and far-reaching approach to AI governance.

The proposed AI policy defines four tiers of AI systems based on their computational requirements:

• Tier 1 (Low Concern AI): AI systems trained on less than 10^24 FLOPs
• Tier 2 (Medium Concern AI): AI systems trained on 10^24 to 10^26 FLOPs
• Tier 3 (High Concern AI): AI systems trained on more than 10^26 FLOPs
• Tier 4 (Extremely High Concern AI): AI systems with capabilities that could pose catastrophic risks

However, this approach of using FLOP-based thresholds to categorize AI systems is problematic. As noted, compute power (FLOPs) does not directly translate to the capabilities of an AI system. Factors such as model architecture, training data, and optimization techniques can significantly impact the abilities of an AI system, independent of its computational requirements.

The policy proposal fails to account for the rapid advancements in AI efficiency, where smaller and more compact models can achieve comparable or even superior performance compared to larger, more computationally intensive models. Examples like LLaMA and Flan-T5 demonstrate that compute-efficient AI systems can possess significant capabilities.

Regulating AI based solely on FLOP thresholds risks creating an inflexible and potentially ineffective framework. Instead, the policy should focus on developing robust and adaptable benchmarks that directly assess the capabilities of AI systems, including their potential risks and safety considerations. This would provide a more accurate and future-proof approach to categorizing and governing AI technologies.

The proposed policy includes several concerning provisions regarding the regulation and monitoring of AI development:

• It defines a "major security risk" as including any existential or catastrophic risks, threats to critical infrastructure, national security, or public safety, as well as the risk of AI systems establishing self-replicating autonomous agents or permanently escaping human control.

• It establishes a tiered system for classifying AI systems based on their computed flops (floating-point operations per second), with the most powerful systems (>10^26 flops) considered "extremely high concern." This approach is flawed, as compute power alone does not determine an AI system's capabilities or risks.

• For "medium concern" AI systems, the policy would require monthly performance testing and reporting to the government, with training required to stop if the system exhibits "unexpectedly high" performance. This could incentivize underreporting or gaming the system.

• The policy grants the government broad emergency powers, including the ability to destroy AI hardware, delete model weights, and physically seize AI labs - even if the specific risks were "surprises" to the developers. This could have a severe chilling effect on AI research and development.

• The policy includes whistleblower protections, which is positive, but even if a whistleblower is mistaken about a violation, they would still be protected from retaliation.

Overall, this policy appears to be an overly heavy-handed and potentially counterproductive approach to AI regulation. While the intent to address existential risks is understandable, the specific mechanisms proposed could significantly hamper beneficial AI progress and innovation.

The proposed policy outlines strict safety requirements for "high-concern" AI systems, defined as those trained on more than 10^26 FLOPS. Key points include:

• Developers must provide conclusive evidence that the AI system poses no significant possibility of catastrophic risks, such as the ability to assist with WMD development, autonomous spread, or destabilization of global power dynamics.

• This evidence must go beyond simply proving the system is not currently dangerous - the burden is on developers to rule out any significant future risks.

• Permits to develop high-concern AI will typically be approved within 90 days, if at all, imposing a slow and uncertain process.

• The policy grants the government broad emergency powers, including the ability to destroy hardware, delete model weights, and physically seize AI labs to prevent further development.

• Whistleblowers are protected even if their concerns about AI safety violations turn out to be incorrect, as long as they had a reasonable good-faith belief.

Overall, this policy represents an extremely rigorous and restrictive approach to regulating advanced AI systems, with the government maintaining tight control and the ability to shut down development at its discretion. The high bar for proving safety and the threat of severe interventions could significantly slow progress in this domain.

The proposal includes a provision for a "fast track" exemption form that allows AI developers who are not posing any major security risks to carry on with their work, even if their AI systems technically qualify as "frontier AI".

The administration is ordered to design a two-page form that will let AI tools like self-driving cars, fraud detection systems, and recommendation engines continue operating without having to participate in the rest of the regulatory framework described in the proposal.

This exemption for narrow AI applications seems to be a reasonable approach, recognizing that many AI-powered systems pose minimal risks and should not be overly burdened by the same regulations intended for more powerful and potentially dangerous AI systems.

The proposed AI policy includes several concerning provisions that grant the government broad authority to regulate and restrict AI development:

• The policy defines "major security risks" very broadly, including any existential or catastrophic risks, threats to critical infrastructure, and risks of AI systems escaping human control. This vague definition could be used to justify heavy-handed intervention.

• The policy establishes a tiered system to categorize AI systems based on their computational power, with higher tiers facing more stringent regulations. However, this approach fails to account for advances in AI efficiency that could allow powerful capabilities at lower computational thresholds.

• The policy requires monthly performance reporting and automatic halting of "medium concern" AI training if results exceed expectations. This could significantly slow AI progress and innovation.

• The policy grants the government emergency powers to suspend AI permits, issue restraining orders, delete model weights, and even physically seize AI labs - all without clear criteria or oversight. This authoritarian approach risks stifling the field.

• The policy provides whistleblower protections, but even if a whistleblower is mistaken, they would still be protected from retaliation. This could incentivize abuse of the system.

Overall, this policy appears to take an excessively restrictive and heavy-handed approach to AI regulation, risking significant harm to technological progress and innovation in the field. A more balanced, evidence-based, and collaborative approach would be preferable.

Anyone who speaks out against, reports, or refuses to participate in any practice forbidden by the AI Act can qualify as a whistleblower. Even if a whistleblower is wrong, they are still protected as long as they had a reasonable good faith belief that the AI Act was being violated.