AI

The Scaling Laws Hypothesis: Why AI Progress Might Be More Predictable Than We Think

A technical exploration of neural scaling laws, their implications for AI development, and why understanding power-law relationships might be the key to predicting—and shaping—AI's future.

April 15, 2025
24 min read

The Scaling Laws Hypothesis: Why AI Progress Might Be More Predictable Than We Think

In January 2020, OpenAI published a paper that changed how AI researchers think about progress: "Scaling Laws for Neural Language Models." The findings were startling—model performance follows predictable power laws based on three variables: model size, dataset size, and compute budget.

This wasn't just an academic curiosity. It was a roadmap. And it's why GPT-4, Claude 3, and Gemini exist today.

But the implications go far beyond "bigger models work better." Scaling laws suggest that AI progress isn't random innovation—it's systematic optimization of a mathematical relationship. Understanding these laws reveals where AI is headed, what the limits might be, and why the race for compute has become the defining competition of this decade.

What Are Scaling Laws? The Power-Law Relationship

The Core Discovery

The Finding: Model performance (measured by loss) scales as a power law with:

1. Model Parameters (N): Number of weights in the neural network 2. Dataset Size (D): Number of training tokens 3. Compute Budget (C): FLOPs spent on training

The Mathematical Relationship:

L(N) ≈ (Nc/N)^αN (loss scales with model size) L(D) ≈ (Dc/D)^αD (loss scales with dataset size) L(C) ≈ (Cc/C)^αC (loss scales with compute)

What This Means:

  • Double your parameters: predictable performance improvement
  • Double your data: predictable performance improvement
  • Double your compute: predictable performance improvement

    • The Controversy: This suggests AI capabilities are computable—not dependent on architectural breakthroughs, but on resource allocation.

    Why Power Laws Matter

    Power laws are everywhere in nature:

  • Earthquake magnitudes
  • City populations
  • Word frequencies (Zipf's Law)
  • Brain connectivity
  • Market movements

    • The Implication: When a phenomenon follows a power law, it's usually because of a fundamental underlying process, not random chance.

    For AI: The power-law relationship suggests that neural networks are tapping into something fundamental about learning and intelligence—not just clever engineering tricks.

    The Bitter Lesson, Revisited

    Rich Sutton's "Bitter Lesson" (2019) argued that general methods that leverage computation are ultimately most effective.

    Translation: Throwing more compute at problems works better than clever algorithms.

    The AI Community's Reaction: Initially skeptical. "Surely there's more to intelligence than scaling?"

    The Evidence: Scaling laws vindicated Sutton. GPT-3 (2020) worked better than GPT-2 (2019) primarily because it was bigger—175B vs 1.5B parameters.

    Why This Is "Bitter"

    Researchers spend careers developing sophisticated architectures:

  • Attention mechanisms
  • Mixture of Experts
  • Constitutional AI
  • Chain-of-thought prompting

    • The Bitter Truth: Most improvements come from scale, not cleverness. A larger, simpler model often outperforms a smaller, sophisticated one.

    The Implication for Startups: You can't out-algorithm OpenAI or Anthropic. You need compute—or you need a different strategy entirely.

    The Three Scaling Dimensions

    1. Parameter Scaling: Size Matters

    The Relationship: Loss decreases as N^(-0.076)

    What This Means: Every 10x increase in parameters reduces loss by ~40%

    The History:

  • GPT-1 (2018): 117M parameters
  • GPT-2 (2019): 1.5B parameters (13x increase)
  • GPT-3 (2020): 175B parameters (117x increase)
  • GPT-4 (2023): ~1.8T parameters (est., 10x increase)

    • The Pattern: Each generation is 10-100x larger.

    The Problem: Parameters require:

  • More memory (1.8T parameters ≈ 7TB in FP16)
  • More compute (inference cost scales with parameters)
  • More engineering (distributed training at scale is hard)

    • The Solution: Mixture of Experts (MoE)—only activate subset of parameters per token, reducing compute while maintaining capacity.

    2. Data Scaling: The Internet Bottleneck

    The Relationship: Loss decreases as D^(-0.095)

    What This Means: Every 10x increase in training data reduces loss by ~45%

    The History:

  • GPT-2 (2019): 40GB text (~10B tokens)
  • GPT-3 (2020): 570GB text (~300B tokens)
  • PaLM (2022): 780B tokens
  • Llama 2 (2023): 2T tokens
  • Llama 3 (2024): 15T tokens

    • The Critical Insight: We're running out of human-generated text on the internet.

    Epoch AI's Estimate: We'll exhaust high-quality text data by 2026-2027.

    The Implications:

    1. Synthetic Data: Models generate training data for themselves - Risk: Model collapse (errors compound over generations) - Solution: Careful curation and verification

    2. Multimodal Data: Video, audio, images contain more information than text - YouTube: 500 hours uploaded per minute - Problem: Harder to process than text

    3. Interactive Data: Reinforcement learning from human feedback (RLHF) - Higher quality than passive data - More expensive to generate

    3. Compute Scaling: The Trillion-Dollar Race

    The Relationship: Loss decreases as C^(-0.050)

    What This Means: Every 10x increase in compute reduces loss by ~30%

    The Math:

  • GPT-3 training: ~3.14e23 FLOPs (~$5M on optimized hardware)
  • GPT-4 training: ~2.15e25 FLOPs (est., ~$100M)
  • Next-gen models: ~1e27 FLOPs (est., ~$1B+)

    • The Hardware Reality:

    2020:

  • NVIDIA A100: 312 TFLOPS (FP16)
  • Cost: ~$15,000
  • Training GPT-3: ~10,000 GPU-days

    • 2024:

  • NVIDIA H100: 1,979 TFLOPS (FP16)
  • Cost: ~$30,000
  • Training GPT-4: ~100,000 GPU-days (est.)

    • 2025+:

  • NVIDIA B100: ~4,000 TFLOPS (est.)
  • Training next-gen: ~1,000,000 GPU-days

    • The Arms Race:

  • Google: 1M+ TPUs
  • Microsoft/OpenAI: 100,000+ H100s
  • Meta: 350,000 H100s by year-end
  • Amazon: Custom Trainium chips
  • Anthropic: Partnership with AWS

    • The Constraint: Not just money—physical constraints:

  • Power: Training clusters need 100+ MW (small city's worth)
  • Cooling: Liquid cooling systems required
  • Networking: Specialized interconnects (InfiniBand, NVLink)
  • Space: Warehouse-sized datacenters

    • The Chinchilla Scaling Laws: Optimal Allocation

    The Revision

    Original Assumption (GPT-3 era): More parameters = better performance

    Chinchilla Discovery (2022): GPT-3 was undertrained.

    The Optimal Ratio: For a given compute budget, parameters and data should scale equally.

    Specifically: For every doubling of parameters, double the training data.

    The Implication: GPT-3 (175B params, 300B tokens) should have been trained on 3.5T tokens.

    The Post-Chinchilla World

    Llama 2 (2023): 70B params, 2T tokens

  • Follows Chinchilla scaling more closely
  • Outperforms much larger models

    • Llama 3 (2024): 70B params, 15T tokens

  • Dramatically over-trains relative to Chinchilla
  • Why? Data quality and diversity matter more than quantity

    • The New Understanding: Chinchilla is a guideline, not a law. Over-training on diverse, high-quality data can yield better results.

    The Emergent Abilities Puzzle

    The Phenomenon

    As models scale, they suddenly develop capabilities they weren't explicitly trained for:

    Examples:

  • Few-shot learning (GPT-3)
  • Chain-of-thought reasoning (scale threshold ~60B params)
  • Theory of mind (understanding others' mental states)
  • Tool use (calling APIs, writing code)
  • Multilingual translation (never trained on parallel corpora)

    • The Mystery: Why do these abilities emerge discontinuously? Why not gradual improvement?

    Two Hypotheses

    Hypothesis 1: Phase Transitions

    Like water turning to ice, models cross capability thresholds.

    Evidence:

  • Sharp performance jumps on specific tasks
  • Happens at predictable parameter counts
  • Suggests fundamental computational limits

    • Implication: Some capabilities require minimum scale—no amount of architecture cleverness can bypass this.

    Hypothesis 2: Evaluation Artifacts

    The abilities were always there, just hard to measure at small scale.

    Evidence:

  • Different evaluation metrics show different emergence points
  • Continuous improvement on some metrics, discontinuous on others
  • May be measurement problem, not capability problem

    • Implication: We might be overestimating the "magic" of scale.

    My Take: Probably both. Some capabilities genuinely require minimum compute (phase transitions). Others are gradual but appear sudden due to measurement (artifacts).

    The Limits: Where Do Scaling Laws Break Down?

    Physical Limits

    Compute Wall (2030-2035?):

    At current scaling rates:

  • 2025: $1B training runs
  • 2027: $10B training runs
  • 2030: $100B training runs
  • 2032: $1T training runs (5% of US GDP)

    • At some point, throwing more money at compute becomes economically unfeasible.

    Data Wall (2026-2028?):

    If we exhaust internet text data by 2027, what then?

    Options: 1. Synthetic data (risky—model collapse) 2. Multimodal scaling (video, audio—way more data) 3. Interactive learning (expensive but higher quality) 4. Reasoning compute (use inference-time compute, not just training compute)

    Energy Wall (2028-2030?):

    Training GPT-4: ~50 GWh Next-gen: ~500 GWh (annual output of small power plant) 10x larger: ~5 TWh (0.1% of global electricity)

    At some point, energy consumption becomes the limiting factor.

    Algorithmic Limits

    The Optimist Case: New architectures break through scaling law ceiling

    Examples:

  • Transformer replaced RNNs (2017)
  • Mixture of Experts (2024)
  • State Space Models (Mamba, 2024)
  • Test-time compute scaling (o1, 2024)

    • The Pessimist Case: We're near the limits of what scaling can achieve

    Evidence:

  • GPT-4 to GPT-5 improvement smaller than GPT-3 to GPT-4
  • Diminishing returns on benchmark performance
  • Compute costs growing faster than capabilities

    • Intelligence Limits

    The Hard Question: Do scaling laws lead to AGI (Artificial General Intelligence)?

    The Scaling Optimist Position:

  • Current models are general-purpose (can do many tasks)
  • Each scale jump adds new capabilities
  • Sufficient scale = human-level (or beyond)
  • Timeline: 2027-2030

    • The Scaling Skeptic Position:

  • Current models lack true reasoning (they pattern-match)
  • Scaling won't give consciousness or agency
  • Missing pieces: world models, causal reasoning, persistent memory
  • Timeline: 2035+ or never

    • The Data:

  • GPT-4 fails at novel reasoning tasks
  • Can't reliably count letters in words
  • Struggles with basic logic puzzles
  • Hallucinates confidently

    • The Counter-Data:

  • GPT-4 passes the bar exam (90th percentile)
  • Writes production code
  • Translates between 100+ languages
  • Explains complex scientific concepts

    • My Take: Scaling will get us surprisingly far (maybe 80-90% of human cognitive tasks), but the last 10-20% might require fundamental breakthroughs.

    The Strategic Implications: Winner-Take-Most Dynamics

    Why Scale Creates Moats

    1. Compute Access

    The companies with most compute can train the best models:

  • OpenAI/Microsoft
  • Google DeepMind
  • Anthropic/Amazon
  • Meta

    • Barrier to Entry: Securing 100,000+ GPUs requires:

  • Billions in capital
  • Contracts with NVIDIA (limited supply)
  • Datacenter infrastructure
  • Technical expertise

    • Result: Oligopoly in frontier models.

    2. Data Network Effects

    Better models attract more users → more interaction data → better fine-tuning → better models

    OpenAI's Advantage:

  • ChatGPT has 100M+ weekly active users
  • Every interaction generates training data
  • RLHF feedback loop improves model continuously

    • The Catch-22: New entrants need users to get data, but need data to compete for users.

    3. Talent Concentration

    Top AI researchers go where they can train the largest models.

    Salary Arms Race:

  • Senior ML researcher at Big Tech: $500K-$1M+
  • Top researchers (Ilya Sutskever, Demis Hassabis): Strategic hires
  • Academic brain drain: Most top professors consult or leave for industry

    • 4. Infrastructure Lock-In

    Training at scale requires:

  • Custom silicon (Google TPUs, Amazon Trainium, Microsoft Maia)
  • Proprietary interconnects
  • Specialized software stacks
  • Years of engineering investment

    • First-Mover Advantage: Companies that built infrastructure early (Google, Microsoft) can scale faster.

    The Open-Source Counter-Movement

    The Thesis: Open weights models (Llama, Mistral) can compete by:

  • Community innovation (fine-tunes, quantization)
  • Lower inference costs (can run locally)
  • Customization for specific use-cases

    • The Reality: Llama 3 (405B) approaches GPT-4 performance but:

  • Required Meta's massive compute ($500M+ training cost)
  • Still behind on cutting-edge capabilities
  • Fine-tuning doesn't close frontier gap

    • The Pattern: Open-source lags frontier by 6-12 months.

    The Question: Can community coordination beat proprietary scale?

    Historical Analogy: Linux vs Windows

  • Linux "won" servers, lost desktop
  • Open models might "win" inference, lose frontier training

    • The Product Implications: Building in a Scaling World

    For Startups

    The Bad News: You can't out-scale OpenAI.

    The Good News: You don't need to.

    Winning Strategies:

    1. Vertical Integration

  • Build models for specific domains (legal, medical, code)
  • Use smaller, fine-tuned models (cheaper inference)
  • Example: Harvey (legal AI), Glean (enterprise search)

    • 2. Application Layer

  • Assume foundation models are commodity
  • Build value in UI/UX, data pipelines, workflows
  • Example: Jasper (marketing copy), Notion AI (productivity)

    • 3. Distribution

  • Foundation models need distribution
  • Embed AI in existing products users love
  • Example: GitHub Copilot (developer workflow)

    • 4. Cost Arbitrage

  • Open-source models good enough for many use-cases
  • Offer 10x cheaper inference
  • Example: Together AI, Replicate, Modal

    • The Trap: Assuming architectural innovation beats scale. It (usually) doesn't.

    For Enterprises

    The Strategic Question: Build vs Buy vs Partner?

    Build (Train your own foundation model):

  • Only makes sense if:
    • - You have proprietary data at massive scale (Bloomberg, Salesforce) - Domain is specialized enough that general models fail - Regulatory requirements prevent external models

    Buy (Use existing foundation models):

  • Default choice for 95% of companies
  • Trade-off between cost (open models) and capability (proprietary)

    • Partner (Strategic relationship with model provider):

  • Anthropic + AWS
  • OpenAI + Microsoft
  • Makes sense if you're a cloud provider or have distribution

    • The Reality: Most companies should buy, not build.

    The Research Frontier: Beyond Simple Scaling

    1. Test-Time Compute Scaling (OpenAI o1)

    The Idea: Scale compute at inference time, not just training time.

    How it works:

  • Model "thinks" longer (more inference passes)
  • Uses chain-of-thought reasoning internally
  • Scales performance with compute budget

    • The Breakthrough: Performance on math/coding improves with more inference compute—even without more training.

    The Implications:

  • Training might not be the only scaling dimension
  • "Slow AI" that thinks carefully could be more capable
  • Changes cost structure (pay for inference, not training)

    • 2. Mixture of Experts (MoE)

    The Idea: Don't activate all parameters—route tokens to specialized experts.

    Benefits:

  • GPT-4 style capacity (1.8T params) with GPT-3 style compute
  • Faster inference (fewer active params)
  • Specialization (different experts for different tasks)

    • Challenges:

  • Engineering complexity (load balancing)
  • Training instability
  • Unclear how to optimally allocate experts

    • The Models: GPT-4, Mixtral 8x7B, DeepSeek V2

    3. Multimodal Scaling

    The Idea: Language models are data-limited, but vision/audio aren't.

    The Scale:

  • YouTube: 1B+ hours of video
  • Images: 10B+ labeled images (ImageNet, LAION)
  • Audio: Unlimited podcast/music data

    • The Hypothesis: Training on video (text + vision + audio + temporal reasoning) is the next scaling frontier.

    The Models: GPT-4V, Gemini, Claude 3

    The Question: Does multimodal training improve language reasoning? (Evidence: Yes, surprisingly.)

    4. Reasoning Compute vs Training Compute

    The Distinction:

  • Training compute: One-time cost, builds the model
  • Reasoning compute: Per-query cost, runtime thinking

    • The Tradeoff:

  • Spend more on training → cheaper inference
  • Spend more on inference → can use smaller models

    • The Innovation: OpenAI o1 shifts compute to inference, gets better performance.

    The Future: Hybrid approach—medium training compute, high reasoning compute.

    The Societal Implications

    1. Compute Becomes Geopolitical

    The Chip War: US restricts advanced GPU exports to China.

    Why: Training frontier models requires cutting-edge chips (H100s).

    China's Response:

  • Stockpile older chips (A100s)
  • Develop domestic alternatives (Huawei Ascend)
  • Focus on inference efficiency

    • The Stakes: Whichever country controls compute controls AI.

    2. Energy and Climate

    The Math:

  • Current AI: ~0.1% of global electricity
  • 2030 AI (projected): 1-2% of global electricity
  • Equivalent to adding Argentina's entire consumption

    • The Solutions:

  • Nuclear power (Microsoft buying reactors)
  • Renewable energy (Google's solar)
  • Efficiency improvements (better hardware, quantization)

    • The Trade-off: AI progress vs carbon emissions.

    3. Concentration of Power

    The Concern: Only 5-10 companies can train frontier models.

    The Risk:

  • Oligopoly control over critical infrastructure
  • Censorship (models trained with certain values)
  • Surveillance (all queries logged)

    • The Counter: Open-source models decentralize access.

    The Reality: Open models can't close the capability gap (yet).

    Conclusion: Scaling Laws as Roadmap

    Scaling laws transformed AI from alchemy to engineering. We now know:

    1. Performance is predictable: More compute, data, parameters → better models 2. The path forward is clear: Scale all three dimensions 3. The limits are real: Physics, economics, data exhaust 4. The winners are obvious: Whoever has most compute

    But scaling laws don't tell the whole story:

  • Emergent abilities aren't fully explained
  • Architecture innovations still matter (Transformers, MoE)
  • Test-time compute opens new dimensions
  • AGI might require breakthroughs beyond scale

    • For builders, the lesson is clear:

  • Don't try to out-scale the giants
  • Build where scale isn't the primary advantage
  • Assume foundation models are commodity (eventually)

    • For researchers, the challenge is:

  • Push beyond scaling laws
  • Find the missing pieces (reasoning, world models)
  • Make AI work with less compute (efficiency)

    • For society, the question is:

  • How do we govern systems that get more capable predictably?
  • How do we ensure AI benefits are widely distributed?
  • How do we handle the concentration of AI power?

    • Scaling laws showed us the path to powerful AI. Now we need to decide: How far should we go, and who gets to control the journey?

    ---

    Further Reading:

  • "Scaling Laws for Neural Language Models" (Kaplan et al., 2020) - The original paper
  • "Training Compute-Optimal Large Language Models" (Hoffmann et al., 2022) - Chinchilla laws
  • "Sparks of Artificial General Intelligence" (Bubeck et al., 2023) - GPT-4 capabilities analysis
  • Epoch AI Reports - Tracking compute trends and limits

    • Key Resources:

  • Compute Trends: epoch.ai/blog/compute-trends
  • Model Performance: huggingface.co/spaces/lmsys/chatbot-arena-leaderboard
  • Scaling Laws Calculator: scaling-laws.com (interactive tool)
  • AI Index Report: aiindex.stanford.edu (comprehensive annual report)