Machine Learning Trends

Our Trends dashboard offers curated key numbers, visualizations, and insights that showcase the significant growth and impact of artificial intelligence.

Last updated on Apr 09, 2024

Display growth values in:

Training compute

Likely

4.2 x/year

Growth rate in the training compute of milestone systems since 2010.

Training data

Plausible

2024

Median projected year in which most publicly available high-quality human-generated text will be used in a training run.

Computational performance

Likely

1.35 x/year

Growth rate in the amount of FLOP/s for GPUs in FP32 precision. A similar trend is observed for FP16.

Algorithmic improvements

Plausible

2.5 %/year

Decline rate in the physical compute requirements for image classification.

Training costs

Likely

3.1 x/year

Growth rate in USD of the cost of training notable ML models since 2009.

Most compute-intensive biological sequence model

Likely

6.2e23 FLOP

Training compute estimate for xTrimoPGLM, the most compute-intensive biological sequence model to date.

Compute Trends

Deep Learning compute

Likely

4.2 x

Growth rate in the training compute of milestone systems since 2010.

Pre-Deep Learning compute

Likely

1.5 x

Growth rate in the training compute of milestone systems from 1956 to 2010.

Large-scale vs regular scale

Uncertain

100x

Gap between the compute used by large-scale models and other notable models in the same year.

Paper

Training Compute of Milestone Machine Learning Systems Over Time

We’ve compiled a dataset of the training compute for over 120 machine learning models, highlighting novel trends and insights into the development of AI since 1952, and what to expect going forward.

Most compute-intensive training run

Plausible

5e25 FLOP

Total training compute for the final training run of Gemini Ultra, likely the most compute-intensive model to date.

Data Trends

Language training dataset size

Likely

2.2 x

Growth rate in the training dataset size for language models since 2010.

When will the largest training runs use all high-quality text?

Plausible

2024

Median projected year in which most publicly available high-quality human-generated text will be used in a training run.

When will the largest training runs use all text?

Uncertain

2040

Median projected year in which most publicly available human-generated text will be used in a training run.

Paper

Will We Run Out of ML Data? Evidence From Projecting Dataset Size Trends

Based on our previous analysis of trends in dataset size, we project the growth of dataset size in the language and vision domains. We explore the limits of this trend by estimating the total stock of available unlabeled data over the next decades.

Largest training dataset

Uncertain

9 trillion words

Training dataset size of DBRX, the language model trained on the largest known dataset to date.

Stock of data on the internet

Plausible

100 trillion words

Number of words in Common Crawl, the largest public repository of web crawl data on the internet.

Hardware Trends

Computational performance

Likely

1.35 x

Growth rate in the amount of FLOP/s for GPUs in FP32 precision. A similar trend is observed for FP16.

Lower-precision number formats

Plausible

8 x

Average performance gain in FLOP/s from switching from FP32 to tensor-FP16.

Memory capacity

Likely

1.2 x

Growth rate in DRAM capacity (Byte).

Memory bandwidth

Likely

1.18 x

Growth rate in DRAM bandwidth in Byte/s.

Report

Trends in Machine Learning Hardware

We analyze recent trends in machine learning hardware performance, focusing on metrics such as computational performance, memory, interconnect bandwidth, price-performance, and energy efficiency across different GPUs and accelerators. The analysis aims to provide a holistic view of ML hardware capability and bottlenecks.

Highest performing GPU in Tensor-FP16

Likely

2.25e15 FLOP/s

Highest performance of a GPU in Tensor-FP16, from the NVIDIA B200 SXM.

Highest performing GPU in INT8

Likely

4.5e15 FLOP/s

Highest performance of a GPU in INT8, from the NVIDIA B200 SXM.

Algorithmic Progress

Compute-efficiency in language models

Plausible

2.7 %

Decline rate in the physical compute required to achieve a given performance.

Compute-efficiency in computer vision models

Plausible

2.5 %

Decline rate in the physical compute required to achieve a given performance.

Contribution of algorithmic innovation

Plausible

35%

Improvements to compute efficiency explain roughly 35% of performance improvements in language modeling since 2014, vs 65% explained by increases in scale.

Paper

Algorithmic Progress in Language Models

We study how algorithmic improvements and increases in computational power have improved the performance of language models from 2014 to 2024. We find that the progress from new algorithms surpasses what we’d expect from merely increasing our computing resources, occurring at a pace equivalent to doubling computational power every 5 to 14 months.

Chinchilla scaling laws

Plausible

20 tokens per parameter

Ratio of data to parameters to achieve compute-optimal scaling for LLMs, according to (Hoffmann et al, 2022)

Investment Trends

Training costs

Likely

3.1 x

Growth rate in USD of the cost of training notable ML models since 2009.

Report

Trends in the Dollar Training Cost of Machine Learning Systems

We combine training compute and GPU price-performance data to estimate the cost of compute in US dollars for the final training run of 124 machine learning systems published between 2009 and 2022, and find that the cost has grown by approximately 0.5 orders of magnitude per year.

Most expensive AI system

Uncertain

$630 million

Total cost of developing Gemini Ultra, including hardware, electricity, and salaries.

Biological Models

Training compute

Likely

8.7 x

Growth rate in the training compute of biological sequence models since 2018.

Key DNA sequence database

Likely

8.3 x

Growth rate in the number of sequences stored in GenBank (INSDC) between 2022 and 2023.

Report

Biological Sequence Models in the Context of the AI Directives

The White House recently issued an Executive Order requiring enhanced oversight for AI models trained on biological data exceeding 1e23 operations. We provide an overview of our expanded data coverage to biological sequence models, revealing a significant increase in computational resources and the extensive availability of protein and DNA sequence data. Our analysis identifies critical trends in training compute, data stock, and potential regulatory gaps.

Most compute-intensive biological sequence model

Likely

6.2e23 FLOP

Training compute estimate for xTrimoPGLM, the most compute-intensive biological sequence model to date.

Protein sequence data

Uncertain

~7 billion entries

Estimate of the number of unique entries across protein sequence databases.

Acknowledgements

We thank Tom Davidson, Lukas Finnveden, Charlie Giattino, Zach-Stein Perlman, Misha Yagudin, Robi Rahman, Jai Vipra, Patrick Levermore, Carl Shulman, Ben Bucknall and Daniel Kokotajlo for their feedback.

Several people have contributed to the design and maintenance of this dashboard, including Jaime Sevilla, Pablo Villalobos, Anson Ho, Tamay Besiroglu, Ege Erdil, Ben Cottier, Matthew Barnett, David Owen, Robi Rahman, Lennart Heim, Marius Hobbhahn, David Atkinson, Keith Wynroe, Christopher Phenicie, Nicole Maug, Alex Haase, Robert Sandler and Edu Roldan.

Citation

Cite this work as

    Epoch AI (2023), "Key Trends and Figures in Machine Learning". Published online at epochai.org. Retrieved from: 'https://epochai.org/trends' [online resource]
  

BibTeX citation

@misc{epoch2023aitrends,
  title = "Key Trends and Figures in Machine Learning",
  author = {Epoch AI},
  year = 2023,
  url = {https://epochai.org/trends},
  note = "Accessed: "
}

Cite this work as

              
                Epoch AI (2023), "Key Trends and Figures in Machine Learning". Published online at epochai.org. Retrieved from: 'https://epochai.org/trends' [online resource]

BibTeX citation

@misc{epoch2023aitrends,
  title={Key Trends and Figures in Machine Learning},
  author={Epoch AI},
  year={2023},
  url={https://epochai.org/trends},
  note={Accessed: }
}


            

If you spot an error or would like to provide feedback, please reach out at .