Scaling Laws of Data-Driven Machine Learning Models: A Survey and Taxonomy

In recent years, there has been a notable surge in research dedicated to data-driven paradigms, which use existing data to train models capable of applying learned principles to real-world tasks. As the scale effects of the model are empirically confirmed, the importance of the data-driven paradigm becomes more apparent. However, scaling up also entails increased costs, necessitating theoretical estimations to provide reasonable guidance. This underscores the importance of foundational theoretical work on model scaling laws. Despite the wealth of research exploring model scaling laws from various perspectives, there lacks a comprehensive framework to integrate these findings. This work systematically summarizes and classifies existing theoretical work through the construction of a model triangular pyramid, considering data, computation, and algorithms as the three key factors. By organizing and categorizing these theories systematically, our aim is to provide a cohesive framework for navigating the complex and evolving landscape of scaling laws within the realm of data-driven models.