Large recommendation models (LRMs) are fundamental to the multi-billion dollar online advertising industry, processing massive datasets of hundreds of billions of examples before transitioning to continuous online training to adapt to rapidly changing user behavior. The massive scale of data directly impacts both computational costs and the speed at which new methods can be evaluated (R D velocity). This paper presents actionable principles and high-level frameworks to guide practitioners in optimizing training data requirements. These strategies have been successfully deployed in Google's largest Ads CTR prediction models and are broadly applicable beyond LRMs. We outline the concept of data convergence, describe methods to accelerate this convergence, and finally, detail how to optimally balance training data volume with model size.

The concern that Artificial Intelligence (AI) and Machine Learning (ML) are entering a “reproducibility crisis” has spurred significant research in the past few years. Yet with each paper, it is often unclear what someone means by “reproducibility” and where it fits in the larger scope of what we will call the “scientific rigor” literature. Ultimately, the lack of clear rigor standards can affect the manner in which businesses seeking to adopt AI/ML implement such capabilities. In this survey, we will use 66 papers published since 2017 to construct a proposed set of 8 high-level categories of scientific rigor, what they are, and the history of work conducted in each. Our proposal is that these eight rigor types are not mutually exclusive and present a model for how they influence each other. To encourage more to study these questions, we map these rigors to the adoption process in real-world business use cases. In doing so, we can quantify gaps in the literature that suggest an under focus on the issues necessary for scientific rigor research to transition to practice

For industrial-scale advertising systems, prediction of ad click-through rate (CTR) is a central problem. Ad clicks constitute a significant class of user engagements and are often used as the primary signal for the usefulness of ads to users. Additionally, in cost-per-click advertising systems where advertisers are charged per click, click rate expectations feed directly into value estimation. Accordingly, CTR model development is a significant investment for most Internet advertising companies. Engineering for such problems requires many machine learning (ML) techniques suited to online learning that go well beyond traditional accuracy improvements, especially concerning efficiency, reproducibility, calibration, credit attribution. We present a case study of practical techniques deployed in Google's search ads CTR model. This paper provides an industry case study highlighting important areas of current ML research and illustrating how impactful new ML methods are evaluated and made useful in a large-scale industrial setting.

2 機械学習システムにおける技術的負債に関する既存研究 2.1 システム設計レベルの技術的負債の分類 Sculleyらは,機械学習システムのシステム設計レベルで生じるリスク要因を 5つ 挙げている [5] [6]. • Complex Models Erode Boundaries • Data Dependencies Cost More than Code Dependencies • Feedback Loops • Conguration Debt • Dealing with Changes in the External World また,機械学習システムのシステムレベルのアンチパターンとして,以下を 7つ挙 げている. • Glue Code • Pipeline Jungles • Dead Experimental Codepaths • Abstraction Debt • Plain-Old-Data Type Smell • Multiple-Language Smell • Prototype Smell

Machine learning offers a fantastically powerful toolkit for building useful complex prediction systems quickly. This paper argues it is dangerous to think of these quick wins as coming for free. Using the software engineering framework of technical debt, we find it is common to incur massive ongoing maintenance costs in real-world ML systems. We explore several ML-specific risk factors to account for in system design. These include boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, configuration issues, changes in the external world, and a variety of system-level anti-patterns.

Machine learning offers a fantastically powerful toolkit for building complex systems quickly. This paper argues that it is dangerous to think of these quick wins as coming for free. Using the framework of technical debt, we note that it is remarkably easy to incur massive ongoing maintenance costs at the system level when applying machine learning. The goal of this paper is highlight several machine learning specific risk factors and design patterns to be avoided or refactored where possible. These include boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, changes in the external world, and a variety of system-level anti-patterns. 1 Machine Learning and Complex Systems Real world software engineers are often faced with the challenge of moving quickly to ship new products or services, which can lead to a dilemma between speed of execution and quality of engineering. The concept of technical debt was first introduced by Ward Cunningham in 1992 as a way to help quantify the cost of such decisions. Like incurring fiscal debt, there are often sound strategic reasons to take on technical debt. Not all debt is necessarily bad, but technical debt does tend to compound. Deferring the work to pay it off results in increasing costs, system brittleness, and reduced rates of innovation. Traditional methods of paying off technical debt include refactoring, increasing coverage of unit tests, deleting dead code, reducing dependencies, tightening APIs, and improving documentation [4]. The goal of these activities is not to add new functionality, but to make it easier to add future improvements, be cheaper to maintain, and reduce the likelihood of bugs. One of the basic arguments in this paper is that machine learning packages have all the basic code complexity issues as normal code, but also have a larger system-level complexity that can create hidden debt. Thus, refactoring these libraries, adding better unit tests, and associated activity is time well spent but does not necessarily address debt at a systems level. In this paper, we focus on the system-level interaction between machine learning code and larger systems as an area where hidden technical debt may rapidly accumulate. At a system-level, a machine learning model may subtly erode abstraction boundaries. It may be tempting to re-use input signals in ways that create unintended tight coupling of otherwise disjoint systems. Machine learning packages may often be treated as black boxes, resulting in large masses of “glue code” or calibration layers that can lock in assumptions. Changes in the external world may make models or input signals change behavior in unintended ways, ratcheting up maintenance cost and the burden of any debt. Even monitoring that the system as a whole is operating as intended may be difficult without careful design.

Predicting ad click-through rates (CTR) is a massive-scale learning problem that is central to the multi-billion dollar online advertising industry. We present a selection of case studies and topics drawn from recent experiments in the setting of a deployed CTR prediction system. These include improvements in the context of traditional supervised learning based on an FTRL-Proximal online learning algorithm (which has excellent sparsity and convergence properties) and the use of per-coordinate learning rates. We also explore some of the challenges that arise in a real-world system that may appear at first to be outside the domain of traditional machine learning research. These include useful tricks for memory savings, methods for assessing and visualizing performance, practical methods for providing confidence estimates for predicted probabilities, calibration methods, and methods for automated management of features. Finally, we also detail several directions that did not turn out to be beneficial for us, despite promising results elsewhere in the literature. The goal of this paper is to highlight the close relationship between theoretical advances and practical engineering in this industrial setting, and to show the depth of challenges that appear when applying traditional machine learning methods in a complex dynamic system.

We present NetMesh, a new algorithm that produces a conforming Delaunay mesh for point sets in any fixed dimension with guaranteed optimal mesh size and quality. Our comparison-based algorithm runs in O(n log n + m) time, where n is the input size and m is the output size, and with constants depending only on the dimension and the desired element quality. It can terminate early in O(n log n) time returning a O(n) size Voronoi diagram of a superset of P, which again matches the known lower bounds. The previous best results in the comparison model depended on the log of the spread of the input, the ratio of the largest to smallest pairwise distance. We reduce this dependence to O(log n) by using a sequence of ε-nets to determine input insertion order into a incremental Voronoi diagram. We generate a hierarchy of well-spaced meshes and use these to show that the complexity of the Voronoi diagram stays linear in the number of points throughout the construction.

Voronoi refinement is a powerful tool for efficiently generating meshes for finite element simulation. The classic definition of quality in a mesh can be achieved by bounding the aspect ratio of the Voronoi cells measured as the ratio of the circumscribing and inscribing radii as measured from the site. There are tight upper and lower bounds on the number of extra points needed to achieve such a Voronoi diagram. The use of good aspect ratio Voronoi diagrams is central to both quadtree methods [1] and Voronoi refinement algorithms [3]. Unfortunately, bounding the aspect ratio in this way is often an overkill, with lower bounds on the size and runtime that depend on the spread of the input set, a geometric quantity that may be unbounded in n. In this paper, we give a relaxed definition of Voronoi cell quality called fatness that captures many of the nice properties of the old definition without being subject to the lower bounds on the size. We give upper and lower bounds on the complexity of such Voronoi diagrams and provide an algorithm to generate such a Voronoi diagram with only a linear number of extra points. In future work we hope to understand fat Voronoi diagrams well enough to design the next generation meshing algorithm with them. The first and simplest question that arises in this area is whether or not a cell in a fat Voronoi diagram can have an unbounded number of neighbors. We prove that this is not possible for fat Voronoi diagrams in the plane and conjecture that similar bounds hold in higher dimensions. As Figure 1 demonstrates, this is a property peculiar to fat Voronoi diagrams; it holds neither for general fat complexes nor for weighted Voronoi diagrams.

We present a new meshing algorithm to mesh an arbitrary piecewise-linear complex in three dimensions. The algorithm achieves an O ( n log ∆ + m ) runtime where n , m , and ∆ are the input size, the output size, and spread respectively. This runtime represents the first non-trivial runtime guarantee for this class of input. The new algorithm extends prior work on runtime-efficient meshing by allowing the input to have acute input angles (called creases). Features meeting at creases are handled with protective collars. A new procedure is given for creating these collars in an unstructured fashion, without the need for expensive size precomputations as in prior work. Output tetrahedra have quality radius-edge ratios in a region away from the “creases”. Adjacent to creases, output tetrahedra have no large dihedral angles. The collar surface dividing these two regions is represented implicitly using surface reconstruction techniques. This new approach allows the algorithm to run in a single pass.

We present a test for the invertibility of second order Bézier elements. Our test that can determine precisely if such the function is invertible in only a constant number of operations. Although our formulation is specific to triangles, it is simple to extend it to other polygonal domains. Our test is used in the routines of the Lagrangian finite element fluid solver Tumble. Tumble uses moving mesh algorithms that might cause elements to become inverted such that the fluid’s velocity field is not reconstructible. Our test is used to ensure that elements in the mesh are invertible after each operation. Furthermore, we present possible invariants that might allow certain operations to be performed without the need to test for invertibility at every step. In turn, we present pathological examples of Bézier triangles that fail under naive invariants. Introduction Bézier elements have use in finite element method to represent a polynomial mapping from a linear unit element such as a triangle or square. A set of Bézier elements is often constructed to form a mesh. Operations on Bézier meshes include moving the control structure under invariants, or reconstructing the mesh entirely. In finite element method in particular, a common invariant is that it is necessary for the Bézier polynomials to be invertible, so as to make reconstructing data possible. Often stronger and easier to ensure invariants are used in the place of precise invertibility. This however, sacrifices the set of operations that can be performed and thus also the precision of the calculation. It is thus desirable to have precise tests for invertibility. Figure 1: An example of a Bézier triangle We present a simple test for determining the invertibility of a second degree polynomial function defined on a triangle. Our test can determine precisely if such a function is invertible in only a constant number of arithmetic operations. It is possible to show that a function is invertible iff the determinant of its Jacobian is positive everywhere on its domain. By simply performing calculus on the determinant of the Jacobian of the Bézier polynomial, we find that it is unnecessary to determine the minimum within the domain in the case of second degree Bézier triangles. We can determine whether a Bézier triangle is invertible by checking the global minimum and the minimum along the edges, the former of which is solvable by matrix inversion, and the latter of which is solvable using the quadratic equation. This test has been successfully implemented in the subroutines of the Lagrangian finite element fluid solver Tumble, and has been shown to increase the length of the simulation before stability is lost or elements become inverted. Given this test, special cases have been observed for the purpose of designing more powerful invariants. During a fluid simulation, vortexes might occur in the fluid, causing the surrounding mesh to curve undesirably to the point of inversion. To prevent this, we attempt to straighten the misbehaving edges. However, straightening an edge does not necessarily preserve invertibility on the triangle for which that edge is being made more concave. To rectify this, we would like to impose the condition that the convex hulls of the edge’s control meshes are disjoint. While this condition is not sufficient for invertibility, it is seemingly sufficient for preserving invertibility when straitening a convex edge. To prove this as well as simpler tests for invertibility and conjectures, we This work was partially supported by the National Science Foundation under grant number CCF-0635257. Department of Computer Science, Carnegie Mellon University, glmiller@cs.cmu.edu Department of Computer Science, Carnegie Mellon University, mmirman@cs.cmu.edu Department of Computer Science, Carnegie Mellon University, tp517@cs.cmu.edu conjecture that if det(J) > 0 everywhere on the boundary of the domain triangle, the boundary of domain triangle has a winding number of 1, then if the function is second degree, it must be invertible everywhere within the domain. Because straitening edges preserves the disjoint edge hull condition, using it as an invariant, would reduce the need for the invertibility test and allow simulations to run for longer without getting stuck. Related Work Johnson[7] showed simple necessary and sufficient conditions in the case of second degree Bézier triangles in the case where two of the edges were linear. Vavasis [6] showed sufficient conditions for the Bézier elements defined on a more general set of domains. That invertible meshes with Bézier elements could be generated was demonstrated by Boivin et al [1]. A framework for performing finite element method on fluids using Bézier elements was laid out by Miller, et al. [2] Tumble is a software package built for the purposes of simulating blood flow for the Sangria project. In order to reduce the error introduced, it uses purely Lagrangian finite element method on a moving mesh composed of second degree Bézier triangles. [3] Background We define a 2D unit triangle in three dimensions as ∆ = {(α, β, γ) | 0 < α, β, γ ∧ α + β + γ = 1} Its boundary is ∆ = {(α, β, γ) |0 ≤ α, β, γ ∧ (0 = α ∨ 0 = β ∨ 0 = γ) ∧ α + β + γ = 1} A Bézier triangle polynomial is a polynomial function P : ∆ → R where

Typical volume meshes in three dimensions are designed to conform to an underlying two-dimensional surface mesh, with volume mesh element size growing larger away from the surface. The surface mesh may be uniformly spaced or highly graded, and may have fine resolution due to extrinsic mesh size concerns. When we desire that such a volume mesh have good aspect ratio, we require that some space-filling scaffold vertices be inserted off the surface. We analyze the number of scaffold vertices in a setting that encompasses many existing volume meshing algorithms. We show that under simple preconditions, the number of scaffold vertices will be linear in the number of surface vertices.

The tremendous usefulness of Voronoi diagrams is tempered by their worst-case O(n) size blowup. This makes them an obvious target for approximation, and indeed, several methods have been proposed that produce linear size approximations to the Voronoi diagram supporting logarithmic-time approximate nearest neighbor queries. All such methods use quadtrees to approximate the Voronoi cells. But what if the input does not have a “bad” Voronoi diagram? There is a huge gap between the best-case and the worst case complexity. Sometimes, the exact solution is both simpler and more precise than an approximation (Figure 1). We present a new method for constructing approximate Voronoi diagrams that uses the Voronoi diagram of a superset of the input as an approximation to the true Voronoi diagram. The approximate Voronoi cells are unions of Voronoi cells of the superset. If the input has a Voronoi diagram with good aspect ratio (and thus has linear size) then the approximate Voronoi diagramwe produce will simply be the Voronoi diagram of the input. The size of the diagram is O(n log∆) where ∆ is the spread of the input (the ratio of largest to smallest interpoint distances). Moreover, it supports approximate nearest neighbor queries in time O(log∆). We also discuss methods for eliminating the dependence on the spread in the size and reducing it to O(log n) for queries. The construction will be based on sparse meshing technology [4]. Formally, a (1 + e)-approximate Voronoi diagram of an input set N ∈ R is a spatial decomposition of

Abstract The mesh generation problem is to output a set of tetrahedra that discretize

We present a new algorithm to mesh an arbitrary piecewise linear complex in three dimensions. The algorithm achieves an O(n log∆ + m) runtime where n, m, and ∆ are the input size, the output size, and spread respectively. This represents the first non-trivial runtime guarantee for this class of input. The new algorithm extends prior work on runtime-efficient meshing by allowing the input to have acute input angles (called creases). Features meeting at creases are handled with protective collars. A new procedure is given for creating these collars in an unstructured fashion, without the need for expensive sizing precomputation as in prior work. The collar surface dividing these two regions is represented implicitly using surface reconstruction techniques. This new approach allows the collar to be dynamically generated , allowing the whole algorithm to run in a single pass. For inputs with∆ bounded by a polynomial in n, this runtime is optimal.

The mesh generation problem is to output a set of tetrahedra that discretize an input geometry. The input is given as a piecewise linear complex (PLC), a set of points, lines, and polygons to which the output tetrahedra must conform. Additionally, a mesh generation algorithm must make guarantees on the quality and number of output tetrahedra. Downstream applications in scientific computing and visualization necessitate these guarantees on the mesh. Recent advances have led to provably correct algorithms for a number of input classes. Particular difficulties arise when the input contains creases, regions where input segments or polygons meet at acute angles. When the input is without creases, the mesh generation problem is better understood. Algorithms for such inputs exist with near-optimal runtimes of O(n log Δ+m), where n and m are the size of the input and output, and Δ is the ratio of largest-to-smallest distances in the input geometry. The principle result of this thesis is to extend this result to the general case of piecewise linear complexes with creases. Correct algorithms to handle inputs with creases involve explicitly constructing a system of specially designed collars around the creases. These collars must be specifically sized according to the input geometry. I give a new procedure to compute the needed collar sizes in near-optimal O(n log Δ + c), where c is the description complexity of the collar system. Additionally, I give a procedure for implicitly constructing a collar system on the fly, so that a complete meshing algorithm for a PLC can be run in one pass with total work in O(n log Δ + m) and space usage in O(m). Central to the analysis is the Scaffold-Sizing Theorem, a structural result governing the number of vertices created during mesh generation. The theorem is general enough to have an added benefit of retroactively improving the analysis of almost all existing meshing algorithms.

There are many depth measures on point sets that yield centerpoint theorems. These theorems guarantee the existence of points of a specified depth, a kind of geometric median. However, the deep point guaranteed to exist is not guaranteed to be among the input, and often, it is not. The �-wedge depth of a point with respect to a point set is a natural generalization of halfspace depth that replaces halfspaces with wedges (cones or cocones) of angle �. We introduce the notion of a centervertex, a point with depth at least n d+1 among the set S. We prove that for any finite set S ⊂ R d , a centervertex exists. We also present a simple algorithm for computing an approximate centervertex.

We generalize the Tukey depth to use cones instead of halfspaces. We prove a generalization of the center point theorem that for S ⊂ R, there is a point s ∈ S, with depth at least n d+1 for cones of half-angle 45 . This gives a notion of data depth for which an approximate median can always be found among the original set.

The recent Sparse Voronoi Refinement (SVR) Algorithm for mesh generation has the fastest theoretical bounds for runtime and memory usage. We present a robust practical software implementation of the SVR for meshing a piecewise linear complex in 3 dimensions. Our software is competitive in runtime with state of the art freely available packages on generic inputs, and on pathological worse cases inputs, we show SVR indeed leverages its theoretical guarantees to produce vastly superior runtime and memory usage. The theoretical algorithm description of SVR leaves open several data structure design options, especially with regard to point location strategies. We show that proper strategic choices can greatly effect constant factors involved in runtime.

Most modern meshing algorithms produce asymptotically optimal size output. However, the size of the optimal mesh may not be bounded by any function of n. In this paper, we introduce well-paced point sets and prove that these will produce linear size outputs when meshed with any “size-optimal” meshing algorithm. This work generalizes all previous work on the linear cost of balancing quadtrees. We also present an algorithm that uses well-paced points to produce a linear size Delaunay mesh of a point set in R d .