Many organizations process big data for important business operations and decisions. Hence, data quality greatly affects their success. Data quality problems continue to be widespread, costing US businesses an estimated $600 billion annually. To date, addressing data quality in production environments still poses many challenges: easily defining properties of high-quality data; validating production-scale data in a timely manner; debugging poor quality data; designing data quality solutions to be easy to use, understand, and operate; and designing data quality solutions to easily integrate with other systems. Current data validation solutions do not comprehensively address these challenges. To address data quality in production environments at LinkedIn, we developed Data Sentinel, a declarative production-scale data validation platform. In a simple and well-structured configuration, users declaratively specify the desired data checks. Then, Data Sentinel performs these data checks and writes the results to an easily understandable report. Furthermore, Data Sentinel provides well-defined schemas for the configuration and report. This makes it easy for other systems to interface or integrate with Data Sentinel. To make Data Sentinel even easier to use, understand, and operate in production environments, we provide Data Sentinel Service (DSS), a complementary system to help specify data checks, schedule, deploy, and tune data validation jobs, and understand data checking results. The contributions of this paper include the following: 1) Data Sentinel, a declarative production-scale data validation platform successfully deployed at LinkedIn 2) A generic design to build and deploy similar systems for production environments 3) Experiences and lessons learned that can benefit practitioners with similar objectives.

Good estimates of join result sizes are critical for query optimization in relational database management systems. We address the problem of incrementally obtaining accurate and consistent estimates of join result sizes. We have invented a new rule for choosing join selectivities for estimating join result sizes. The rule is part of a new unified algorithm called Algorithm ELS (Equivalence and Largest Selectivity). Prior to computing any result sizes, equivalence classes are determined for the join columns. The algorithm also takes into account the effect of local predicates on table and column cardinalities. These computations allow the correct selectivity values for each eligible join predicate to be computed. We show that the algorithm is correct and gives better estimates than current estimation algorithms. 1 I n t r o d u c t i o n The join is an important operation in relational database management systems. When a user-generated query involves multiple joins, the cost of executing the query can vary dramatically depending on the query evaluation plan (QEP) chosen by the query optimizer. The join order and access methods used are important determinants of the lowest cost QEP. The query optimizer estimates the eventual result size, or cardinality, of joining the specified tables and uses this information to choose between different join orders and access methods. Thus, the estimation of join result sizes in a query is an important problem, as the estimates have a significant influence on the QEP chosen. A survey of the use of statistics and estimation techniques in query optimization is given in [7]. There has been a lot of work done on estimating setectivities, e.g., ([1, 10, 8]). In [6], a number of new selectivity estimation methods were proposed for highly skewed distributions such as Zipf distributions [17, 3]. Errors in the statistics maintained by the database system can affect the various estimates computed by the query optimizer. An analytical model was used in [4] to study the propagation of errors in the estimated size of join results as the number of joins increases in queries with a single equivalence class. When more than two tables are involved, the query optimization algorithm often needs to estimate the join result sizes incrementally. Incremental estimation is used, for example, in the dynamic programming algorithm [13], the A B algorithm [15] and randomized algorithms [14, 5]. The optimizer first determines the join result size from joining the first two tables, then determines the join result size from the join with a third table, and so on. Thus, the query optimizer

Good estimates of join result sizes are critical for query optimization in relational database management systems. We address the problem of incrementally obtaining accurate and consistent estimates of join result sizes. We have invented a new rule for choosing join selectivities for estimating join result sizes. The rule is part of a new unified algorithm called Algorithm ELS (Equivalence and Largest Selectivity). Prior to computing any result sizes, equivalence classes are determined for the join columns. The algorithm also takes into account the effect of local predicates on table and column cardinalities. These computations allow the correct selectivity values for each eligible join predicate to be computed. We show that the algorithm is correct and gives better estimates than current estimation algorithms.

We propose an indexing method for time sequences for processing similarity queries. We use the Discrete Fourier Transform (DFT) to map time sequences to the frequency domain, the crucial observation being that, for most sequences of practical interest, only the rst few frequencies are strong. Another important observation is Parseval's theorem, which speci es that the Fourier transform preserves the Euclidean distance in the time or frequency domain. Having thus mapped sequences to a lower-dimensionality space by using only the rst few Fourier coe cients, we use R -trees to index the sequences and e ciently answer similarity queries. We provide experimental results which show that our method is superior to search based on sequential scanning. Our experiments show that a few coe cients (1-3) are adequate to provide good performance. The performance gain of our method increases with the number and length of sequences.

With the increase in popularity of the Internet, the latency experienced by an individual, while accessing the web, is increasing. In this paper, we investigate one approach to reducing latency by increasing the hit rate for a web cache. To this effect, we developed a predictive model for pre-fetching and a modified Least Recently Used (LRU) method called AssocLRU. This paper investigates the application of a data mining technique, called Association rules to the web domain. The association rules, predict the URLs a user might reference next, and this knowledge is used in our web caching and prefetching model. We developed a trace driven cache simulator to compare the performance of our predictive model with the widely used replacement policy, namely, LRU. The traces we used in our experiments were the traces of Web proxy activity taken at Virginia Tech and EPA HTTP. Our results show that our predictive pre-fetching model using association rules achieves a better hit rate than both LRU and AssocLRU.

Data mining is being applied with profit in many applications . Clustering or segmentation of data is an important data mining application. One of the problems with traditional clustering methods is that they require the analyst to define distance functions that are not always available. In this paper, we describe a new method for clustering without distance functions.

The authors consider the problem of clustering two-dimensional association rules in large databases. They present a geometric-based algorithm, BitOp, for performing the clustering, embedded within an association rule clustering system, ARCS. Association rule clustering is useful when the user desires to segment the data. They measure the quality of the segmentation generated by ARCS using the minimum description length (MDL) principle of encoding the clusters on several databases including noise and errors. Scale-up experiments show that ARCS, using the BitOp algorithm, scales linearly with the amount of data.

Amir, Amihood, 299 Andries, Marc, 357 Astesiano, Egidio, 143 … Baker, Brenda S., 28 Bartlett, Peter L., 434 Beeri, Catriel, 493 Benson, Gary, 299 Bshouty, Nader H., 268, 421 … Chen, Tsong Yueh, 80 Cherubini, Alessandra, 61 Cleve, Richard, 421 … Ehrenfeucht, Andrzej, 170 Engelfriet, Joost, 170 Eppstein, David, 3 … Fang, Shao C., 374 Farach, Martin, 299 Fortnow, Lance, 191 Frazier, Michael, 471 … Galil, Zvi, 3 Gasarch, William, 391 GavaldaÁ, Ricard, 421 Gibson, Gavin J., 349 Goldman, Sally, 471 Goldman, Sally A., 255, 268 Goodrich, Michael T., 1 Grumbach, Stephane, 570 Gunopulos, Dimitrios, 453 … Haas, Peter J., 550 Hancock, Thomas R., 268 Harel, David, 522 Hirst, Tirza, 522 Honkala, Juha, 185 … Kanellakis, Paris, 589 Kannan, Sampath, 421 Krishnamurthy, Ravi, 100 Kshemkalyani, Ajay D., 287 Kummer, Martin, 214, 403 … La Poutre, Han, 87 Libkin, Leonid, 125 Li, Ming …

We compare the performance of sampling-based procedures for estimating the selectivity of a join. While some of the procedures have been proposed in the database literature, their relative performance has never been analyzed. A main result of this paper is a partial ordering that compares the variability of the estimators for the different procedures after an arbitrary fixed number of sampling steps. Prior to the current work, it was also unknown whether these fixed-step procedures could be extended to fixed-precision procedures that are both asymptotically consistent and asymptotically efficient. Our second main result is a general method for such an extension and a proof that the method is valid for all the procedures under consideration. We show that, under plausible assumptions on sampling costs, the partial ordering of the fixed-step procedures with respect to variability of the selectivity estimator implies a partial ordering of the corresponding fixed-precision procedures with respect to sampling cost. Our final result is a collection of fixed-step and fixed-precision procedures for estimating the cost of processing a join query according to a fixed join plan.

This paper demonstrates the use of generalized partial indexes for efficient query processing. We propose that partial indexes be built on those portions of the database that are statistically likely to be the most useful for query processing. We identify three classes of statistical information, and two levels at which it may be available. We describe indexing strategies that use this information to significantly improve average query performance. Results from simulation experiments demonstrate that the proposed generalized partial indexing strategies perform well compared to the traditional approach to indexing.<>

In this paper, we present a new parameterized parallel sort algorithm, called Round-Robin Partitioned (or RRP), for the message passing (shared-nothing) architecture. This is a parameterized sort algorithm because a parameter is provided which can be used to determine the amount of memory used and to allocate differing amounts of work to different sets of sites. We utilize pipelining to hide disk I/O time, exploit high degrees of parallelism at all phases, apply sampling to determine the partition key values and use less memory than previous known methods while repairing the minimum number of physical I/Os. The basic version of the RRP algorithm is simple in terms of coding and complexity. It does not require disk I/O parallelism or data prefetch within a single process. We develop an analytical model for our algorithm and compare our sort algorithm with four other classes of external parallel sort algorithms. The RRP algorithm are shown to be superior to the other algorithms for almost all configurations.< >

We describe an algorithm for estimating the number of page fetches for a partial or complete scan of a B-tree index. The algorithm obtains estimates for the number of page fetches for an index scan when given the number of tuples selected and the number of LRU buffers currently available. The algorithm has an initial phase that is performed exactly once before any estimates are calculated. This initial phase, involving LRU buffer modeling, requires a scan of all the index entries and calculates the number of page fetches for different buffer sizes. An approximate empirical model is obtained from this data. Subsequently, an inexpensive estimation procedure is called by the query optimizer whenever it needs an estimate of the page fetches for the index scan. This procedure utilizes the empirical model obtained in the initial phase.

We present an algorithm for nding the quantile values of a large unordered dataset with unknown distribution. The algorithm has the following features: i) it requires only one pass over the data; ii) it is space e cient | it uses a small bounded amount of memory independent of the number of values in the dataset; and iii) the true quantile is guaranteed to lie within the lower and upper bounds produced by the algorithm. Empirical evaluation using synthetic data with various distributions as well as real data show that the bounds obtained are quite tight. The algorithm has several applications in database systems, for example in database governors, query optimization, load balancing in multiprocessor database systems, and data mining.