Using Recon for Data Cleaning

To aid in making investment decisions, financial analysts purchase large amounts of data from financial data providers. They use this historical data to develop financial models for making predictions and assessing risk about current dat about current dataa. Unfortunately, these database often contain errors and omissions of important information. Analysts are dependent upon the quality of these databases—missing data can prevent computation of key values and, more dangerously, incorrect data can cause their models to produce erroneous results without the analyst's knowledge. Because of the importance of accurate data, and the large volume of data involved, data providers and consumers have a need to develop advanced methods for data cleaning: the process of identifying and correcting incomplete and incorrect information in databases. This paper describes how the Recon data mining system has been used to clean financial databases. Recon incorporates several data mining modules into a single, uniform framework: data visualization, deductive databases, and rule induction. The data visualization component supports the visual detection of outliers and other unusual phenomena. The deductive database enables analysts to capture, codify, and apply corporate knowledge about data integrity. The rule induction module creates error detection rules by generalizing from known errors to detect suspicious data entries in the rest of the data. The collaborative use of these three modules yields superior error detection over the application of any single data mining technique.