Data curation is the process of acquiring multiple sources of data, assessing and improving data quality, standardizing, and integrating the data into a usable information product, and eventually disposing of the data. The research describes the building of a proof-of-concept for an unsupervised data curation process addressing a basic form of data cleansing in the form of identifying redundant records through entity resolution and spelling corrections. The novelty of the approach is to use ER as the first step using an unsupervised blocking and stop word scheme based on token frequency. A scoring matrix is used for linking unstandardized references, and an unsupervised process for evaluating linking results based on cluster entropy. The ER process is iterative, and in each iteration, the match threshold is increased. The prototype was tested on 18 fully-annotated test samples of primarily synthetic person data varied in two different ways, good data quality versus poor data quality, and a single record layout versus two different record layouts. In samples with good data quality and using both single and mixed layouts, the final clusters had an average F-measure of 0.91, precision of 0.96, and recall of 0.87 outcomes comparable to results from a supervised ER process. In samples with poor data quality whether mixed or single layout, the average F-measure was 0.78, precision 0.74, and recall 0.83 showing that data quality assessment and improvement is still a critical component of successful data curation. The results demonstrate the feasibility of building an unsupervised ER engine to support data integration for good quality references while avoiding the time and effort to standardize reference sources to a common layout, design, and test matching rules, design blocking keys, or test blocking alignment. Also, the paper proposes how unsupervised data quality improvement processes could also be incorporated into the design allowing the model to address an even broader range of data curation applications.

Information fusion is the process of combining different sources of information for use in a particular application. The production of almost every information product incorporates some level of data fusion. Poor implementation of data and information fusion will have an impact on many other key data processes, most particularly data quality management, data governance, and data analytics. In this chapter we focus on a particular type of data fusion process called entity-based data fusion (EBDF) and on the application of EBDF in high-risk applications where accuracy of the fusion must be very high. One of the foremost examples is in healthcare. Fusing information belonging to different patients or failing to bring together all of the information for the same patient can both have dire, even life-threatening, implications.

Information Quality (IQ) is a core tenant of contemporary data management practices. Across many disciplines and industries, it has become a necessary process to improve value and reduce liability in data driven processes. Information quality is a multifaceted discipline with many degrees of complexity in implementation, especially in healthcare. Data profiling is one of the simpler tasks that an organization can perform to understand and monitor the intrinsic quality of its data. This case study demonstrates the application of core concepts of data profiling to entity resolution of multi-institutional Electronic Health Record (EHR) data. We discuss the benefits of using data profiling to better understand quality issues and their impact on entity resolution and how data profiling might be augmented to increase utility to clinical data.

EHR-based phenotype development and validation are extremely time-consuming and have considerable monetary cost. The creation of a phenotype currently requires clinical experts and experts in the data to be queried. The new approach presented here demonstrates a computational alternative to the classification of patient cohorts based on automatic weighting of ICD codes. This approach was applied to data from six different clinics within the University of Arkansas for Medical Science (UAMS) health system. The results were compared with phenotype algorithms designed by clinicians and informaticians for asthma and melanoma. Relative to traditional phenotype development, this method shows potential to considerably reduce time requirements and monetary costs with comparable results.

Software development is increasingly moving toward distributed processing environments such as Hadoop MapReduce and Spark in order to exploit the power of parallel processing. At the same time, distributed processing environments present a new set of challenges for the software development life cycle. This is particularly true for application testing. Because distributed platforms are often designed to optimize performance by dynamically reconfiguring themselves at run time, creating and maintaining standard test cases can be difficult. Even creating a single event log or statistical summary is not that simple. This chapter presents a case study describing how these challenges were addressed in the development and deployment of a regression testing framework on the Hadoop MapReduce platform. Even though the framework was specifically built to support a large-scale entity resolution application, the design principles can be applied to almost any type of distributed processing application.

This paper presents ongoing research conducted through collaboration between the University of Arkansas at Little Rock and the Arkansas Department of Education to develop an entity resolution and identity management system. The process includes a multi-phase approach consisting of data-quality analysis, selection of entity-identity attributes for entity resolution, defined a rule set using the open source entity-resolution system named OYSTER and used entropy approach to identify the potential false positive and false negative. The research is the first known of its kind to evaluate privacy-enhancing, entity-resolution rule sets in a state education agency.

The pair-wise nature of Entity Resolution makes it impractical to perform on large datasets without the use of blocking. Many blocking techniques have been researched and applied to effectively reduce pair-wise comparisons in Boolean rule based systems while also providing 100 % match recall. However, these approaches do not always work when applied to probabilistic matching. This paper discusses an approach to blocking for probabilistic scoring rules through the use of match key indexing.

The pair-wise nature of Entity Resolution makes it impractical to perform on large datasets without the use of blocking. Many blocking techniques have been researched and applied to effectively reduce pair-wise comparisons in Boolean rule based systems while also providing 100 % match recall. However, these approaches do not always work when applied to probabilistic matching. This paper discusses an approach to blocking for probabilistic scoring rules through the use of match key indexing.

Entity Resolution is the process of determining if two information system records are referring to the same entities, and is a crucial part in Information Quality research. The ER process becomes exponentially more complex and time consuming as datasets approach Big Data volumes. Due to the special characters of transitive closure in Entity Resolution and high volume of input data, traditional ER pairwise matching algorithms are not able to solve the problem efficiently. This paper presents a methodology to perform Entity Resolution without pairwise matching using match keys. Transitive closure occurs when each input reference can potentially create more than one match key. This paper also introduces a novel distributed parallel transitive closure algorithm in Entity Resolution context and an optimized version, which applies the method on multiple match keys. The implementation of the methodology is built upon the Hadoop MapReduce for distributed computation.

This paper compares entity resolution results obtained by using both probabilistic and deterministic matching when applied to the deduplication of student enrollment data. The approach outlined in this paper uses deterministic matching to represent equivalence for the calculation of weights to be used in probabilistic matching based on the Fellegi-Sunter model.

This paper describes methodology for creating a truth set for the evaluation of entity resolution (ER) results. The methodology combines the techniques of benchmarking and truth set development into an iterative, easy to use process. The paper also describes how the truth set developed by the methodology can be applied to calculate five key measures of ER outcome.

This paper describes a process for developing and refining matching rules for Entity Resolution. The process is an iterative scheme that has two phases. The first phase tries to identify the primary entity identity attributes and baseline matching rules. The second phase consists of identifying and incrementally augmenting the baseline rules with supporting attributes in a way that incrementally reduces the number of false positive and false negative resolutions and ultimately brings the resolution results closer to the truth. The process design was implemented in the open source entity resolution system named OYSTER.