—Data quality assessment is a challenging but neces- sary task to ensure that business decisions that are derived from data can be trusted. A number of data quality metrics have been developed to measure dimensions like accuracy, completeness, and timeliness. The tool QuaIIe (developed as part of our previous research) facilitates the calculation of different data quality metrics on both, schema- and data-level, and for heterogeneous information systems. However, to gain meaningful results from the automatically calculated metrics, it is key that humans understand the results of such metrics. This understanding is specifically important when contextual information needs to be considered, which is not encoded in the data. In this paper, we present a visualization approach to enable human-centered data quality assessment across multiple dimensions and arbitrary complex data sources. The approach has been implemented as graphical user interface in QuaIIe.
Schema and data integration have been a challenge for more than 40 years. While data warehouse technologies are quite a success story, there is still a lack of information integration methods, especially if the data sources are based on different data models or do not have a schema. Enterprise Information Integration has to deal with heterogeneous data sources and requires up-to-date high-quality information to provide a reliable basis for analysis and decision making. The paper proposes virtual integration using the Typed Graph Model to support schema mediation. The integration process first converts the structure of each source into a typed graph schema, which is then matched to the mediated schema. Mapping rules define transformations between the schemata to reconcile semantics. The mapping can be visually validated by experts. It provides indicators and rules to achieve a consistent schema mapping, which leads to high data integrity and quality.
—Code generators often have something mystical about them. Especially undergraduate students, who can still remember their first steps in programming, become in awe when they hear the term ”Software Generator”. The paper is an attempt to take this awe away from the students and to show them, by means of a very simple example implementation with well-known tools and technologies, that software generators are not witches’ work, but a powerful, but easily understandable tool to support the software development process.
Experimental studies are based on data that, sometimes, needs to be manually created. Moreover, the data is handled in relational databases to exploit their capabilities of manipulating (i.e., sorting, combining, and inserting) data. In this study, we show how this approach was successful in solving a combinatorics challenge to create a data set used in a separate research study that involves all the possible card combinations of the SET game®. The data required for the study was very extensive. The exact number was unknown, as this is an open combinatorics question, but the estimate was in the order of hundreds of millions. We solved this challenge by using a relational database (i.e., MySQL) as a computational tool to generate the data set. Advanced SQL scripts, based on cross joins, were applied to generate all the data. Table partitioning was also applied to improve the database performance of tables whose number of records exceeded the size capability of the database table. The data set created from this project was then used to support a Web based user interface that collects data to be used in a separate research study based on the SET® game. Keywords-MySQL; partitioning; computation; cross join.
Big Data has been the subject of increased research since data has been termed the new oil for the 21st century. Recently, smart grids have been used by energy providers to store the massive amount of data that is generated at regular time intervals. K-Electric is one such company in Pakistan that provides the residents of Karachi City with electrical energy. The company stores their data in a Not only Structured Query Language (NoSQL) database, since the smart grid data has a high volume, accelerated velocity, and tremendous variety. Hence, we feel that we can provide an important comparison of NoSQL tools using this data. NoSQL tools have been actively used for storage purposes in the industry. Companies like eBay, GitHub, and Amazon have been using these tools for storage and analytical purposes alike. In this paper, we compare and analyze four different technologies: MySQL, MongoDB, MonetDB, and InfluxDB using the data generated by the smart grids of KElectric. Keywords–NoSQL; Big Data; RDBMS; Performance Comparison; Smart Grid.
Data stream management has received considerable attention lately, due to continuous sources of collection and devices and is delivered in a different variety of formats to be process within small period of time. Interesting objects will be extract from the continuous data stream by processing the skyline query. However, skyline query processing over data stream poses inherent challenges and demands and require non- traditional techniques. This paper reviews the aims and challenge of skyline approaches that are relevant for data stream. Moreover, studies on the types of queries, type of data and type of sliding window were presented. Finally, an interesting research topic over data stream that have not yet been explored is identified.
Knowledge discovery from academic articles has received increasing attention since full text has been made available by the development of the digital databases. In a corpus of scientific articles, documents are connected by citations and one document has two different parts in the corpus: citation context and autonomous text. We believe that the topic distributions of these two parts are different and related in a certain way. In the existing topic models, little effort is made to incorporate the citation context. In this paper, we propose a citation context topic model which considers the corpus at two levels: cited topic level and citing topic level, utilizing citation context extracted from the full text. Each document has two different representations in the latent topic space. We apply our model to a dataset of PubMed Central, where the full text is available from the XML data. The results clearly show that the citation context can help to discover the latent two-level topics and demonstrate a very promising knowledge discovery capability. Keywords-Topic model; Citation context; Knowledge Discovery; XML data.
—This paper presents an interactive tool developed for the search and exploration of named entities and their relationships. The tool sits on top of an entity-based search engine, which previously has extracted and indexed all entities from a potentially huge document corpus. Relatedness between different entities is calculated based on entity n-tuples in the document corpus. The relatedness measure between entities is calculated during indexing time, which makes the algorithm very fast and usable for interactive application. Furthermore, the user can search for entities and their relationships to other entities using an interactive auto-completion and suggestion service. Related entities can then be filtered further by a multi-prefix search as well as based on type restrictions from an existing classification taxonomy. Another powerful feature is the merging of multiple entities into a group which allows the extraction of entities related to this group. A graphical interface is proposed with an entity or entity group as a central point, surrounded by the most n -related entities, based on some restrictions formulated by the user.
Data is central to decision-making in enterprises and organizations (e.g., smart factories and predictive maintenance), as well as in private life (e.g., booking platforms). Especially in artificial intelligence applications, like self-driving cars, trust in data-driven decisions depends directly on the quality of the underlying data. Therefore, it is essential to know the quality of the data in order to assess the trustworthiness and to reduce the uncertainty of the derived decisions. In this paper, we present QuaIIe (Quality Assessment for Integrated Information Environments, pronounced [’kvAl@]), a Java-based tool for the domain-independent ad-hoc measurement of an information system’s quality. QuaIIe is based on a holistic approach to measure both schema and data quality and covers the dimensions accuracy, correctness, completeness, pertinence, minimality, and normalization. The quality measurements are presented as machineand human-readable reports, which can be generated periodically in order to observe how data quality evolves. In contrast to most existing data quality tools, QuaIIe does not necessarily require domain knowledge and thus offers an initial ad-hoc estimation of an information system’s quality.
Origin of digital artifacts is asserted by digital provenance information. Provenance information is queried for proof statement validations, failure analysis, as well as replication and attributio ...
head output the first part of files -n: print the first lines -n -: print all but the last lines tail -n: print the last lines -n + : print, starting from line wc print newline, word, and byte counts for each file -c: print byte counts -m: print the character counts -w: print the word counts -l: print the newline counts grep print lines matching a pattern -E: support extended regexp -i: ignore case -v: invert match -o: print only the matched part (output: one line per match) -f: obtain patterns from file -l: files with match -L: files without match -c: suppress normal output, instead count matching lines. -m: Stop reading a file after matching lines seq print a sequence of numbers -s: separator (default: \n) split split a file into pieces cut remove sections from each line of files -d: Use instead oft ab as field separator -f: select only fields in --output-delimiter= : use as output delimiter paste merge lines of files -d: use as output delimiter tr Translate, squeeze, and/or delete characters -c: use the complement of set1 -d: delete the characters in set1 -s: replace each sequence of a repeated character with a single character sort sort lines of text files -n: numeric sort -r: reverse sort -R: random shuffle -c: check, if sorted, do not sort -t: field separator -k : sort according to keydef : F[.C][OPTS][,F[.C][OPTS]] -u: output only the first of equal lines join join lines of two files on a common field
We describe a novel approach for cross-domain recommendation for research collaboration. We first constructed a large Neo4j graph database representing authors, their expertise, current collaborations, and general biomedical knowledge. This information comes from MEDLINE and from semantic relations extracted with SemRep. Then, by using an extended literature-based discovery paradigm, implemented with the Cypher graph query language, we recommend novel collaborations, which include author pairs, along with novel topics for collaboration and motivation for that collaboration. KeywordsResearch collaboration; Recommendation system; Literature-based discovery; Semantic MEDLINE; Graph
