Modern organizations amass their datasets into centralized repositories called data lakes, affording analytics as needed. The resultant scale and complexity of these data lakes, however, can make data navigation and monitoring challenging for users. We present DataCockpit, a Python toolkit that leverages datasets, usage logs, and associated meta-data to provision data usage and quality characteristics. DataCockpit computes these characteristics for each attribute (e.g., number of times it was queried for subsequent use in downstream applications) and record (e.g., number of non-missing, valid values) and aggregates them at the level of datasets. We develop a visual monitoring tool, powered by DataCockpit, and demonstrate how it can assist data / system administrators as well as end-users to effectively navigate and monitor a data lake. DataCockpit and the monitoring tool are available as open source software for developers to build custom monitoring applications on top of data lakes.

In database-as-a-service platforms, automated ver-ification of query equivalence helps eliminate redundant computation in the form of overlapping sub-queries. Researchers have proposed two pragmatic techniques to tackle this problem. The first approach consists of reducing the queries to algebraic expressions and proving their equivalence using an algebraic theory. The limitations of this technique are threefold. It cannot prove the equivalence of queries with significant differences in the attributes of their relational operators (e.g., predicates in the filter operator). It does not support certain widely-used SQL features (e.g., NULL values). Its verification procedure is computationally intensive. The second approach transforms this problem to a constraint satisfaction problem and leverages a general-purpose solver to determine query equivalence. This technique consists of deriving the symbolic representation of the queries and proving their equivalence by determining the query containment relationship between the symbolic expressions. While the latter approach addresses all the limitations of the former technique, it only proves the equivalence of queries under set semantics (i.e., output tables must not contain duplicate tuples). However, in practice, database applications use bag semantics (i.e., output tables may contain duplicate tuples) In this paper, we introduce a novel symbolic approach for proving query equivalence under bag semantics. We transform the problem of proving query equivalence under bag semantics to that of proving the existence of a bijective, identity map between tuples returned by the queries on all valid inputs. We classify SQL queries into four categories, and propose a set of novel category-specific verification algorithms. We implement this symbolic approach in SPES and demonstrate that it proves the equivalence of a larger set of query pairs (95/232) under bag semantics compared to the SOTA tools based on algebraic (30/232) and symbolic approaches (67/232) under set and bag semantics, respectively. Furthermore, SPES is 3X faster than the symbolic tool that proves equivalence under set semantics.

Predicate-centric rules for rewriting queries is a key technique in optimizing queries. These include pushing down the predicate below the join and aggregation operators, or optimizing the order of evaluating predicates. However, many of these rules are only applicable when the predicate uses a certain set of columns. For example, to move the predicate below the join operator, the predicate must only use columns from one of the joined tables. By generating a predicate that satisfies these column constraints and preserves the semantics of the original query, the optimizer may leverage additional predicate-centric rules that were not applicable before. Researchers have proposed syntax-driven rewrite rules and machine learning algorithms for inferring such predicates. However, these techniques suffer from two limitations. First, they do not let the optimizer constrain the set of columns that may be used in the learned predicate. Second, machine learning algorithms do not guarantee that the learned predicate preserves semantics. In this paper, we present SIA, a system for learning predicates while being guided by counter-examples and a verification technique, that addresses these limitations. The key idea is to leverage satisfiability modulo theories to generate counter-examples and use them to iteratively learn a valid, optimal predicate. We formalize this problem by proving the key properties of synthesized predicates. We implement our approach in SIA and evaluate its efficacy and efficiency. We demonstrate that it synthesizes a larger set of valid predicates compared to prior approaches. On a collection of 200 queries derived from the TPC-H benchmark, SIA successfully rewrites 114 queries with learned predicates. 66 of these rewritten queries exhibit more than 2X speed up.

Machine learning (ML) is increasingly being used to make decisions in our society. ML models, however, can be unfair to certain demographic groups (e.g., African Americans or females) according to various fairness metrics. Existing techniques for producing fair ML models either are limited to the type of fairness constraints they can handle (e.g., preprocessing) or require nontrivial modifications to downstream ML training algorithms (e.g., in-processing). We propose a declarative system OmniFair for supporting group fairness in ML. OmniFair features a declarative interface for users to specify desired group fairness constraints and supports all commonly used group fairness notions, including statistical parity, equalized odds, and predictive parity. OmniFair is also model-agnostic in the sense that it does not require modifications to a chosen ML algorithm. OmniFair also supports enforcing multiple user declared fairness constraints simultaneously while most previous techniques cannot. The algorithms in OmniFair maximize model accuracy while meeting the specified fairness constraints, and their efficiency is optimized based on the theoretically provable monotonicity property regarding the trade-off between accuracy and fairness that is unique to our system. We conduct experiments on commonly used datasets that exhibit bias against minority groups in the fairness literature. We show that OmniFair is more versatile than existing algorithmic fairness approaches in terms of both supported fairness constraints and downstream ML models. OmniFair reduces the accuracy loss by up to 94.8% compared with the second best method. OmniFair also achieves similar running time to preprocessing methods, and is up to 270x faster than in-processing methods.

Information technologies, such as building information modeling (BIM), significantly improve the clash detection process, but the clash resolution process is still time consuming. Although some studies have devoted attention to this field, they rarely discuss the dependency relations between clashes, which impact clash resolution in practice. This paper proposes to adopt graph theory to capture clash dependency and harnesses the information embedded in BIM models. This provides decision support for clash resolution, specifically focusing on automatically generating a component change list to minimize potential change impacts. This paper maps this situation as a minimum-weight vertex cover problem and discusses how to use component attributes to calculate vertex weights for approximating component change impact. Then, a branch and bound algorithm is designed to search for an optimal component change list. The paper tests the framework in two construction projects, and the results show the efficiency of the designed algorithm. The validity of the proposed method is further supported by comparing the list generated by the proposed method with actual project solutions.

Building information modeling has demonstrated its advantage to support design coordination, specifically for automatic clash detection. Detecting clashes helps us identify problems, but the process for solving these problems is still manual and time-consuming. This paper proposes using network theory to improve clash resolution by optimizing the clash correction sequence. Building systems are often interdependent of each other, and the dependency relations between building components propagate the impacts of clashes. Ignoring the dependency may cause new clashes when solving a clash or cause iterative adjustments for a single building component. However, a well-organized clash correction sequence can help reduce these issues. Therefore, it is necessary to holistically discuss the clash correction sequence by considering the dependence between clashes. This paper analyzes clash dependencies based on building component dependency relations. We design an optimization algorithm for determining the optimal sequence based on the clash dependency network to minimize feedback dependency, which may cause design rework on a project in project practice. The proposed method is validated on a real building project. After comparing with the natural sequence detected by commercial software, we find that the optimized sequence significantly reduces feedback and automatically groups dependent clashes, which facilitates design coordination.

Database-as-a-service offerings enable users to quickly create and deploy complex data processing pipelines. In practice, these pipelines often exhibit significant overlap of computation due to redundant execution of certain sub-queries. It is challenging for developers and database administrators to manually detect overlap across queries since they may be distributed across teams, organization roles, and geographic locations. Thus, we require automated cloud-scale tools for identifying equivalent queries to minimize computation overlap. State-of-the-art algebraic approaches to automated verification of query equivalence suffer from two limitations. First, they are unable to model the semantics of widely-used SQL features, such as complex query predicates and three-valued logic. Second, they have a computationally intensive verification procedure. These limitations restrict their efficacy and efficiency in cloud-scale database-as-a-service offerings. This paper makes the case for an alternate approach to determining query equivalence based on symbolic representation. The key idea is to effectively transform a wide range of SQL queries into first order logic formulae and then use satisfiability modulo theories to efficiently verify their equivalence. We have implemented this symbolic representation-based approach in EQUITAS. Our evaluation shows that EQUITAS proves the semantic equivalence of a larger set of query pairs compared to algebraic approaches and reduces the verification time by 27X. We also demonstrate that on a set of 17,461 real-world SQL queries, it automatically identifies redundant execution across 11% of the queries. Our symbolic-representation based technique is currently deployed on Alibaba's MaxCompute database-as-a-service platform.

Local transportation services are an essential component of urban infrastructures. The constantly growing requirements and expectations from several stakeholders, ranging from public authorities, operators and passengers, demand a deep rethinking of the entire system. Such services must be oriented towards sustainability in everyday life, requiring systemic approaches capable of fostering Quality of Service (QoS) improvements. This paper aims at highlighting the main facets of QoS from different perspectives (service providers', stakeholders', passengers' and commuters') and proposes a systemic modeling approach that supports the design, implementation and monitoring of a QoS system in smart mobility. The whole architectural framework is described and UCoMS, the platform for analyzing QoS, is proposed. The approach has been validated in the Apulia region in the South of Italy, using taxonomy of quality indicators and service levels on the basis of the European and Italian regulatory backgrounds. (C) 2019 Elsevier Inc. All rights reserved.

OBJECTIVE:To evaluate the prevalence of seven social factors using physician notes as compared to claims and structured electronic health records (EHRs) data and the resulting association with 30-day readmissions.STUDY SETTING:A multihospital academic health system in southeastern Massachusetts.STUDY DESIGN:An observational study of 49,319 patients with cardiovascular disease admitted from January 1, 2011, to December 31, 2013, using multivariable logistic regression to adjust for patient characteristics.DATA COLLECTION/EXTRACTION METHODS:All-payer claims, EHR data, and physician notes extracted from a centralized clinical registry.PRINCIPAL FINDINGS:All seven social characteristics were identified at the highest rates in physician notes. For example, we identified 14,872 patient admissions with poor social support in physician notes, increasing the prevalence from 0.4 percent using ICD-9 codes and structured EHR data to 16.0 percent. Compared to an 18.6 percent baseline readmission rate, risk-adjusted analysis showed higher readmission risk for patients with housing instability (readmission rate 24.5 percent; p < .001), depression (20.6 percent; p < .001), drug abuse (20.2 percent; p = .01), and poor social support (20.0 percent; p = .01).CONCLUSIONS:The seven social risk factors studied are substantially more prevalent than represented in administrative data. Automated methods for analyzing physician notes may enable better identification of patients with social needs.

The ADBIS conferences provide an international forum for the presentation of research on database theory, development of advanced DBMS technologies, and their applications. The 22nd edition of ADBIS, held on September 2–5, 2018, in Budapest, Hungary, includes six thematic workshops collecting contributions from various domains representing new trends in the broad research areas of databases and information systems.

The explosion of user generated content in social media published from mobile devices has led to the concept known as “citizen sensing.” Although English has been adopted by many as a de facto standard international language, reports about events, such as disasters, are frequently provided by citizens in their local language in addition to English. Attempting to integrate citizen reports from many languages is a significant challenge. This article describes the tools that address this challenge to enable the support of citizen-sensing of landslide events reported worldwide. Multilingual support is based on the first unified cross-lingual dataset of word vectors for representing texts in multiple languages. The classification model based on the proposed cross-lingual word vectors outperforms the “native” and “translated” approaches based on monolingual word vectors. Furthermore, it does not require the creation of a separate training set in a local language or its translation to English.

A considerable amount of research has addressed Internet of Things and connected communities. It is possible to exploit the sensing capabilities of connected communities, by leveraging the continuously growing use of cloud computing solutions and mobile devices. The pervasiveness of mobile sensors also enables the Mobile Crowd Sensing (MCS) paradigm, which aims at using mobile-embedded sensors to extend monitoring of multiple (environmental) phenomena in expansive urban areas. In this article, we discuss our approach with a cloud-based platform to pave the way for applying crowd sensing in urban scenarios. We have implemented a complete solution for environmental monitoring of several pollutants, like noise, air, electromagnetic fields, and so on in an urban area based on this paradigm. Through extensive experimentation, specifically on noise pollution, we show how the proposed infrastructure exhibits the ability to collect data from connected communities, and enables a seamless support of services needed for improving citizens’ quality of life and eventually helps city decision makers in urban planning.

Finding patterns in graphs has become a vital challenge in many domains from biological systems, network security, to finance (e.g., finding money laundering rings of bankers and business owners). While there is significant interest in graph databases and querying techniques, less research has focused on helping analysts make sense of underlying patterns within a group of subgraph results. Visualizing graph query results is challenging, requiring effective summarization of a large number of subgraphs, each having potentially shared node-values, rich node features, and flexible structure across queries. We present VIGOR, a novel interactive visual analytics system, for exploring and making sense of query results. VIGOR uses multiple coordinated views, leveraging different data representations and organizations to streamline analysts sensemaking process. VIGOR contributes: (1) an exemplar-based interaction technique, where an analyst starts with a specific result and relaxes constraints to find other similar results or starts with only the structure (i.e., without node value constraints), and adds constraints to narrow in on specific results; and (2) a novel feature-aware subgraph result summarization. Through a collaboration with Symantec, we demonstrate how VIGOR helps tackle real-world problems through the discovery of security blindspots in a cybersecurity dataset with over 11,000 incidents. We also evaluate VIGOR with a within-subjects study, demonstrating VIGOR's ease of use over a leading graph database management system, and its ability to help analysts understand their results at higher speed and make fewer errors.

Locating and extracting subgraphs from large network datasets is a challenge in many domains, one that often requires learning new querying languages. We will present the first demonstration of VISAGE, an interactive visual graph querying approach that empowers analysts to construct expressive queries, without writing complex code (see our video: https://youtu.be/l2L7Y5mCh1s). VISAGE guides the construction of graph queries using a data-driven approach, enabling analysts to specify queries with varying levels of specificity, by sampling matches to a query during the analyst's interaction. We will demonstrate and invite the audience to try VISAGE on a popular film-actor-director graph from Rotten Tomatoes.

Large data-rich network datasets are ubiquitous. Increasing research and development in graph databases offers analysts the ability to query large networks for interesting patterns. Graph querying allows analysts to quickly find matches for desired subgraphs from a large network (e.g., a clique of suspiciously communicating machines on a company intranet). Many of these systems do not support the exploration of results and leave the analyst with ineffective visualizations like tables and lists. We present our work in progress, VIGOR, an interactive visual system for summarizing and investigating large numbers of graph querying results.

OBJECTIVE:A significant challenge in treating rare forms of cancer such as Glioblastoma (GBM) is to find optimal personalized treatment plans for patients. The goals of our study is to predict which patients survive longer than the median survival time for GBM based on clinical and genomic factors, and to assess the predictive power of treatment patterns. METHOD:We developed a predictive model based on the clinical and genomic data from approximately 300 newly diagnosed GBM patients for a period of 2years. We proposed sequential mining algorithms with novel clinical constraints, namely, 'exact-order' and 'temporal overlap' constraints, to extract treatment patterns as features used in predictive modeling. With diverse features from clinical, genomic information and treatment patterns, we applied both logistic regression model and Cox regression to model patient survival outcome. RESULTS:The most predictive features influencing the survival period of GBM patients included mRNA expression levels of certain genes, some clinical characteristics such as age, Karnofsky performance score, and therapeutic agents prescribed in treatment patterns. Our models achieved c-statistic of 0.85 for logistic regression and 0.84 for Cox regression. CONCLUSIONS:We demonstrated the importance of diverse sources of features in predicting GBM patient survival outcome. The predictive model presented in this study is a preliminary step in a long-term plan of developing personalized treatment plans for GBM patients that can later be extended to other types of cancers.