Due to the popularity of JSON, several systems have been developed that store data in collections of JSON documents. Each system has developed its own query language, sometimes in an ad-hoc manner. This makes difficult to formally define and analyze query optimization techniques. We propose an algebra tailored to JSON documents. First, we argue that JSON is different from nested relations and XML and therefore requires its own solution. Then, we propose an algebra on 3 levels: the first level defines operators to manipulate individual documents, providing an abstraction over different serializations. The second level provides operators over collections of JSON documents, while the third level defines also collection operators which are not primitive, but that enable direct and efficient implementation of data manipulation operations. We provide a number of properties of the algebraic operators which provide a solid basis for query optimization.

With the transformation to smarter cities and the development of technologies, a large amount of data is collected from sensors in real time. Services provided by ride-sharing systems such as taxis, mobility-on-demand autonomous vehicles, and bike sharing systems are popular. This paradigm provides opportunities for improving transportation systems’ performance by allocating ride-sharing vehicles toward predicted demand proactively. However, how to deal with uncertainties in the predicted demand probability distribution for improving the average system performance is still a challenging and unsolved task. Considering this problem, in this work, we develop a data-driven distributionally robust vehicle balancing method to minimize the worst-case expected cost. We design efficient algorithms for constructing uncertainty sets of demand probability distributions for different prediction methods and leverage a quad-tree dynamic region partition method for better capturing the dynamic spatial-temporal properties of the uncertain demand. We then derive an equivalent computationally tractable form for numerically solving the distributionally robust problem. We evaluate the performance of the data-driven vehicle balancing algorithm under different demand prediction and region partition methods based on four years of taxi trip data for New York City (NYC). We show that the average total idle driving distance is reduced by 30% with the distributionally robust vehicle balancing method using quad-tree dynamic region partitions, compared with vehicle balancing methods based on static region partitions without considering demand uncertainties. This is about a 60-million-mile or a 8-million-dollar cost reduction annually in NYC.

We design and implement an experimental analysis comparing a relational system (PostgreSQL) and two document-based NoSQL systems (MongoDB and Couchbase) on a single server, Decision Support (DSS) scenario. We argue that DSS is becoming an important case study for NoSQL. We experiment with several database designs and several query formulations in order to investigate the effect of physical design and query optimization in document-based stores. Our results show that design is very important for document stores performance, and that query optimization over documents is less sophisticated than in relational systems and needs to improve. Our results offer some guidance in this area.

We design and implement an experimental analysis comparing two relational systems (PostgreSQL and MariaDB) and two document-based NoSQL systems (MongoDB and CouchBase). We compare their performance on a single server, Decision Support (DSS) scenario. We argue that DSS is becoming an important case study for NoSQL. We experiment with several database designs and several query translations in order to investigate the effect of physical design and query optimization in document-based stores. Our results show that design is very important for MongoDB's performance, and that query optimization over documents is much less sophisticated on document-based stores than in relational data bases and needs to improve. Our results also offer some ideas to guide further development in this area.

Finding relevant objects in a large repository is a fundamental research problem occurring in many applications, such as: data cleaning, data integration, web search, and information retrieval. Instant type-ahead fuzzy search, where user types her query character by character and find the top-k relevant objects, has become widely involved in many applications because it provides the users with rapid response results and improves the user’s experience. The state-of-the-art algorithms are generally inefficient due to their breadth first search algorithm that results in repeated computations.To this end, we propose a novel depth-oriented instant type-ahead fuzzy search algorithm, that largely avoids repeated computations. The efficiency and effectiveness of the proposed approach are empirically demonstrated using real-world datasets. Experimental results show that our approach is 5–10 times faster than state-of-the-art approaches.

Building energy simulation plays an increasingly important role in building design and operation. This paper presents an open computing infrastructure, Virtual Information Fabric Infrastructure (VIFI), that allows building designers and engineers to enhance their simulations by combining empirical data with diagnostic or prognostic models. Based on the idea of dynamic data-driven application systems (DDDAS), the VIFI infrastructure complements conventional data-centric sharing strategies and addresses key data-sharing concerns such as the privacy of building occupants. To demonstrate the potential of the VIFI infrastructure, an empirically derived lighting schedule in the US Department of Energy's small office building reference model is simulated. The case-study simulation is used to explore (1) the possibility and potential of integrating data-centric and analytic-centric sharing strategies; (2) the method of combining empirical data with simulations; (3) the creation, sharing, and execution of analytics using VIFI; and (4) the impact of incorporating empirical data on energy simulations. Although the case study reveals clear advantages of the VIFI data infrastructure, research questions remain surrounding the motivation and benefits for sharing data, the metadata that are required to support the composition of analytics, and the performance metrics that could be used in assessing the applications of VIFI.

Autism Spectrum Disorder, ASD, is prevalent lifelong mental disorder associated with difficulties i n s ocial interaction, communication skills, and behavioral patterns. Numerous studies were presented suggesting abnormal development of neural networks in the brain in shape and connectivity. Magnetic resonance imaging, MRI, is used to diagnose the structural (i.e., sMRI) the functionality (i.e., fMRI) of gray matter and connectivity of the white matter (WM), using diffusion tensor imaging, DTI. The goal of this paper is to integrate these perspective to allow a comprehensive and personalized computer aided diagnostic (CAD) for ASD patients.

Big data analytics traditionally involves download of massive amounts of datasets to common server/cluster for processing. Analytic process gets slower with increasing size of required data and network conditions. Data scientists also need explicit access to data locations to download required data. Explicit access to required data may not always be granted due to security reasons. To simplify and accelerate the analytics process on distributed big data with security considerations, we proposed the Virtual Information Fabric Infrastructure (VIFI) for data driven discoveries. Instead of moving large amounts of data to a common place of processing, VIFI allows automatic transfer of required analytics programs to the distributed data locations for in-place processing of relevant data. VIFI allows data scientists to conduct and coordinate complex analytics processes on distributed data repositories using containerization technology and open-source workflow design tools. VIFI alleviates users from having detailed knowledge of distributed data locations, as well as required dependencies, installation and configuration of analytical libraries. In this paper, we demonstrate our current and future work to improve the VIFI architecture using previous and additional uses cases, data management layer that simplifies search of relevant data sets through addition of metadata, integration with security policies at different institutions with the proposed VIFI security layer, and the use of a user-friendly web interface to carry different VIFI activities.

With the transformation to smarter cities and the development of technologies, a large amount of data is collected from sensors in real-time. This paradigm provides opportunities for improving transportation systems' performance by allocating vehicles towards mobility predicted demand proactively. However, how to deal with uncertainties in demand probability distribution for improving the average system performance is still a challenging and unsolved task. Considering this problem, in this work, we develop a data-driven distributionally robust vehicle balancing method to minimize the worst-case expected cost. We design an efficient algorithm for constructing uncertainty sets of random demand probability distributions, and leverage a quad-tree dynamic region partition method for better capturing the dynamic spatial-temporal properties of the uncertain demand. We then prove equivalent computationally tractable form for numerically solving the distributionally robust problem. We evaluate the performance of the data-driven vehicle balancing framework based on four years of taxi trip data for New York City. We show that the average total idle driving distance is reduced by 30% with the distributionally robust vehicle balancing method using quad-tree dynamic region partition method, compared with vehicle balancing solutions based on static region partitions without considering demand uncertainties. This is about 60 million miles or 8 million dollars cost reduction annually in NYC.

We demonstrate a KDBMS, a prototype system which seamlessly integrates Knowledge base and DBMS. While state-of-the-art approaches, i.e., Ontology-based data access, denoted as OBDA, use ontologies to only query data stored in relational databases using SPARQL. In this demo, we present a high level description of the proposed system, introduce a new knowledge-based query language, denoted as KQL, and highlight some query optimization opportunities by employing knowledge across database layers in query optimization, and query processing, while ease the administrating for a complex database schema.

Electronic health record (EHR) solutions are complex, spanning multiple specialties and domains of expertise. These systems need to handle clinical concepts, temporal data, documents, and financial transactions, which leads to a large code base that is tightly coupled with data models and inherently hard to maintain. These difficulties can greatly increase the cost of developing EHR systems, result in a high failure rate of implementation, and threaten investments in this sector. Moreover, due to the wide variance in the level of detail across different settings, data exchange is becoming a serious problem, further increasing the cost of development and maintenance. To overcome these issues, we adopt ontologies to model our proposed EHR solution, not only allowing code reuse; but also enabling later extension and customization. Adopting software factory techniques, we build tools to transform ontological models into deployment-ready code. This automatically provides handling of data persistence, access, and exchange. Business logic is expressed as ontology-based process flows and rules, ensuring data quality and supporting special needs. This logic is enforced transparently and can be modified on the fly. We optimized the user experience by facilitating fast data entry and retrieval. In this paper, we present the requirements of an effective EHR solution, explain the techniques we employed, describe the main modules of our proposed system, and discuss the technical decisions we made.

A few decades ago, the Internet was created. Since then, searching for information and services has increased exponentially. With the introduction of GPS-enabled devices, a special type of search appeared offering location-aware services. These services customize search results based on users' location. This includes, but not limited to, (1) service finding, e.g., "find the nearest pizza restaurant", (2) routing, e.g., "obtain the shortest path from a user's home to the airport", (3) transportation, e.g., "what are the bus links to get a user from downtown to the mall", and (4) monitoring, e.g., "alert a parent if their child school-bus deviates from its regular route". Though new hardware and software technologies such as smart watches, voice search, and big-data platforms have been introduced and widely used, each single type of the above services has benefited very little from these technologies. On the local level of each service (the micro level), a full-fledged view is still missing. On the global level of all service types (the macro level), all Location-aware services are still acting as isolated islands and a global optimized service is not available. This paper presents our vision of how to provide an integrated macro location-aware service that acts harmoniously, and how each micro service can be further improved by better incorporation of novel technologies. We also overview the key challenges associated with these suggested improvements. Then, we highlight the potential value-added by the application of our vision.

Traffic congestion is a major problem in many big cities around the world. According to a study performed by the world bank in Egypt in 2010 and concluded in 2012, the traffic congestion was estimated to 14 Billion EGP in the Cairo metropolitan area and to 50 Billion EGP (4\% of the GDP) in the entire Egypt. Few of the reasons of the high monetary cost of the traffic congestion are: (1) travel time delay, (2) travel time unreliability, and (3) excess fuel consumption. Smart traffic management addresses some of the causes and consequences of traffic congestion. It can predict congested routes, take preventive decisions to reduce congestion, disseminate information about accidents and work zones, and identify the alternate routes that can be taken. In this project, we develop a real-time and scalable data storage and analysis framework for traffic prediction and management. The input to this system is a stream of GPS and/or cellular data that has been cleaned and mapped to the road network. Our proposed framework allows us to (1) predict the roads that will suffer from traffic congestion in the near future, and traffic management decisions that can relieve this congestion; and (2) a what-if traffic system that is used to simulate what will happen if a traffic management or planning decision is taken. For example, it answers questions, such as: "What will happen if an additional ring road is built to surround Cairo?" or "What will happen if point of interest X is moved away from the downtown to the outskirts of the city. This framework has the following three characteristics. First, it predicts the flow of the vehicles in the road based on historical data. This is done by tracking vehicles every day trajectories and using them in a statistical model to predict the vehicles movement on the road. It then predicts the congested traffic zones based on the current vehicles in the road and their predicted paths. Second, historical traffic data are heavily exploited in the approach we use to predict traffic flow and traffic congestion. Therefore, we develop new techniques to efficiently store traffic data in the form of graphs for fast retrieval. Third, it is required to update the traffic flow of vehicles and predict congested areas in real-time, therefore we deploy our framework in the cloud and employ optimization technique to speedup the execution of our algorithms.

The so-called smart grid is emerging in the energy domain as a solution to provide a stable, efficient and sustainable energy supply accommodating ever growing amounts of renewable energy like wind and solar in the energy production. Smart grid systems are highly distributed, manage large amounts of energy related data, and must be able to react rapidly (but intelligently) when conditions change, leading to substantial real-time business intelligence challenges. This paper discusses these challenges and presents data management solutions in the European smart grid project MIRABEL. These solutions include real-time time series forecasting, real-time aggregation of the flexibilities in energy supply and demand, managing subscriptions for forecasted and flexibility data, efficient storage of time series and flexibilities, and real-time analytical query processing spanning past and future (forecasted) data. Experimental studies show that the proposed solutions support important real-time business intelligence tasks in a smart grid system.

Personalized database systems give users answers tailored to their personal preferences. While numerous preference evaluation methods for databases have been proposed (e.g., skyline, top-k, k-dominance, k-frequency), the implementation of these methods at the core of a database system is a double-edged sword. Core implementation provides efficient query processing for arbitrary database queries, however, this approach is not practical since each existing (and future) preference method requires implementation within the database engine. To solve this problem, this article introduces FlexPref, a framework for extensible preference evaluation in database systems. FlexPref, implemented in the query processor, aims to support a wide array of preference evaluation methods in a single extensible code base. Integration with FlexPref is simple, involving the registration of only three functions that capture the essence of the preference method. Once integrated, the preference method “lives” at the core of the database, enabling the efficient execution of preference queries involving common database operations. This article also provides a query optimization framework for FlexPref, as well as a theoretical framework that defines the properties a preference method must exhibit to be implemented in FlexPref. To demonstrate the extensibility of FlexPref, this article also provides case studies detailing the implementation of seven state-of-the-art preference evaluation methods within FlexPref. We also experimentally study the strengths and weaknesses of an implementation of FlexPref in PostgreSQL over a range of single-table and multitable preference queries.

Flexibility objects, objects with flexibilities in time and amount dimensions (e.g., energy or product amount), occur in many scientific and commercial domains. Managing such objects with existing DBMSs is infeasible due to the complexity, data volume, and complex functionality needed, so a new kind of flexibility database is needed. This paper is the first to consider flexibility databases. It formally defines the concept of flexibility objects (flex-objects), and provide a novel and efficient solution for aggregating and disaggregating flex-objects. This is important for a range of applications, including smart grid energy management. The paper considers the grouping of flex-objects, alternatives for computing aggregates, the disaggregation process, their associated requirements, as well as efficient incremental computation. Extensive experiments based on data from a real-world energy domain project show that the proposed solution provides good performance while still satisfying the strict requirements.

The so-called smart grid is emerging in the energy domain as a solution to provide a stable, efficient and sustainable energy supply accommodating ever growing amounts of renewable energy like wind and solar in the energy production. Smart grid systems are highly distributed, manage large amounts of energy related data, and must be able to react rapidly (but intelligently) when conditions change, leading to substantial real-time business intelligence challenges. This paper discusses these challenges and presents data management solutions in the European smart grid project MIRABEL. These solutions include realtime time series forecasting, real-time aggregation of the flexibilities in energy supply and demand, managing subscriptions for forecasted and flexibility data, efficient storage of time series and flexibilities, and realtime analytical query processing spanning past and future (forecasted) data. Experimental studies show that the proposed solutions support important real-time business intelligence tasks in a smart grid system.

We demonstrate TimeTravel, an efficient DBMS system for seamless integrated querying of past and (forecasted) future values of time series, allowing the user to view past and future values as one joint time series. This functionality is important for advanced application domain like energy. The main idea is to compactly represent time series as models. By using models, the TimeTravel system answers queries approximately on past and future data with error guarantees (absolute error and confidence) one order of magnitude faster than when accessing the time series directly. In addition, it efficiently supports exact historical queries by only accessing relevant portions of the time series. This is unlike existing approaches, which access the entire time series to exactly answer the query. To realize this system, we propose a novel hierarchical model index structure. As real-world time series usually exhibits seasonal behavior, models in this index incorporate seasonality. To construct a hierarchical model index, the user specifies seasonality period, error guarantees levels, and a statistical forecast method. As time proceeds, the system incrementally updates the index and utilizes it to answer approximate and exact queries. TimeTravel is implemented into PostgreSQL, thus achieving complete user transparency at the query level. In the demo, we show the easy building of a hierarchical model index for a real-world time series and the effect of varying the error guarantees on the speed up of approximate and exact queries.

Preference queries are essential to a wide spectrum of applications including multi-criteria decision-making tools and personalized databases. Unfortunately, most of the evaluation techniques for preference queries assume that the set of preferred attributes are stored in only one relation, waiving on a wide set of queries that include preference computations over multiple relations. This paper presents PrefJoin, an efficient preference-aware join query operator, designed specifically to deal with preference queries over multiple relations. PrefJoin consists of four main phases: Local Pruning, Data Preparation, Joining, and Refining that filter out, from each input relation, those tuples that are guaranteed not to be in the final preference set, associate meta data with each non-filtered tuple that will be used to optimize the execution of the next phases, produce a subset of join result that are relevant for the given preference function, and refine these tuples respectively. An interesting characteristic of PrefJoin is that it tightly integrates preference computation with join hence we can early prune those tuples that are guaranteed not to be an answer, and hence it saves significant unnecessary computations cost. PrefJoin supports a variety of preference function including skyline, multi-objective and k-dominance preference queries. We show the correctness of PrefJoin. Experimental evaluation based on a real system implementation inside PostgreSQL shows that PrefJoin consistently achieves from one to three orders of magnitude performance gain over its competitors in various scenarios.

This article provides an overview of the CareDB context and preference-aware database system. CareDB provides efficient and scalable personalized query answers to users based on their preferences and current surrounding context. CareDB moves beyond the rigid query processing semantics of traditional relational database systems, which employ a boolean “all or nothing” query model, and addresses support for ”preference-aware” query processing methods. Specifically, CareDB supports a plethora of multi-objective preference methods capable of finding the “best alternatives” according to users’ given preference objectives. This article describes the architecture of the CareDB system and describes the details for three of its novel query processing characteristics: (1) a generic and extensible preferenceaware query processing engine, (2) a framework to gracefully handle contextual attributes that are expensive to retrieve, and (3) a framework to efficiently process queries over uncertain contextual data.