ABSTRACT It is not uncommon for database systems to have hundreds or even thousands of gigabytes of RAM at their disposal. High-performance systems such as HyPer [3], and ClickHouse [1] fully utilize the available memory and perform much better on analytical workloads than their traditional disk-based counterparts. Because these systems usually run on machines with such large memory capacities, the assumption is often that the workload fits in memory. While laptops have also enjoyed increased memory capacity, their physical design has limited space. Therefore they typically have only 16GB of memory. Laptops are often used in interactive data analysis, with tools like Pandas [5] and dplyr [8], showing that there is a need for analytical data management technology that runs on a laptop. However, these tools operate only in memory. As a result, users cannot process datasets that are slightly larger than memory, on their own machine. Disk-based database systems, on the other hand, have long solved the problem of processing larger-than-memory datasets. These systems are generally much slower than in-memory systems on analytical workloads. When a user wants to process a larger-thanmemory dataset using an in-memory system, usually one of two things happens 1) The system throws an error stating it is out of memory, 2) The system switches to an external strategy that is much less efficient than the in-memory strategy, which results in a slow execution time, even when, for example, the input is only 10% larger than memory. Fast queries may become slow or run into an error when a table grows in size, creating a frustrating experience for users. We can mitigate this problem by implementing operators such that they optimally use the amount of available memory and only write data to disk when this is necessary. I/O quickly becomes the bottleneck on machines with low-bandwidth storage devices. However, most modern laptops have nVME storage with high write speeds, making I/O less of a limiting factor. We have implemented a parallel, external sorting operator in DuckDB [7] that demonstrates this. Our implementation seamlessly transitions from in-memory to external sorting by storing data in buffer-managed blocks that are offloaded to disk using a leastrecently-used queue, similar to LeanStore [4]. Transitioning frommemory to disk ismade possible byDuckDB’s buffer manager and unified internal row layout, shown in Figure 1, which can be spilled to disk using pointer swizzling [2]. When, swizzled, pointers are replaced by relative offsets, that can easily be restored, shown in Figure 2. We compare our implementation against four other systems using an improvised relational sorting benchmark on two tables from TPC-DS [6]. Our implementation achieves excellent performance when data fits in memory and shows a graceful degradation in performance as we go over the limit of available memory. intA stringA intB stringB
It is common for machine learning applications to have complex operator pipelines, with several steps before and after the main execution of the core machine learning algorithm. Consequently, data movement between processes has a performance cost for the execution of the application. In this paper, we propose an end-to-end machine learning framework based on the principle of homoiconicity: data and code are represented in a single unified syntax. This approach allows the efficient interpretation of custom operators at the core of a relational database system. We also present the implementation of a data cleaning framework as an initial milestone. This work is a part of a larger research project to implement BOSS, a novel, general-purpose relational database system that stores and processes homoiconic data.
Organizations collect data from various sources, and these datasets may have characteristics that are unknown. Selecting the appropriate statistical and machine learning algorithm for data analytical purposes benefits from understanding these characteristics, such as if it contains temporal attributes or not. This paper presents a theoretical basis for automatically determining the presence of temporal data in a dataset given no prior knowledge about its attributes. We use a method to classify an attribute as temporal, non-temporal, or hidden temporal. A hidden (grouping) temporal attribute can only be treated as temporal if its values are categorized in groups. Our method uses a Ljung-Box test for autocorrelation as well as a set of metrics we proposed based on the classification statistics. Our approach detects all temporal and hidden temporal attributes in 15 datasets from various domains.
OrientDB is a full-function, NoSQL MMDMS (multi-model database management system), addressing the big data variety problem with one single, multi-model store and a SQL-based, multi-model query language. It combines graph and semistructured data management with object-oriented, text and spatial capabilities, and features a variety of deployment and distribution / replication options, transactional / ACID storage and indexing, making it a commercially successful MMDMS. With that, OrientDB is well-suited for novel adaptations of applications like smart logistics, asset management, social data storage and analysis, and other use cases that require multiple perspectives on the data. While OrientDB’s initial open source release was in 2010, many improvements like the multi-model API / query language and full ACID support came with a recent re-design. This paper gives the first comprehensive description of the revamped system, focusing on its data model, query language, distribution models and software architecture. We assess the current state of OrientDB’s performance compared to other multi-model, but also “best-in-class” NoSQL single-model document systems.
The impending information storage crisis is coming. Typical archival storage mediums must be replaced and re-written via expensive processes every 3-5 years. Exponentially increasing global data creation demands innovation in the methods we use to store this data. DNA oligos/origami offer a novel way to store information that is more durable, higher density, and scalable. With DNA information storage pipelines in their infancy, many areas of developmental progress are underway. One such area that requires attention is the back-end decoding and processing of the stored information. As the building material of DNA provides a continued and increasing scope to be used as the next-generation information storage medium, significant challenges arise. One such challenge is the cost-barrier to producing real microscopic images of folded DNA origami structures, which can amount to many thousands of pounds/dollars even if your institution is equipped with a well-resources bioengineering laboratory. Another barrier is time; real images can take weeks or months to produce. We therefore present a method for generating synthetic datasets for origami-based DNA information storage systems using a custom GAN pipeline. We find a customised ProjGAN the best performer for generating microscopy-based synthetic images of data stored in DNA origami structures. The work presented here therefore acts a useful tool for the further development of computational and back-end decoding pipelines for DNA origami information storage, that can now be developed without the aforementioned barriers. With DNA origami storage methods already here and many advancements on the horizon we are excited to present our synthetic data generation tool.
Decision modeling is a key aspect in modern BPM complementing and working alongside process models, such as BPMN. DMNisthemainstandardfordecisionmodelingandinthisworkweaimtotackleamaindrawbackofDMNs,namelythelack ofoptimizationtechniquesintermsofminimizingtheexecutiontimeandexecutioncost.WeaddressthislimitationthroughmappingDMNstoPDMs.PDMsaredeclarativeandemphasizeonthedatainputrequirementsforperformingoperationsto deriveadditionaldataelements,oneofwhichisthefinaltargetoftheprocess.Moreover,effectiveoptimizationheuristicshavebeenproposedforPDMs.Wealsopresenttwoillustrativeexamplesshowingthebenefitsstemmingfromourapproach.
In this work we introduce a new protocol for vector aggregation in the context of the Shuffle Model, a recent model within Differential Privacy (DP). It sits between the Centralized Model, which prioritizes the level of accuracy over the secrecy of the data, and the Local Model, for which an improvement in trust is counteracted by a much higher noise requirement. The Shuffle Model was developed to provide a good balance between these two models through the addition of a shuffling step, which unbinds the users from their data whilst maintaining a moderate noise requirement. We provide a single message protocol for the summation of real vectors in the Shuffle Model, using advanced composition results. Our contribution provides a mechanism to enable private aggregation and analysis across more sophisticated structures such as matrices and higher-dimensional tensors, both of which are reliant on the functionality of the vector case.
Data Wrangling (DW) is the subject of growing interest given its potential to improve data quality. DW applies interactive and iterative data profiling, cleaning, transformation, integration and visualization operations to improve the quality of data. Several domain independent DW tools have been developed to tackle data quality issues across domains. Using generic data wrangling tools requires a time-consuming and costly DW process often involving advanced IT knowledge beyond the skills set of traffic analysts. In this paper, we propose a conceptual approach to data wrangling for traffic data by creating a domain-specific language for specifying traffic data wrangling tasks and an abstract set of wrangling operators that serve as the target conceptual construct for mapping domain-specific wrangling tasks. The conceptual approach discussed in this paper is tool-independent and platform agnostic and can be mapped into specific implementations of DW functions available in existing scripting languages and tools such as R, Python, Trifacta. Our aim is to enable a typical traffic analyst without expert Data Science knowledge to be able to perform basic DW tasks relevant to his domain.
A pervasive problem in Data Science is that the knowledge generated by possibly expensive analytics processes is subject to decay over time as the data and algorithms used to compute it change, and the external knowledge embodied by reference datasets evolves. Deciding when such knowledge outcomes should be refreshed, following a sequence of data change events, requires problem-specific functions to quantify their value and its decay over time, as well as models for estimating the cost of their re-computation. Challenging is the ambition to develop a decision support system for informing re-computation decisions over time that is both generic and customisable. With the help of a case study from genomics, in this paper we offer an initial formalisation of this problem, highlight research challenges, and outline a possible approach based on the analysis of metadata from a history of past computations.
Query processing techniques have been shown to considerably reduce software development efforts for wireless sensor networks. However, deploying a sensor network remains a difficult and error-prone task. This paper presents a comprehensive tool to assist with the design of sensor network deployments, which takes into account a broader range of concerns than previous approaches. Users specify high-level requirements, such as location of nodes, using a simple graphical interface. The tool invokes a sensor network query processor to statically determine the validity of the deployment set out by the user for a given query workload, estimates performance in terms of the query processor's QoS metrics, and generates executable code.
Small credit-card-sized single-board computers, such as the Raspberry Pi, are becoming ever more popular in areas unrelated to the education of children, for which they were originally intended. So far, these computers have mainly been used in small-scale projects focusing very often on hardware aspects. We want to take single-board computer architectures a step further by showing how to deploy part of an orchestration platform (OpenStack Swift) on a Raspberry Pi cluster to make it a useful platform for more sophisticated data collection and analysis applications located at the edge of a cloud. Our results illustrate that this is indeed possible, but that there are still shortcomings in terms of performance. Nevertheless, with the next generation of small single-board computers that have been introduced recently, we believe that this is a viable approach for certain application domains, such as private clouds or edge computing in harsh environments.
User data can be acquired from various domains. This data is characterized by a combination of demographics such as age and occupation and user actions such as rating a movie, reviewing a restaurant or buying groceries. User data is appealing to analysts in their role as data scientists who seek to conduct large-scale population studies, and gain insights on various population segments. It is also appealing to novice users in their role as information consumers who use the social Web for routine tasks such as finding a book club or choosing a restaurant. User data exploration has been formulated as identifying group-level behavior such as Asian women who publish regularly in databases. Group-level exploration enables new findings and addresses issues raised by the peculiarities of user data such as noise and sparsity. I will review our work on one-shot [1–4] and interactive [5] user data exploration. I will then describe the challenges of developing a visual analytics tool for finding and connecting users and groups.
Cloud databases offer flexible off-premise data storage and data processing. Security requirements might however impede the use of cloud databases if sensitive or business-critical data are accumulated at a single cloud storage provider. Hence, partitioning the data into less sensitive fragments that are distributed among multiple non-communicating cloud storage providers is a viable method to enforce confidentiality constraints. In this paper, we express this enforcement as an integer linear program. At the same time visibility of certain data combinations can be enabled. Yet in case of violated visibility constraints, the number of different servers on which data is distributed can still be optimized. We introduce novel closeness constraints to express these requirements.
The development of smart city infrastructures is seen as an important strategic component for many countries with applications in the areas of government services, healthcare, transport and traffic management, energy, water, and the management of waste services. In general, each of these infrastructural components generates data as it delivers its service. The usage of this data is often crucial both to the continued development of the service and for forward planning. While the data sources are often not complex in structure, traditional decision support systems are not built to handle continuous data streams, new or disappearing data sources, or the integration of multiple online data sources. In this work, we develop a flexible ETL process to quickly process and integrate online smart city data sources to deliver information that can be used in close to real-time.
In this paper, we provide three authoritative application scenarios of TPC-H*d. The latter is a suitable transformation of TPC-H benchmark. The three application scenarios are (i) OLAP cube calculus on top of columnar relational DBMS, (ii) parallel OLAP data cube processing and (iii) virtual OLAP data cube design. We assess the effectiveness and the efficiency of our proposal, using open source systems, namely, Mondrian ROLAP server and its OLAP4j driver, MySQL -row oriented relational database management system and MonetDB -a column-oriented relational database management system.
Data wrangling is the process whereby data is cleaned and integrated for analysis. Data wrangling, even with tool support, is typically a labour intensive process. One aspect of data wrangling involves carrying out format transformations on attribute values, for example so that names or phone numbers are represented consistently. Recent research has developed techniques for synthesising format transformation programs from examples of the source and target representations. This is valuable, but still requires a user to provide suitable examples, something that may be challenging in applications in which there are huge data sets or numerous data sources. In this paper we investigate the automatic discovery of examples that can be used to synthesise format transformation programs. In particular, we propose an approach to identifying candidate data examples and validating the transformations that are synthesised from them. The approach is evaluated empirically using data sets from open government data.
Recent technology trends such as Cloud and Mobile Computing generate huge amounts of heterogeneous data to be processed by business and analytic applications. Currently, the data is stored in (relational) databases, before being queried by these applications. The heterogeneity of sender and application formats requires integration processes for decoupling sender from receiver (i.e., no shared schema), routing (i.e., "which application receives which parts of the data?") and format transformations. We argue that this kind of database (integration) process does not exist and propose a suitable processing model that fits the general integration semantics. The analysis is conducted for a real-world integration scenario on a well-established relational database system.
This paper argues that Data Analytics in the Web of Data asks for a metaquery language. One such language, based on second-order Description Logics, is sketched and illustrated with reference to a use case in the context of interest.
Real-time applications often analyze data coming from sensor networks using relational and domain-specific operations such as signal processing and machine learning algorithms. To support such increasingly important scenarios, many data management systems integrate with numerical frameworks like R. Such solutions, however, incur significant performance penalties as relational engines and numerical tools operate on fundamentally different data models with expensive inter-communication mechanisms. In addition, none of these solutions supports efficient real-time and incremental analysis. In this work, we advocate a deep integration of domain-specific operations into general-purpose query processors with the goal of providing unified query and data models for both online and offline processing. Our proof-of-concept system tightly integrates relational and digital signal processing operations and achieves orders of magnitude better performance than existing loosely-coupled data management systems.
Top-k shortest path routing problem is an extension of finding the shortest path in a given network. Shortest path is one of the most essential measures as it reveals the relations between two nodes in a network. However, in many real world networks, whose diameters are small, top-k shortest path is more interesting as it contains more information about the network topology. In this paper, we apply an efficient top-k shortest distance routing algorithm to the link prediction problem and test its efficacy. We compare the results with other base line and state-of-the-art methods as well as with the shortest path. Our results show that using top-k distances as a similarity measure outperforms classical similarity measures such as Jaccard and Adamic/Adar.
