The stream data warehouse is an answer to the rapidly changing world of data analysis, which demands reliable and up-to-date results, obtained in a near real-time manner. Therefore it is a subject of recent research involving such areas as continuous updates and low-latency response, for example. In this paper, we study the stream adaptation of the OLAP cube and, in particular — its memory paging mechanism. It is driven by the page replacement algorithm, which manages the efficient data transfer and thus supplies users with constantly updatable data cubes. The following paper introduces an entirely novel approach to this topic. By perceiving the page replacement process as a multi-objective optimization problem, we propose three new algorithms that constantly analyze their varying environment and adapt to those changes by adjusting their behavior. Moreover, they consider user-provided constraints, which impose maximal values of specific parameters that cannot be exceeded. In addition to the page replacement algorithms, we propose two distinct quality of service metrics that measure the overall efficiency of data transfer inside the stream data warehouse. In order to verify and compare the new algorithms with their older counterparts, a series of experiments were conducted. Their results have confirmed that the proposed algorithms meet their requirements and visibly outperform the original solutions. The average wait time decreased between 25% and 66% (from 1.3x to 3.0x respectively, depending on the chosen algorithm), whereas the peak wait time decreased by approximately 99% (between 100x and 190x respectively).

The purpose of this paper is to highlight the performance issues of the matrix transposition algorithms for large matrices, relating to the Translation Lookaside Buffer (TLB) cache. The existing optimisation techniques such as coalesced access and the use of shared memory, regardless of their necessity and benefits, are not sufficient enough to neutralise the problem. As the data problem size increases, these optimisations do not exploit data locality effectively enough to counteract the detrimental effects of TLB cache misses. We propose a new optimisation technique that counteracts the performance degradation of these algorithms and seamlessly complements current optimisations. Our optimisation is based on detailed analysis of enumeration schemes that can be applied to either individual matrix entries or blocks (sub-matrices). The key advantage of these enumeration schemes is that they do not incur matrix storage format conversion because they operate on canonical matrix layouts. In addition, several cache-efficient matrix transposition algorithms based on enumeration schemes are offeredan improved version of the in-place algorithm for square matrices, out-of-place algorithm for rectangular matrices and two 3D involutions. We demonstrate that the choice of the enumeration schemes and their parametrisation can have a direct and significant impact on the algorithm's memory access pattern. Our in-place version of the algorithm delivers up to 100% performance improvement over the existing optimisation techniques. Meanwhile, for the out-of-place version we observe up to 300% performance gain over the NVidia's algorithm. We also offer improved versions of two involution transpositions for the 3D matrices that can achieve performance increase up 300%. To the best of our knowledge, this is the first effective attempt to control the logical-to-physical block association through the design of enumeration schemes in the context of matrix transposition.

Abstract In online gambling, poker hands are one of the most popular and fundamental units of the game state and can be considered objects comprising all the events that pertain to the single hand played. In a situation where tens of millions of poker hands are produced daily and need to be stored and analysed quickly, the use of relational databases no longer provides high scalability and performance stability. The purpose of this paper is to present an efficient way of storing and retrieving poker hands in a big data environment. We propose a new, read-optimised storage model that offers significant data access improvements over traditional database systems as well as the existing Hadoop file formats such as ORC, RCFile or SequenceFile. Through index-oriented partition elimination, our file format allows reducing the number of file splits that needs to be accessed, and improves query response time up to three orders of magnitude in comparison with other approaches. In addition, our file format supports a range of new indexing structures to facilitate fast row retrieval at a split level. Both index types operate independently of the Hive execution context and allow other big data computational frameworks such as MapReduce or Spark to benefit from the optimized data access path to the hand information. Moreover, we present a detailed analysis of our storage model and its supporting index structures, and how they are organised in the overall data framework. We also describe in detail how predicate based expression trees are used to build effective file-level execution plans. Our experimental tests conducted on a production cluster, holding nearly 40 billion hands which span over 4000 partitions, show that multi-way partition pruning outperforms other existing file formats, resulting in faster query execution times and better cluster utilisation.

Leaks and spills of hazardous fluids like petroleum endanger the environment, while remediation costs and penalties imposed when petroleum contaminates the ecosystem affect economics heavily. Therefore, it is crucial to detect any possible symptoms of a leak as soon as possible. Most of existing leak detection techniques require specialized equipment to be used, while purely software-based methods rely solely on data analysis and are very desirable since they can be deployed on petrol stations without any changes to the existing infrastructure. Moreover, such techniques can be considered as complementary to the hardware leak detection systems, as they provide additional security level. In this paper we present the TUBE algorithm, which detects fuel leaks from underground storage tanks, using only standard measurements that are normally registered on petrol stations, i.e. the amount of stored, sold, and delivered fuel. The TUBE algorithm is an autonomous solution capable of making decisions independently as well as supporting human-made decisions and thus can be considered as an expert leak detection system. The TUBE algorithm introduces a new data mining technique for trend detection and cleaning data over time series, which can be easily adapted to any other problem domain. A trend detection technique, called tubes, created for the TUBE algorithm is a novel data analysis method that allows to envelop uncertainties and oscillations in data and produce stable trends. Trend interpretation technique described in this paper has been designed especially for fuel leak detection purposes using our industrial experience. This paper includes a step-by-step usage example of the TUBE algorithm and its evaluation according to the United States Environmental Protection Agency requirements for leakage detection systems (the EPA SIR standard). Such an evaluation involves calculating the probability of detection and the probability of false alarm. The TUBE algorithm has obtained 98.84% probability of detection and 0.07% probability of false alarm while rejecting 42.22% of analyzed datasets due to their uncertainty. Rejecting datasets from analysis is compliant with the EPA SIR standard; however, rejection rate higher than 20% is not acceptable. Therefore we have evaluated the two-phase filtering stage of the algorithm in order to find the best combination of filters as means of data cleaning. Moreover, we have discussed the results pointing at the overall data quality problem, since it is the main cause of rejecting some datasets from the analysis. Finally, the TUBE algorithm has obtained 93.11% probability of detection and 0.73% probability of false alarm for the best combination of all parameters with 15.56% rejection rate, which is acceptable by the EPA SIR standard. The value of probability of detection is not fully compliant with the EPA SIR standard where 95% probability of detection with probability of false alarm lower than 5% is required. We have found that the requirements for the aforementioned probabilities have been completely fulfilled for datasets representing manifolded tank systems but not for single tank datasets. Such a situation was unexpected since manifolded tank systems are generally claimed to be more complex for analysis as they are in fact systems of multiple single tanks directly connected. In this paper we have also measured the time and memory complexity of the TUBE algorithm as well as discussed the issues connected to the TUBE algorithm deployment on petrol stations using our industrial experience in the topic. (C) 2017 The Authors. Published by Elsevier Ltd.

The fuel tank autocalibration problem is an important issue in managing the amount of fuel stored in the tank. Current values are calculated basing on fuel sold going out through nozzles - dispensing and fuel pumped into the tank by a tanker delivered. The difference in these values may point to different reasons - leakage, theft, or other errors. To pinpoint the cause it is important to rule out the case of wrong tank calibration, hence the tank autocalibration method is required. In this paper we present autocalibration method based on a neural networks algorithm, along with method's drawbacks and an alternative calibration method proposition.

In this paper we introduce a new approach to multiparty association rules mining based on a polynomial representation of sets encrypted with a homomorphic threshold cryptosystem. We describe a homogeneous collaborative multiparty association rules mining protocol that is secure in a malicious model. Presented algorithm is designed to enhance security and privacy in distributed environments where a malicious adversary may deviate arbitrarily from the prescribed protocol as it attempts to compromise the privacy of the other parties' inputs or the correctness of the obtained result. To the best of our knowledge, the protocol presented in this paper is the first multiparty association rules mining protocol that is secure against malicious adversaries in distributed systems.

The greatest threat to the environment and aquatic life is an uncontrolled fuel leakage, which is also extremely hazardous to health and safety of people. Guaranteeing the reliability of a leak detection system is probably the ultimate purpose of fuel management systems. However, there are more problems that ought to be solved before or simultaneously with detecting possible outflows of fuel products. In this paper we highlight major research opportunities consistent with wetstock management and statistical inventory reconciliation. The main goal is to outline thesis on the nature and impact of numerous phenomena on the inventory reconciliation methods. Issues considered in this paper include but are not limited to sensor miscalibration, data acquisition, and transmission problems as well as leak detection from both, tanks and connected pipeline.

The problem of low-latency processing of large amounts of data acquired in continuously changing environment has led to the genesis of Stream Processing Systems (SPS). However, sometimes it is crucial to process both historical (archived) and current data, in order to obtain full knowledge about various phenomena. This is achieved in a Stream Data Warehouse (StrDW), where analytical operations on both historical and current data streams are performed. In this paper we focus on Stream Materialized Aggregate List (StrMAL) – a stream repository tier of StrDW. As a motivating example, the liquefied petrol storage and distribution system, containing continuous telemetric data acquisition, transmission and storage, will be presented as possible application for Stream Materialized Aggregate List.

The systems dedicated for Zero-Latency Data Warehouses must meet the growing requirements for the most up-to-date data. The currently used sequential algorithms are not suited to deal with the pressure on receiving the freshest data. The one-module architecture implemented in current solutions, limits the development opportunities and increases the risk of critical system failure. In this paper we propose a new, innovative, multi-modular system that is based on parallel Choose Transaction by Election (CTBE) algorithm. Additionally we utilize the CUDA architecture to boost system efficiency, using computing power of multi-core graphic processors. The aim of this paper is to highlight pros and cons of such a solution. Performed tests and results show the potential and capabilities of the multi-modular system, using CUDA architecture.

Zero–Latency Data Warehouse (ZLDW) cannot be developed and formed on the basis of a standard ETL process, where time frames are limiting access to current data and blocking the ability to take users needs into account. Therefore, after profound analysis of this issue and ones related to workload balancing, an innovative system based on a Workload Balancing Unit (WBU) was created. In this paper we present innovative workload balancing algorithm – CTBE (Choose Transaction By Election), which allows to analyze all incoming transactions and create a schema of dependencies between them. Also, cache in the created WBU ensures ability to store information on incoming transactions and exchange messages with systems transmitting updates and users’ queries. By this work we intend to present an innovative system designed to support Zero–Latency Data Warehouse.

Matrix transposition is an essential mathematical operation broadly used in many scientific and engineering domains. In this paper, we present a new approach to the matrix transposition problem in massively parallel computing environments. We propose to utilize recursive functions to translate worker (thread) coordinates on a grid so that they can be used to locate and transpose matrix elements. In principal, it is possible to define a grid of threads in such a way that, based on their uniquely assigned index, we can decompose it using above mentioned functions so that each individual thread can exchange two relevant matrix elements across the main diagonal. Currently, our in-place method can be applied to arbitrary square matrices. In addition, we present a basic implementation of our algorithm for CUDA. Finally, we demonstrate initial performance tests performed on GPU and compare our results with naive versions of the in-place and out-of-place algorithms for CUDA.

Continuously growing importance of information assisted by rapid development of systems that collect and process huge volumes of data has become a great problem in terms of processing and analyzing data. The response to current and future needs of market is a data warehouse assisted by process of data extraction. Mentioned stream ETL process enables loading real-time data without interrupting processing or conducting analysis that supports decision-making processes. This paper presents first implementation of the stream ETL process which origins from model and concept of a Stream Data Warehouse. In the first part of this paper the concept of the Stream Data Warehouse and its major components, including stream ETL, will be presented. The second part contains description of a developed stream ETL engine, as well as results of performed accuracy and efficiency analysis. Finally, paper concludes with description of future research issues that will be addressed in further research on the presented solution.

This paper presents an index-based algorithm named SSAPP for exploring frequent sequential patterns in a distributed environment with privacy preservation. The SSAPP algorithm uses an equivalent form of a sequential pattern to reduce the number of cryptographic operations, such as decryption and encryption. In order to improve the efficiency of sequential pattern mining, the SSAPP algorithm keeps track of patterns in a tree data structure called SS-Tree. This tree is used to compress and represent sequences from a sequence database. Moreover, a SS-Tree allows one to obtain frequent sequential patterns without generation of candidate sequences. The conducted experiments show the effectiveness of the proposed approach. The SSAPP algorithm greatly reduces the number of cryptographic operations and it has good scalability.

In current international context boundaries set for applications are being pushed by the emergence of bursty and time-varying data streams required to be processed in near real-time. Furthermore, traditional techniques for data mining cannot be applied to data streams. Thus, stream-based applications must exhibit to excel at a plurality of requirements. According to defined rules presented in previous promulgated researches on this subject we differ stream-based applications and evaluate their aptitude to stream sources management. By this work we intend to present features and drawbacks of existing software coming from both industry and academic world, along with outlining our contribution to this field.

The growing importance of data warehousing [1-3] and the need to provide up-to-date information, changed procedures of data processing [4-8]. Classic data warehouses which are based on a traditional ETL process, proved to be ineffective and limited further development, due to the need of time-sharing of an access time between updates and analysis [9, 10]. Introduction of the zero-latency data warehouse, solved the problem of data mining time limit, however it enforces the need to use larger computing power for processing updates and queries in the ETL process. The article presents two ETL systems for zero-latency data warehouses which implement the WINE-HYBRIS algorithm. The first ETL system processes tasks in CUDA and CPU architectures, while the second uses Cloud Computing. The purpose of the article is to describe advantages and disadvantages of each solution.

Data mining applied to social media is gaining popularity. It is worth noticing that most e-commerce services also cause the formation of small communities not only services oriented toward socializing people. The results of their analysis are easier to implement. Besides, we can expect a better perception of the business by its own users, therefore the analysis of their behavior is justified. In the paper we introduce an algorithm which identifies particular customers among not logged or not registered users of a given e-commerce service. The identification of a customer is based on data that was given so as to accomplish selling procedure. Customers rarely use exactly the same identification data each time. In consequence, it is possible to check if customers create a group of unrelated individuals or if there are symptoms of social behavior.

The following paper introduces the performance evaluation of effective components in queries analysis and classification modules, as a part of the ETL management system. The main research focuses on usage of context-free grammars while analyzing queries arriving to the system. Also in the paper we present use of several methods of machine learning in a query processing time prediction. In previous research classifiers like Ridor were used, while query analysis was based on regular expression. Obtained results were quite encouraging and led to solutions presented in the paper.

This paper illustrates how CUDA can be successfully integrated into a RDBMS. It describes how heavyweight algebraic calculations can be efficiently performed by RDBMS by utilizing CUDA's programming model. We focus on the implementationof the complex user data type which is stored and managed by the RDBMS. In addition, we examine practicality of the custom aggregate functions and their performance when applied to CUDA powered user defined data types. We demonstrate this concept by using matrices and their multiplication as an example of a mathematical operation that requires great computational power which can be delivered by CUDA. This presented solution employs Microsoft SQL Server and the .NET platform.

The following paper describes some common aspects of stream data processing systems. The paper consists of two main parts – first showing the short description, tests results and conclusions of an implemented system – the AGKPStream, while the second part focuses on proposed solutions, created upon experiences gained during development of mentioned system, as well as knowledge collected during learning about some concepts of a StreamAPAS system. The first discussed issue is a tuple construction – basic data representation. It concerns tuple time model, tuple schema and a tuple decorator. Afterwards, the stream query and scheduling problems are described.

Currently used latency models in stream databases are based on the average values analysis that results from Little's law. The other models apply theory of M/G/1 queuing system. Theses solutions are fast and easy to implement but they omit the impact of streams synchronization. In this paper, we introduce a heuristic method which measures the synchronization impact. Then we have used this solution to extend the popular model based on average values analysis. This modification allows us to achieve better accuracy of latency estimation. Because schedulers and stream operator optimization require a fast and accurate model, we find our model a good starting point to create better optimizers.