Pattern-based aggregate (PBA) queries constitute an important and widely used type of analytical queries in sequence OLAP (S-OLAP) systems. Unfortunately, finding accurate answers to PBA queries in the S-OLAP system is often very expensive both in terms of time and memory consumption. In this paper we propose an efficient and easily maintainable index structure called the ABBA Index, which addresses the problem of PBA query processing. Experiments conducted using the KDD Cup data and public transport passengers' travel behavior data show that our index outperforms state-of-the art solutions while requiring much less memory. The ABBA Index can be easily extended to support pattern-based aggregate queries over hierarchy (PBA-H), a novel class of analytical queries which we introduce as the second main contribution of the paper. Sensitivity, scalability and complexity analysis of the ABBA Index is also provided.

In this paper, we investigate mining Mixed-Drove spatio-temporal Co-Occurrence Patterns (MDCOPs). MDCOPs represent sets of object types frequently located together for a given minimum fraction of time. Current solutions fail to address several important factors in practical applications. Specifically, state-of-the-art methods rely on a series of snapshots, i.e., discrete object positions recorded at predefined timestamps rather than their trajectories. However, spatio-temporal data gathering often depends on unsynchronized distributed sensors that independently register positions for each object.To tackle this issue using traditional methods, one can interpolate object positions at snapshot timestamps. However, this raises another challenge: determining the optimal number of snapshots while balancing accuracy, processing time, and memory requirements. To overcome these limitations, we formulate a generalized MDCOP mining problem and introduce GMDCOP-Miner, an algorithm that employs a new, generalized time-prevalence measure. The proposed algorithm provides the most accurate results, equal to those obtained via state-of-the-art methods with the number of snapshots tending to infinity. Moreover, our experiments demonstrate that GMDCOP-Miner surpasses existing approaches in both processing time and memory efficiency.

In this tutorial we present the results of researching, designing, implementing, and deploying data deduplication pipelines for customer records in a big financial institution. The tutorial is based on our experience gained within a R&D project. In the project we developed two deduplication pipelines. The first one is based on statistical modeling, whereas the second one is based on machine learning. Both pipelines were extensively tested on a real data set including customer records. The pipeline based on statistical modeling has already been deployed in the production system of the financial institution and processes batches of over 20 million of customer records.

Data lineage is an important part of today’s information systems. Several methods for providing means of recording and querying original sources for data stored in the system have been developed. Unfortunately, unless all of the system components provide support for recording/extracting and exchange of data lineage, obtaining full lineage information may be impossible. Consequently, lineage can sometimes be "broken." Possible causes for breaking lineage include: (1) usage of auxiliary data structures during processing, such as in-memory or temporary tables, (2) data cleaning performed semi-automatically and involving user intervention, and (3) ETL processes employing user-implemented “black box” steps which are not integrated into lineage tracking subsystem. In this paper, we present ideas for deducing lineage by observing data only. We assume almost no prior knowledge about the datasets and almost no meta-data available. We do not aim to provide a complete lineage tracking system, but rather provide a set of algorithms that can be integrated into existing data lineage systems and provide additional information about possible sources for data.

Data lineage is the set of techniques for tracking the flow of data throughout its lifecycle. These techniques are crucial for data management, governance, and compliance with regulations. Lineage links are maintained between data and database objects, but they are often broken by temporary objects and user defined functions. To the best of our knowledge, discovering broken lineage links has not been addressed yet in research. In this paper, we present a method for detecting broken lineage links between database objects. To this end we apply machine learning techniques on available metadata. We extract feature vectors and employ a classification approach to determine whether one database object is a source for another. Initial experiments on large database schemas show that the discovery of broken lineage links is possible at an acceptably high probability.

This paper is associated with a tutorial presented at DEXA 2025 Conferences and Workshops. The tutorial shares the practical experience gained from a 3-year R&D project for a big financial institution in Poland. The project aimed at developing deduplication pipelines for customer records. It involved the development of two distinct end-toend deduplication pipelines that are based on (1) statistical/probabilistic modeling and on (2) machine learning. This tutorial focuses on lessons learned from developing the machine learning pipeline, within the context of a real-world industrial setting. Moreover, this tutorial provides an overview of approaches to data deduplication, including the traditional state-of-the-art baseline deduplication pipeline, solutions based on machine learning and neural networks that apply pre-trained and large language models.

Co-location discovery plays an important role in spatial data mining. It aims to find types of objects that are frequently located together in a spatial neighborhood. A very popular interestingness measure for co-locations requires knowledge of all objects participating in co-location instances. A common requirement for co-location instances in the literature is that all objects contained in them are pairwise neighbors. Typically, this is determined in quadratic time with respect to the number of objects. In this paper, we introduce a new framework for determining pairwise neighborhoods in linear time. The framework utilizes a new metric that generates approximately the same neighborhoods as the Euclidean metric. We provide modifications of two algorithms for co-location instance identification that employ the proposed approach. Experiments performed on two real-world datasets demonstrate that we can achieve better processing times than using the state-of-the-art approach.

Data stored in information systems are often erroneous. Duplicate data are one of the typical error type. To discover and handle duplicates, the so-called deduplication methods are applied. They are complex and time costly algorithms. In data deduplication, pairs of records are compared and their similarities are computed. For a given deduplication problem, challenging tasks are: (1) to decide which similarity measures are the most adequate to given attributes being compared and (2) defining the importance of attributes being compared, and (3) defining adequate similarity thresholds between similar and not similar pairs of records. In this paper, we summarize our experience gained from a real R&D project run for a large financial institution. In particular, we answer the following three research questions: (1) what are the adequate similarity measures for comparing attributes of text data types, (2) what are the adequate weights of attributes in the procedure of comparing pairs of records, and (3) what are the similarity thresholds between classes: duplicates, probably duplicates, non-duplicates? The answers to the questions are based on the experimental evaluation of 54 similarity measures for text values. The measures were compared on five different real data sets of different data characteristic. The similarity measures were assessed based on: (1) similarity values they produced for given values being compared and (2) their execution time. Furthermore, we present our method, based on mathematical programming, for computing weights of attributes and similarity thresholds for records being compared. The experimental evaluation of the method and its assessment by experts from the financial institution proved that it is adequate to the deduplication problem at hand. The whole data deduplication pipeline that we have developed has been deployed in the financial institution and is run in their production system, processing batches of over 20 million of customer records.

PurposeTo understand how users of online marketplaces process market signals in their decision making and whether this depends on if the good is of high or low involvement.Design/methodology/approachThe paper employs a mixed methods approach. Study 1 draws on an analysis of interviews with online marketplace users using hypothetical eBay purchases as stimuli, understanding how users conceptualize specific market signals and whether their importance varies depending on the type of purchase (high versus low involvement good). Study 2 tests hypotheses derived from signaling theory, using an eye tracking experiment.FindingsPrice and photographs act as “fast and frugal” signals for inclusion in consideration sets for low involvement purchases, but consumers deem them insufficient for high involvement purchases where high-cost signals that help establish seller credibility are far more salient. Users pay relatively greater attention to costly market signals, which are beyond sellers’ direct control, for high involvement goods.Practical implicationsThe paper offers insights for sellers regarding the presentation of quality cues and strategies online marketplaces can employ to reduce information asymmetry.Originality/valueDrawing on and extending signaling theory, the paper introduces and confirms hypotheses for understanding users’ attention to market signals when making purchase decisions on online marketplaces. It identifies how the degree of involvement of a product affects the processing of market signals.

In this tutorial we present the results of researching, designing, implementing, and deploying data deduplication pipelines for customer records in a big financial institution. The tutorial is based on our experience gained within a R D project. In the project we developed two deduplication pipelines. The first one is based on statistical modeling, whereas the second one is based on machine learning. Both pipelines were extensively tested on a real data set including customer records. The pipeline based on statistical modeling has already been deployed in the production system of the financial institution and processes batches of over 20 million of customer records .

Co-location Pattern Mining (CPM) is the task of discovering sets of spatial features (object types) whose instances are frequently located close to each other in space. Popular co-location discovery methods consist of iteratively: (1) generating co-location candidates, (2) determining instances of these candidates and calculating a measure of potential interestingness, and (3) determining the set of co-locations based on that measure. In this paper, we focus on the second step, as it is the most time-consuming element of CPM. We assume that the distance function is either the Chebyshev or the Manhattan metric. We provide an instance identification method that is characterized by a lower complexity than the state-of-the-art approach. In particular, (1) we introduce a new representation of co-location instances based on bounding boxes, (2) we formulate and prove several theorems regarding such a representation that can improve instances identification step, (3) we provide a novel algorithm that uses the above-mentioned theorems, and (4) we analyze its complexity. To verify our approach, we performed a series of experiments using two real-world datasets.

In this paper, we summarize the results obtained while evaluating 44 similarity measures for text values, which represent real institutional customers data. These data come from a project conducted for a large financial institution in Poland. The similarity measures were assessed based on similarity values they returned and based on their execution times. To the best of our knowledge, it is the first report that evaluates such a large selection of different similarity measures.

Data stored in information systems are often erroneous. The most typical errors include: inconsistent, missing, and outdated values, typos as well as duplicates. To handle data of poor quality, data cleaning (a.k.a. curation) and deduplication (a.k.a. entity resolution) methods are used in projects realized by research and industry. Data deduplication is of particular challenge due to its computational complexity and the complexity of finding the most adequate method for comparing records and computing similarities of these records. The similarity value of two records is a compound value, whose computation is based on similarities of individual attribute values. To compute these similarities, multiple similarity measures were proposed in research literature and were implemented in various libraries (widely available in Python). For a given deduplication problem, a challenging task is to decide which similarity measures are the most adequate to given attributes being compared, since some similarity measures perform better than others for given characteristics of data being compared. In this paper, we report the experimental evaluation of 45 similarity measures for text values. The need to assess the measures came from a project conducted for a large financial institution in Poland. The measures were compared on five different real data sets, each of which had a different characteristic (e.g., text length, the number of words). The similarity measures were assessed (1) based on similarity values they produced for given values being compared and (2) based on their execution time. To the best of our knowledge, it is the first report that includes such a broad evaluation of a large selection of different similarity measures, on different real data sets.

Assuring high quality of data stored in information systems (ISs) is challenging and it is one of concerns of companies. Typically, data stored in ISs are not free from errors, which include among others wrong and missing values as well as duplicates. Data deduplication has received a lot of attention from the research community. The research efforts have resulted in a state-of-the-art data deduplication pipeline, supported by software tools and algorithms. One of the tasks in the pipeline consists in reducing the complexity of records comparisons. This task is known as blocking. Multiple algorithms for blocking have been proposed and one of them is the sorted neighborhood method. In this paper, we focus on tuning and evaluating the method on a real data set composed of 5.5M of customer records. To the best of our knowledge, this is the largest real data set being used in research. The findings reported in this paper come from a R D project run for a big company in a financial sector.

Assuring high quality of data stored in information systems (ISs) is challenging and it is one of concerns of companies. Typically, data stored in ISs are not free from errors, which include among others wrong and missing values as well as duplicates. Data deduplication has received a lot of attention from the research community. The research efforts have resulted in a state-of-the-art data deduplication pipeline, supported by software tools and algorithms. One of the tasks in the pipeline consists in reducing the complexity of records comparisons. This task is known as blocking. Multiple algorithms for blocking have been proposed and one of them is the sorted neighborhood method. In this paper, we focus on tuning and evaluating the method on a real data set composed of 5.5M of customer records. To the best of our knowledge, this is the largest real data set being used in research. The findings reported in this paper come from a R &D project run for a big company in a financial sector.

Effective heat energy demand prediction is essential in combined heat power systems. The algorithms considered so far do not sufficiently take into account the computational costs and ease of implementation in industrial systems. However, computational cost is of key importance in edge and IoT systems, where prediction algorithms are constantly updated with new arriving data. In this paper, we propose two types of algorithms for heat demands prediction: (1) novel extensions to the algorithm originally proposed by E. Dotzauer and (2) based on a kind of autoregressive predictor. They were developed within an R &D project for a company operating a cogeneration system and for their real dataset. We evaluate the algorithms experimentally focusing on prediction quality and computational cost. The algorithms are compared against two state-of-the art artificial neural networks.

Effective heat energy demand prediction is essential in combined heat power systems. The algorithms considered so far do not sufficiently take into account the computational costs and ease of implementation in industrial systems. However, computational cost is of key importance in edge and IoT systems, where prediction algorithms are constantly updated with new arriving data. In this paper, we propose two types of algorithms for heat demands prediction: (1) novel extensions to the algorithm originally proposed by E. Dotzauer and (2) based on a kind of autoregressive predictor. They were developed within an R&D project for a company operating a cogeneration system and for their real dataset. We evaluate the algorithms experimentally focusing on prediction quality and computational cost. The algorithms are compared against two state-of-the art artificial neural networks.

In this paper, we consider energy demand prediction in district heating systems. Effective energy demand prediction is essential in combined heat power systems when offering electrical energy in competitive electricity markets. To address this problem, we propose two sets of algorithms: (1) a novel extension to the algorithm proposed by E. Dotzauer and (2) an autoregressive predictor based on hour-of-week adjusted linear regression on moving averages of energy consumption. These two methods are compared against state-of-the-art artificial neural networks. Energy demand predictor algorithms have various computational costs and prediction quality. While prediction quality is a widely used measure of predictor superiority, computational costs are less frequently analyzed and their impact is not so extensively studied. When predictor algorithms are constantly updated using new data, some computationally expensive forecasting methods may become inapplicable. The computational costs can be split into training and execution parts. The execution part is the cost paid when the already trained algorithm is applied to predict something. In this paper, we evaluate the above methods with respect to the quality and computational costs, both in the training and in the execution. The comparison is conducted on a real-world dataset from a district heating system in the northwest part of Poland.