Open data plays a critical role in advancing social good, yet access to this open data resource remains fragmented, with different publishers hosting data across various decentralized portals. This decentralization poses significant challenges for building unified search systems, primarily due to heterogeneity and, often, the low quality of metadata across these portals. In this paper, we address these challenges within the context of public dataset search by exploring the application of large language models (LLMs) in a zero-shot or few-shot manner to consolidate and enrich heterogeneous and incomplete metadata. We also investigate how LLMs can be utilized to rerank retrieved datasets, thereby enhancing search accuracy, and to provide relevance explanations to help users make more informed decisions. Using these techniques, we develop the China Public Data Search system (CN-PDS), a unified dataset search system that integrates datasets from public data portals of various provinces and cities in China. Through extensive experiments on China’s public data and other standard test collections for dataset search, we validate the effectiveness of our approach. Our empirical evaluation, coupled with the insights gained from this work, offers valuable lessons in improving real-world dataset search systems using LLMs.
Sustained emerging spatio-temporal co-occurrence patterns (SECOPs) represent subsets of spatial features, whose instances consistently appear in close proximity across both space and time. Discovering SECOPs is valuable for many applied fields. Existing studies of SECOP mining neglect the emergence of new instances over time in dynamic spatial databases, which may lead to complex changes in neighbor relationships of instances and heterogeneity of the change in the number of feature instances. Therefore, the deviation may be caused in the SECOP measure for dynamic spatial databases with new instances. To solve this problem, we propose the weighted emerging participation index (WEPI) that incorporates the effects of the change of neighbor relationships and the heterogeneity on the sustained emerging measure of SECOPs. Because WEPI does not possess the antimonotone property, we prove an upper bound of the weighted emerging participation ratio of features and introduce a pruning strategy to prune some candidate patterns. Building upon this pruning strategy, an incremental mining algorithm based on depth-first search is proposed, i.e., IM-DFS. In IM-DFS algorithm, a set of optimization techniques is presented to accelerate WEPI calculation. Extensive experiments on real-world dynamic spatial datasets demonstrate the effectiveness of WEPI and the efficiency of IM-DFS.
Join order selection is one of the key technologies in database query optimizers. Traditional approaches rely heavily on expert experiences, resulting in a prolonged development cycle. Heuristic methods based on predefined rules also fail to leverage past experiences. In recent years, reinforcement learning-based methods have emerged as a promising approach for query optimization. However, different segments of the join plan in SQL queries carry varying degrees of importance for execution latency, which can cause high variability in reward values. Such discrepancies pose challenges in generating robust and efficient query plans. This paper proposes LIGHT, a learned join order selection network that leverages a dynamic graph attention mechanism. This mechanism propagates and fuses table node vectors, allowing each node to focus on the features of its most relevant neighbors, thereby enhancing the query’s vector representation. Moreover, a beam-random search method with a weighted list is employed to retain more possibilities for exploring the best join order in reinforcement learning action selection. Finally, to mitigate reward volatility, the model introduces a logarithmic transformation to compress reward values. This accelerates the training process and promotes the generation of robust query plans. Extensive comparative experiments on the Join Order Benchmark and TPC-DS demonstrate that LIGHT significantly outperforms existing learned optimizers in terms of query execution performance.
Incomplete multi-view clustering (IMVC) is an important research topic in data mining due to the prevalence of incomplete multi-view data, where samples may be missing in particular views. Although many methods for IMVC have been proposed and achieved encouraging results, these methods ignore the neighbor information for samples with missing views and the global structure of multi-view data, and do not effectively utilize the pseudo labels generated during label allocation. To address these issues, we propose a multiple self-supervised IMVC method based on neighbor contrastive learning (NCMS). Firstly, the missing samples are completed through the transfer of neighbor relationships between views. Then, a two-level node representation learning strategy including view-specific and global-specific contrastive learning is designed to learn the sample features. Next, a multiple self-supervised strategy including intra-view and inter-view self-supervised learning is developed to learn the global representation suitable for clustering tasks. The experimental results on four incomplete multi-view datasets verify the effectiveness of the proposed method, especially on the 100Leaves dataset, when the missing rate is 50
Ultra-wideband (UWB) radios enable decimeter-level positioning. This is posited to unlock unprecedented sophistication and detail in indoor mobility analytics. However, to date industry and academia have resorted to primitive analyses based only on raw trajectories, unable to ascertain where individuals stop and for how long, crucial in many domains. Among these is the museum providing the real-world context and dataset for this paper. We deployed a UWB localization system in a 40 × 15 m^2 area containing 42 exhibits and track more than 1500 visitors over a 3 months period. We confirm that commonplace analyses relying on UWB trajectories alone offer practical insights, yet cannot capture the two key dimensions above. Instead, we exploit state-of-the-art techniques to extract higher-level semantic trajectories directly capturing visitor behavior, and distill a multitude of detailed, multi-layered, actionable insights. Our experience concretely highlights the untapped, disruptive potential of UWB-based mobility analytics—and a way to seize it. To foster further research, in museums and beyond, we publicly release our large UWB dataset (more than 9 million positions) along with the one resulting from our higher-level analyses.
Retrieval Augmented Generation (RAG) has emerged as a mainstream paradigm for mitigating hallucinations in large language models and supporting knowledge-intensive complex reasoning tasks. However, existing retrieval-as-context methods typically employ either one-shot retrieval or blind iteration strategies, lacking global planning and dynamic scheduling for heterogeneous knowledge. This often introduces substantial redundant information and disrupts the reasoning process. To address these issues, we propose structured memory scheduling retrieval augmented generation (SMS-RAG) framework. SMS-RAG explicitly models external knowledge as compound memories with semantic, structural, and episodic features. It introduces a structured scheduling mechanism to explicitly model and control the activation timing, sequence, and dependency relationships among different knowledge. This enables knowledge to participate in reasoning in a structured manner, rather than merely serving as passive contextual input. It consists of three modules: MemPlanner determines which memory modules to activate by generating a task-aware MemoryPlan; MemScheduler decides in what order to activate them by constructing a dependency-aware graph; and MemRouter executes how to activate by selectively routing across semantic, structural, and episodic memory layers to extract critical information for reasoning. Extensive experiments on three multi-hop QA benchmarks and a domain-specific legal QA dataset demonstrate that SMS-RAG significantly outperforms existing state-of-the-art methods in reasoning accuracy, factual coverage, and scheduling efficiency.
Trajectory similarity join is a fundamental problem in spatiotemporal data management, aimed at identifying pairs of similar trajectories within two trajectory datasets. With the growth of mobile devices, streaming trajectory data is continuously uploaded, enabling real-time trajectory stream analysis for location-based services. In this paper, we investigate the problem of Online Semantic Trajectory Similarity Join (OSTS-J). Given two streaming semantic trajectory datasets, a similarity threshold θ , and a time window L , OSTS-J returns all trajectory pairs within the current time window L that meet the threshold θ constraint. This problem has significant applications in real-time route planning and travel itinerary suggestions. To efficiently address the OSTS-J problem over large-scale semantic trajectory datasets in a distributed framework, we propose a two-stage algorithm, REST, based on the principle of “defer computation until necessary." In the first stage, we employ the AIR-index alongside pruning strategies to avoid unnecessary trajectory pair computations at the global level, ensuring effective load balancing. In the second stage, we use an enhanced version of incremental computation, Inc++, to further defer unnecessary refinement of trajectory pairs at the local level. We evaluate the effectiveness, scalability, and load-balancing performance of our framework through experiments on four real-world datasets. The results demonstrate that our algorithm (i) outperforms state-of-the-art methods by achieving approximately a 97 × increase in throughput under optimal conditions, and (ii) enhances load balancing performance and system stability.
Database systems play a vital role in today’s information society. As modern database systems are widely adopted across diverse domains, their workloads and data access patterns are becoming increasingly complex, posing new challenges for traditional buffer management mechanisms. Neither conventional buffer management strategies nor existing adaptive algorithms can effectively address a key issue: When data access hotspots shift dramatically, how to avoid a prolonged drop in the buffer hit rate and ensure that it recovers quickly? We refer to this challenge as the "buffer cold-start" problem. To address this problem, and considering the periodic nature of workloads in many real-world applications, we propose to adopt buffer prefetching to proactively load potential hot data pages. This approach shortens both the duration of buffer performance degradation and the time required for the buffer hit rate to recover during hotspot transitions, thereby effectively addressing the buffer cold-start problem under periodic workloads. In particular, we introduce pre-buffer, a workload-aware buffer prefetching framework. Pre-buffer identifies hot pages within a given worklaod and, upon detecting a workload change, uses similarity metrics to match and predict the hot page set of the new workload. An independent prefetching thread then loads these hot pages into the buffer in advance. Based on the characteristics of real database experimental scenarios, we design two representative workloads to evaluate our approach. Experimental results show that the proposed workload-aware buffer prefetching scheme significantly improves the recovery speed of the buffer hit rate under dynamic workload changes, thereby enhancing the overall performance of the database buffer system.
The rapid development of learned indexes brings a surge in the indexing performance of database management systems. It treats an index as a learned model and learns a mapping between keys and their values’ positions in the database. Existing works mainly focus on improving the regression accuracy of the model itself. However, we observe that the real-world query workload is biased and keys have different access frequencies, which indicates that keys are not equally important. Simply optimizing the regression accuracy by treating keys equivalently may not be the most effective for the workload. We propose the novel Workload-Aware Learned Tree Index (WALDEN) which sets weights to keys and arranges important keys into the shallower levels of the tree index. Specifically, an algorithm which minimizes keys’ weighted collision degree is devised to build this workload-aware tree index efficiently. Furthermore, we also design an index update mechanism to monitor and respond to any significant shift in the query workload distribution. Comprehensive evaluations are conducted on synthetic workloads on real datasets undercommon settings, which show that WALDEN outperforms the state-of-the-art learned indexes with an improvement between 9.7 and 500
In deep learning, the choice of optimizer plays an important role in the training efficiency and final performance of models. Adam is widely used because of its good convergence and stability. However, Adam may exhibit oscillations and fluctuations in the training when the gradient varies significantly, because its second-moment uses the square of the gradient. In addition, Adam suffers from slow convergence due to its sensitivity to sparse gradients changes and overly conservative learning rates. To address these issues, we propose an improved optimizer called AdamMAD (Mean Absolute Deviation). AdamMAD performs the second-moment by using the mean absolute deviation (MAD) instead of the traditional square of the gradient, resulting in smoother Training Loss, fewer spikes and valleys, and more friendly sparse gradients processing. This paper details the design principles and specific implementation of AdamMAD, derives its mathematical formulas, and validates its performance on various datasets and models through extensive experiments. Experimental results demonstrate that AdamMAD offers better stability and convergence in the training, compared to optimizers such as Adam, AdamW, and QHAdam that update the learning rate using the second-moment.
Large-scale, multi-dimensional mixed datasets are characterized by the pervasive "long-tail distribution." This phenomenon results in data sparsity in subspaces defined by multi-dimensional attribute combinations. This sparsity severely hinders data-driven analysis and insights. Existing data-augmentation methods primarily focus on single dimensions, ignoring the complex, intrinsic multi-dimensional correlations of the real world, which leads to generated samples that lack logic and realism. To address this challenge, we propose a systematic, on-demand, and fine-grained sample augmentation framework. Our core idea is to precisely locate and augment data-sparse regions from a multi-dimensional combinatorial perspective. For textual data, we have designe two flexible augmentation modes. The first, "augmentation by subset scope", adopts a strategy of model merging and incremental updates. The second, "augmentation by topic", proposes a heuristic search algorithm based on the "Explore-Exploit" paradigm. For numerical data, we pre-construct a global distribution index to achieve efficient identification of sparse intervals. In the sample generation phase, we combine Large Language Models, Retrieval-Augmented Generation, and Chain-of-Thought techniques to ensure that the generated samples meet high-fidelity standards in semantic logic and contextual style. Extensive experiments on real-world datasets demonstrate that our method outperforms baseline approaches in query response speed and the efficiency of sparse region discovery, while maintaining topic coherence and accuracy.
The network data model can effectively capture relationships between entities, making it useful for applications such as social networks and e-commerce systems. Anomaly detection, a critical task in network analysis, focuses on identifying unusual patterns that deviate from expected behavior and plays a key role in enhancing the reliability of decision-making processes. While most existing studies focus on detecting anomalies in homogeneous networks, modern applications often involve more complex networks characterized by heterogeneity (nodes and edges of different types), diversified node attributes, and edge multiplicity (node pairs with various edges). Motivated by these practical complexities, this paper investigates anomaly detection in Attributed Multiplex Heterogeneous Networks (AMHENs), which remains challenging due to two significant gaps: the seamless integration of AMHENs’ characteristics and the comprehensive assessment of the diverse anomalies. To close these gaps, we introduce a novel Augmentation-based Multi-channel Graph convolutional network (AMG) that incorporates integrated learning and anomaly scoring mechanisms. Firstly, a multi-channel graph convolutional network is devised to capture the diverse characteristics, which can deeply learn local and global information from AMHENs and aggregate them. Secondly, to comprehensively assess varios types of anomalies, we design a hierarchical network augmentation module and an integrated model optimization module to enable the model to score anomalies through the learning of multiple views. Extensive experiments demonstrate that AMG significantly outperforms competing baselines, achieving improvements of 17.4
Shortest path queries are among the most fundamental operations in graph processing. Prior studies have extensively examined label-constrained queries on static graphs with fixed edge weights. However, real-world networks exhibit temporal dynamics and often contain complex heterogeneous structures. Examples include peak-hour variations in travel time and the coordination of multiple transportation modes. These time-dependent and label-dependent characteristics limit the applicability of existing methods to the Time-Dependent Label-Constrained Shortest Path (TD-LCSP) problem. To address this limitation, we propose LP-Tree, a label-partitioned tree decomposition framework that models both label constraints and temporal dependencies in a unified manner. Based on this structure, we design the Parameter-Separated (PS) Index, which decouples key parameters to improve query efficiency and reduce storage costs. The combined approach significantly lowers space consumption and achieves sub-millisecond query latency on large-scale time-dependent graphs. Extensive experiments on multiple datasets demonstrate the effectiveness and efficiency of our proposed approach.
Transformers use a powerful self-attention mechanism to model remote dependency capabilities, showing great potential in various long-term time series prediction tasks. However, the original Transformer was not proposed for the time series prediction tasks and did not make any assumptions about temporal patterns and characteristics. Due to systematic differences and external factors, the heterogeneity of multi-dimensional time series data sets makes Transformer prediction face challenges. A large number of researchers lack inductive bias when modeling multi-dimensional time series and dealing with different structures or patterns. In this work, we addressed the heterogeneity of multivariate time series from both temporal and spatial perspectives, examined the characteristics of multivariate time series and the inherent biases of different models, and proposed IBformer, a time series Transformer with inductive bias. Specifically, we design a key sampling strategy to extend the receptive field of the model by introducing inductive bias, and perform discriminative modeling for different datasets based on decomposition, utilizing encoders and trend prediction blocks to capture seasonal patterns and overall trends, solving the problem of temporal heterogeneity. In addition, we adopt a channel-independence strategy and design a quadratic spatio-temporal enhancement block to correct the original prediction results, which solves the problem of weak temporal correlation caused by direct multi-step prediction. And capture spatial dependence on the premise of avoiding excessive spatial modeling, and solve the problem of spatial heterogeneity. Extensive experiments on seven real-world datasets show that IBformer outperforms all existing models and demonstrates strong versatility and generalization capabilities. The code is made available at https://github.com/shihaoyuan6/IBformer
With the continued digitization of social and industrial processes, diverse and increasingly massive volumes of spatial-temporal data are being accumulated， which enable the use of data mining to fuel important applications. Thus, neural network-based techniques have been developed to capture spatial and temporal dependencies in spatial-temporal data. Most recently, the advances in generative techniques, including large language models, masked autoencoders, sequence-to-sequence models, diffusion models, and others, have led to their increased use in spatial-temporal data mining, thereby driving new advances in spatial-temporal data mining. This paper describes the general types of spatial-temporal data, all kinds of spatial-temporal data instances, and generative techniques, and it proposes a spatial-temporal data mining pipeline. Further, it delivers a structured overview of generative techniques for spatial-temporal data mining, grounded in a novel taxonomy. Moreover, by outlining promising research avenues enabled by generative techniques, the paper seeks to accelerate advances in spatial-temporal data mining.
We study the clique problem on partite graphs. As a special type of clique, cliques on k-partite graphs have extensive applications in areas such as gene data analysis, data classification, protein function determination, recommendation systems, and social network analysis. However, existing research has largely focused on the maximum clique problem on k-partite graphs, neglecting other commonly occurring clique structures. To address this gap, for the first time, we define three special clique structures on k-partite graphs and analyze their interrelationships. We also formulate the clique enumeration problem on k-partite graphs and perform a comprehensive complexity analysis. Furthermore, we propose an efficient clique enumeration algorithm that takes advantage of the unique properties of cliques on k-partite graphs and integrates various techniques, such as pruning strategies and ordered sets, to accelerate the enumeration process. Experimental results on four real-world k-partite graphs validate the efficiency of the proposed algorithm and the effectiveness of the techniques employed.
Cross-modal hashing retrieval faces fundamental challenges from modality-modality (M-M) and modality-label (M-L) inconsistencies inherent in multimodal data. Existing methods rely on coarse-grained disentanglement to address these inconsistencies, but suffer from inaccurate semantic separation and modality-common semantic information loss during cross-modal alignment. Through comprehensive analysis, we demonstrate that coarse-grained approaches fail to effectively alleviate modality inconsistencies. Our validation experiments show that incorporating fine-grained features yields up to 6% accuracy improvements over coarse-grained methods, confirming that fine-grained semantic components are critical for robust cross-modal retrieval. However, existing fine-grained methods require extensive pre-training and lack seamless integration into end-to-end frameworks. In this paper, we propose Inconsistency Alleviated Fine-Grained (IAFG) cross-modal hashing retrieval, a novel framework that enables semantic component-level disentanglement and alignment without extensive pre-training. Our approach introduces two key innovations: Semantic Component Disentanglement (SCD) that achieves fine-grained separation of modality-common and modality-unique information using learnable query vectors and competitive feature routing, and Fine-grained Semantic Alignment (FSA) that realizes accurate cross-modal alignment at the component level while preserving semantic details through component-level cross-attention and cross-modal triplet alignment. Extensive experiments on benchmark datasets demonstrate that our method achieves state-of-the-art performance with significant improvements in retrieval accuracy across different modalities.
Data has become a key asset in today’s digital economies. To handle vast amounts of diverse data with distinct ownership from different contexts and organizational settings, data platforms—along with the concept of data ecosystems—have recently become a research field of high scholarly interest. This article reviews existing literature on data platforms and introduces a reference architecture designed to enable the deployment of data platforms for secure and trustworthy data storage, analysis, and sharing in multi-organizational settings. The proposed architecture is adaptable to various scenarios and requirements. Through a structured analysis of existing literature, the study identifies five key architectural solutions for multi-organizational data platforms. Through an in-depth examination of selected studies, the findings from the literature analysis on the core technological building blocks are substantiated across the different layers of these platforms. The research also incorporates two use cases from our own research projects where we utilize multi-organizational data platforms. These use cases provide practical insights into architectural building blocks. Based on the findings from the literature review and the two use cases, we derive a comprehensive reference architecture tailored for multi-organizational collaborations.
Regional co-location pattern mining is used to discover co-location patterns that are not prevalent globally but are prevalent in local regions. Although several regional co-location pattern mining methods have been developed, they have the following limitations: (1) they cannot capture the dynamic relationship between features; (2) their regional partitioning may disrupt the neighboring relationships between instances and is inefficient. To solve the above issues, the concept of Dynamic Region Co-location Patterns (DRCPs) is first defined to capture dynamic relationships between features in local regions. An interest measure of DRCPs is also presented, which comprehensively considers the effect of increasing or decreasing in the same feature instances. Second, a regional partitioning method based on generalized maximal row instances (GMRIs) is proposed. Since a GMRI typically merges multiple maximal row instances, it achieves more reasonable regional partitioning results. Additionally, due to the reduction in the number of maximal row instances, the partitioning efficiency is significantly improved. Third, a novel DRCP mining algorithm is designed, which fully utilizes the advantages of the GMRI and improves the mining efficiency. Finally, extensive experiments are conducted on both real and synthetic datasets to verify the effectiveness and efficiency of the proposed method.
