Data management applications are growing and require more attention, especially in the "big data" era. Thus, supporting such applications with novel and efficient algorithms that achieve higher performance is critical. Array database management systems are one way to support these applications by dealing with data represented in n-dimensional data structures. For instance, software like SciDB and RasDaMan can be powerful tools to achieve the required performance on large-scale problems with multidimensional data. Like their relational counterparts, these management systems support specific array query languages as the user interface. As a popular programming model, MapReduce allows large-scale data analysis, facilitates query processing, and is used as a DB engine. Nevertheless, one major obstacle is the low productivity of developing MapReduce applications. Unlike high-level declarative languages such as SQL, MapReduce jobs are written in a low-level descriptive language, often requiring massive programming efforts and complicated debugging processes. This work presents a system that supports translating array queries expressed in the Array Query Language (AQL) in SciDB into MapReduce jobs. We focus on translating some unique structural aggregations, including circular, grid, hierarchical, and sliding aggregations. Unlike traditional aggregations in relational DBs, these structural aggregations are designed explicitly for array manipulation. Thus, our work can be considered an array-view counterpart of existing SQL to MapReduce translators like HiveQL and YSmart. Our translator supports structural aggregations over arrays to meet various array manipulations. The translator can also help user-defined aggregation functions with minimal user effort. We show that our translator can generate optimized MapReduce code, which performs better than the short handwritten code by up to 10.84x.

Accurate prediction of electricity generation from diverse renewable energy sources (RES) plays a pivotal role in optimizing power schedules within RES, contributing to the collective effort to combat climate change. While prior research often focused on individual energy sources in isolation, neglecting intricate interactions among multiple sources, this limitation frequently leads to inaccurate estimations of total power generation. In this study, we introduce a hybrid architecture designed to address these challenges, incorporating advanced artificial intelligence (AI) techniques. The hybrid model seamlessly integrates a gated recurrent unit (GRU) and a ResNext model, and it is tuned with the modified jaya algorithm (MJA) to capture localized correlations among different energy sources. Leveraging its nonlinear time-series properties, the model integrates meteorological conditions and specific energy source data. Additionally, principal component analysis (PCA) is employed to extract linear time-series data characteristics for each energy source. Application of the proposed AI-infused approach to a renewable energy system demonstrates its effectiveness and feasibility in the context of climate change mitigation. Results reveal the superior accuracy of the hybrid framework compared to more complex models such as decision trees and ResNet. Specifically, our proposed method achieved remarkable performance, boasting the lowest error rates with a normalized RMSE of 6.51 and a normalized MAPE of 4.34 for solar photovoltaic (PV), highlighting its exceptional precision in terms of mean absolute errors. A detailed sensitivity analysis is carried out to evaluate the influence of every element in the hybrid framework, emphasizing the importance of energy correlation patterns. Comparative assessments underscore the increased accuracy and stability of the suggested AI-infused framework when compared to other methods.

Semi-supervised clustering with pairwise constraints has been a hot topic among researchers and experts. However, the problem becomes quite difficult to manage using random constraints for clustering data when the clusters have different shapes, densities, and sizes. This research proposes an active semi-supervised density-based clustering algorithm, termed "ASS-DBSCAN," designed specifically for clustering multi-density data. By integrating active learning and semi- supervised techniques, ASS-DBSCAN enhances traditional clustering methods, allowing it to handle complex data distributions with varying densities more effectively. This research provides two major contributions. The first contribution of this research is to analyze how to link constraints (including that must be linked and ones that should not be linked) that will be utilized by the clustering algorithm. The second contribution made by this research is the ability to add multiple density levels to the dataset. We perform experiments over real datasets. The ASS-DBSCAN algorithm was evaluated against existing state-of-the-art system for various evaluation metrics in which it performed remarkably well.

Pairwise constraints improve clustering performance in constraint-based clustering issues, especially since they are applicable. However, randomly choosing these constraints may be adverse and minimize accuracy. To address the problem of random choosing pairwise constraints, an active learning method is used to identify the most informative constraints, which are then selected by the active learning technique. In this research, we replaced random selection with an active learning strategy. We provide a semi-supervised selective affinity propagation clustering approach with active constraints, which combines the affinity propagation (AP) clustering algorithm with prior information to improve semi-supervised clustering performance. Based on the neighborhood concept, we select the most informative constraints where neighborhoods include labelled examples of various clusters. The experimental results on eight real datasets demonstrate that the proposed method in this paper outperforms other baseline methods and that it can improve clustering performance significantly.

Semi.-supervised clustering algorithms aim to enhance the performance of clustering using the pairwise constraints. However, selecting these constraints randomly or improperly can minimize the performance of clustering in certain situations and with different applications. In this paper, we select the most informative constraints to improve semi-supervised clustering algorithms. We present an active selection of constraints, including active must.-link (AML) and active cannot.-link (ACL) constraints. Based on Radial-Bases Function, we compute lower-bound and upper-bound between data points to select the constraints that improve the performance. We test the proposed algorithm with the base-line methods and show that our proposed active pairwise constraints outperform other algorithms.

The clustering method divides a dataset into groups with similar data using similarity metrics. However, discovering clusters in different densities, shapes and distinct sizes is still a challenging task. In this regard, experts and researchers opt to use the DBSCAN algorithm as it uses density-based clustering techniques that define clusters of different sizes and shapes. However, it is misapplied to clusters of different densities due to its global attributes that generate a single density. Furthermore, most existing algorithms are unsupervised methods, where available prior knowledge is useless. To address these problems, this research suggests the use of a clustering algorithm that is semi-supervised. This allows the algorithm to use existing knowledge to generate pairwise constraints for clustering multi-density data. The proposed algorithm consists of two stages: first, it divides the dataset into different sets based on their density level and then applies the semi-supervised DBSCAN algorithm to each partition. Evaluation of the results shows the algorithm performing effectively and efficiently in comparison to unsupervised clustering algorithms.

Energy saving is a major requirement in the design of routing algorithms to maximize the throughput and lifetime of wireless sensor networks. To address this requirement, this work proposes a routing algorithm based on a novel Reposition Particle Swarm Optimization (RPSO) algorithm and a new fitness function. The role of the RPSO algorithm is to salvage particles that may have fallen into local minima, if any. The algorithm is evaluated using three types of benchmark functions, namely unimodal, multimodal, and rotated multimodal. The evaluation results demonstrate that RPSO outperforms native Particle Swarm Optimization and Particle Swarm Optimization with Levy Flight in terms of convergence speed and global optimum identification. The proposed RPSO is then employed to develop an efficient routing algorithm. The routing algorithm is tested in terms network lifetime, number of dead sensor nodes, energy consumption, and number of packets delivered to base station, against four competitive Meta-Heuristic algorithms. The test results clearly indicate that the routing algorithm outperforms all the competitive algorithms for all the performance metrics. In particular, it achieves 14% to 29% higher network lifetime, 52% to 58% less dead sensor nodes, 61% to 70% less energy consumption, and 13% to 36% more throughput.

Most existing applications have a large number of evolving data streams. Clustering data streams is still a critical problem for these applications as the data are evolving and changes over time. Most existing algorithms are unsupervised learning in which background information is useless. This paper proposes an active clustering algorithm for data stream based on the affinity propagation method, referred to as AAPStream. The affinity propagation aims to identify exemplars and create clusters based on these exemplars. Thus, the objective is to get the most informative exemplars to create the streaming model and predict the new arrival data. We conduct a set of experiments on real-world datasets to compare our algorithm with a state-of-the-art algorithm, and the experimental results show the effectiveness of the proposed algorithm.

Outlier detection is one of the important tasks in data mining. Detecting outliers over streaming data has become an important task in many applications, such as network analysis, fraud detections, and environment monitoring. One of the well-known outlier detection algorithms called Local Outlier Factor (LOF). However, the original LOF has many drawbacks that can’t be used with data streams: 1- it needs a lot of processing power (CPU) and large memory to detect the outliers. 2- it deals with static data which mean that in any change in data the LOF recalculates the outliers from the beginning on the whole data. These drawbacks make big challenges for existing outlier detection algorithms in terms of their accuracies when they are implemented in the streaming environment. In this paper, we propose a new algorithm called GSILOF that focuses on detecting outliers from data streams using genetics. GSILOF solve the problem of large memory needed as it has fixed memory bound. GSILOF has two phases. First, the summarization phase that tries to summarize the past data arrived. Second, the detection phase detects the outliers from the new arriving data. The summarization phase uses a genetic algorithm to try to find the subset of points that can represent the whole original set. our experiments have been done over real datasets. Our experiments confirming the effectiveness of the proposed approach and the high quality of approximate solutions in a set of real-world streaming data.

In this digital era of technology and software development tools, low-cost digital cameras and powerful video editing software (such as Adobe Premiere, Microsoft Movie Maker, and Magix Vegas) have become available for any common user. Through these softwares, editing the contents of digital videos became very easy. Frame duplication is a common video forgery attack which can be done by copying and pasting a sequence of frames within the same video in order to hide or replicate some events from the video. Many algorithms have been proposed in the literature to detect such forgeries from the video sequences through analyzing the spatial and temporal correlations. However, most of them are suffering from low efficiency and accuracy rates and high computational complexity. In this paper, we are proposing an efficient and robust frame duplication detection algorithm to detect duplicated frames from the video sequence based on the improved Levenshtein distance. Extensive experiments were performed on some selected video sequences captured by stationary and moving cameras. In the experimental results, the proposed algorithm showed efficacy compared with the state-of-the-art techniques.

we have a common problem in wireless sensor networks which is the missing data problem due to the nature of the wireless communication and the limited resources of the sensor nodes. This problem can't be ignored because it has a negative effect on the applications that use the sensor data. Estimating these missing data is important for the applications that concern with the sensor data. However, the traditional estimation techniques failed to be applied with the sensor data and the existing techniques have high computation complexity, high computation time, or low accuracy. So we introduce the simplified Spatial and Temporal Correlation (STC) estimation algorithm which uses the most related surrounding previous data to increase the accuracy of the estimation and reduce incremental error. The proposed algorithm utilizes the time correlation by using the closet data before the time of missing and utilizes the space correlation by using the data of the nearest sensor depending on the missing pattern. The experimental results show that our algorithm can reduce the error in the estimating process compared with the other algorithms in most of the missing patterns.

This study aims to solve the problem of detecting anomalies in big data. A border-based Gird Partition (BGP) algorithm was proposed. The BGP algorithm focuses on calculating the Local Outlier Factor (LOF) for big data in a distributed environment. It splits the data into intersected subsets, then allocates these subsets to the slave nodes in a distributed environment. Some parts of these subsets are replicated between slave nodes. The slave nodes calculate the LOF for each subset that it owns. The splitting of the data between the slave nodes is done in grid-based without considering the size of the data that will be assigned to every slave node. The BGP algorithm results in un-balanced distribution of the subsets between slave nodes. To overcome this problem a modification on the BGP algorithm is proposed to take in consideration the size of the data that will be assigned to every slave node. The modified algorithm called Balanced boarder-based Gird Partition algorithm (BBGP). BBGP splits the data between the slave node equally. So that all the slave nodes will do balanced processing for calculating the LOF for the data. In the end, we evaluate the performance of the two algorithms through a series of simulation experiments over real data sets.

Semi-supervised clustering algorithms use a small amount of supervised data in the form of pairwise constraints to improve the clustering performance. However, most current algorithms are passive in the sense that the pairwise constraints are provided beforehand and selected randomly. This may lead to the use of constraints that are redundant, unnecessary, or even harmful to the clustering results. In this paper, we address the problem of constraint selection to improve the performance of semi-supervised clustering algorithms. Based on the concepts of Maximum Mean Discrepancy, we select a batch of most informative instances that minimize the difference in distribution between the labeled and unlabeled data. Then, querying these instances with the existing neighborhoods to determine which neighborhood they belong. The experimental results with state-of-the-art methods on different real-world dataset demonstrate the effectiveness and efficiency of the proposed method.

Anomaly detection or outlier detection has become a major research problem in the era of big data. It is used in many applications, remove noise from signals and in credit card fraud detection. One type of outlier detection is Density-based outlier detection. Its major uniqueness is in detecting outlier points in different densities. One of the algorithms that are based on density based outlier detection is Local Outlier Factor (LOF). LOF gives every point a score that identifies its outlierness compared to other points. In this paper, we propose a new algorithm called sub-Grid partition (SGP) algorithm. SGP algorithm helps in calculating the LOF for Big Data in a distributed environment. SGP algorithm splits the tuples into small grids each grid is splitted into sub-grids. Sub-grids in the border are duplicated in every processing node for calculating the LOF for every tuple in these grids. Duplication of sub-grids lead to increase in the number of tuples that will be processed but in the other hand reduces the network overhead required for communication between processing nodes and reducing processing node idle time waiting for the requested tuple. In the end, we evaluate the performance of the SGP algorithm through a series of simulation experiments over real data sets.

The purpose of data clustering is to identify useful patterns in the underlying dataset. However, finding clusters in data is a challenging problem especially when the clusters are being of widely varied shapes, sizes, and densities. Density-based clustering methods are the most important due to their high ability to detect arbitrary shaped clusters. Moreover these methods often show good noise-handling capabilities. Existing methods are based on DBSCAN which depends on two specified parameters (Eps and Minpts) that define a single density. Moreover, most of these methods are unsupervised, which cannot improve the clustering quality by utilizing a small number of prior knowledge. In this paper we show how background knowledge can be used to bias a density-based clustering algorithm for multi-density data. First we divide the dataset into different density levels and detect suitable density parameters for each density level. Then we describe how pairwise constraints can be used to help the algorithm expanding the clustering process based on the computed density parameters. Experimental results on both synthetic and real datasets confirm that the proposed algorithm gives better results than other semi-supervised and unsupervised clustering algorithms.

Finding clusters is a challenging problem especially when the clusters are being of widely varied shapes, sizes, and densities. Density-based clustering methods are the most important due to their high ability to detect arbitrary shaped clusters. However, they are depending on two specified parameters (Eps and Minpts) that define a single density. Moreover, most of these methods are unsupervised, which cannot improve the clustering quality by utilizing a small number of prior knowledge. In this paper we show how background knowledge can be used to bias a density-based clustering method for multi-density data. Experimental results confirm that the proposed method gives better results than other semi-supervised and unsupervised clustering algorithms.

Clustering data stream is an active research area that has re cently emerged to discover knowledge from large amounts of continuously generated data. Several clustering algorith ms have been proposed for static data. Nevertheless, data stream clustering imposes several challenges to be addressed, such as dealing with dynamic data that arrive in an online fashion, capable of performing fast and incremental processing of data objects, suitably a ddressing time and memory limitations, and how to handle the evolving patterns that are important characteristics of streaming d ata with dynamic distributions. In this paper, we propose an algorithm that extends Affinity Propagation (AP) to handle evolving data st eam with dynamic distribution. Affinity Propagation was pro posed as a clustering algorithm extracted a set of exemplars that best represent the dataset using a message passing method. We present a semi- supervised clustering technique (SSAP) that incorporates labeled exemplars into the AP algorithm to deal with changes in the data distribution, which requires the stream model to be updated as soon as possible. Experimental results with state-of-th e-art data stream clustering methods demonstrate the effectiveness and effic iency of the proposed method.

Semi-supervised clustering uses a small amount of supervised data in the form of pairwise constraints to improve the clustering performance. However, most current methods are passive in the sense that the pairwise constraints are provided beforehand and selected randomly. This may lead to the use of constraints that are redundant, unnecessary, or even harmful to the clustering results. In this paper, we address the problem of constraint selection to improve the performance of constraint-based clustering algorithms. Based on the concepts of Maximum Mean Discrepancy, we select the set of most informative instances that minimizes the difference in distribution between the labeled and unlabeled data. Then, we query these instances with the existing neighborhoods to determine which neighborhood they belong. The experimental results with state-of-the-art methods on different real world dataset demonstrate the effectiveness and efficiency of the proposed method.