Resource management systems like YARN or Mesos allow sharing cluster resources by running dataparallel processing jobs in temporarily reserved containers. Containers, in this context, are logical leases of resources as, for instance, a number of cores and main memory, allocated on a particular node. Typically, containers are used without resource isolation to achieve high degrees of overall resource utilization despite the often fluctuating resource usage of single analytic jobs. However, some combinations of jobs utilize the resources better and interfere less with each other when running on the same nodes than others. This paper presents an approach for improving the resource utilization and job throughput when scheduling recurring distributed data-parallel processing jobs in shared cluster environments. Using a reinforcement learning algorithm, the scheduler continuously learns which jobs are best executed simultaneously on the cluster. We evaluated a prototype implementation of our approach with Hadoop YARN, exemplary Flink jobs from different application domains, and a cluster of commodity nodes. Even though the measure we use to assess the goodness of schedules can still be improved, the results of our evaluation show that our approach increases resource utilization and job throughput. This is an extended work of Thamsen, Rabier, Schmidt, Renner, & Kao, c © 2017 IEEE, published in the Proceedings of the 6th 2017 IEEE International Congress on Big Data (BigData Congress 2017).
The unique behaviours of drivers have many emerging applications. These include the personalization of automated/selfdriving vehicles so the owners are more comfortable with them, and the identification of changing driving behaviours that may be associated with aging or disease. This paper explores measures of driving behaviours that might allow for the differentiation of drivers based on their individual driving characteristics. An emerging challenge within longitudinal studies of drivers is to distinguish between different drivers of a shared vehicle. It also has application in the insurance industry where insurance risk and associated owner premium depends on the diversity or lack thereof of drivers for a vehicle such as a vehicle driven/never driven by secondary drivers that have higher risk driving behaviours. In this paper, a big data set of driving data for 14 older drivers is analyzed a single year of data includes over 250,000 km and almost 5000 hours of driving for the 14 drivers. A set of 162 trip level calculated features are analyzed to determine their ability to be used to distinguish between two drivers of a vehicle. The results show that features based on road choice and driver chosen velocity provide the best performance individually and in feature pairs with 2 features providing error rates less than 5% for some driver pairs. The set of features that provided the best performance differed for each driver pair and was found to include features from measures of a driver’s road choice, velocity and velocity ratio in addition to the features measuring trip similarity to two phase acceleration and deceleration relationships for the driver. The best error rate obtained was 1.5% for a driver pair. On the other hand, the results suggest that a number of features and feature groups do not allow for older driver differentiation. For instance, overnight driving and high rates of acceleration are not sufficiently exhibited by these drivers to be useful.
Users nowadays are normally involved in multiple (usually more than two) online social networks simultaneously to enjoy more social network services. Some of the networks that users are involved in can share common structures either due to the analogous network construction purposes or because of the similar social network features. However, the social network datasets available in research are usually pre-anonymized and accounts of the shared users in different networks are mostly isolated without any known connections. In this paper, we want to identify such connections between the shared users' accounts in multiple social networks (i.e., the anchor links), which is formally defined as the M-NASA (Multiple Anonymized Social Networks Alignment) problem. M-NASA is very challenging to address due to (1) the lack of known anchor links to build models, (2) the studied networks are anonymized, where no users' personal profile or attribute information is available, and (3) the "transitivity law" and the "one-to-one property" based constraints on anchor links. To resolve these challenges, a novel two-phase network alignment framework UMA (Unsupervised Multi-network Alignment) is proposed in this paper. Extensive experiments conducted on multiple real-world partially aligned social networks demonstrate that UMA can perform very well in solving the M-NASA problem.
Nowadays, with the fast development of social media, the instant financial news can be quickly spread over the Internet and consequently results in the strong vibration of s tock market within a short period of time. To capture the effect of financial news, various classification algorithms are proposed in financial data mining domain. Unfortunately, it has been shown that the positive news might not necessarily drive up the stock price. It is believed that there must exist some underlying events, called anchor events in this paper, which drive the stock market up or down. Therefore, this paper proposed the classification should be performed not only based on the textual features of financial news, but also the semantic distance between the testing set of news and the set of news of anchor events. Accordingly, such semantic distance is measured as deviated expectation calculated based on the distance between two set of extracted topics. The proposed method involves two steps, i.e., calculating whether the testing news is consistent with anchor events or not, and calculating how far the testing news is from the anchor ev ents. We then evaluate our method as well as the state-of-the-art classification algorithms on some real data sets. The promising experimental results have demonstrated that the proposed method is superior to the state-of-the-art classification algorithms in terms of classification accuracy.
Data analytics involves the process of data collection, data analysis, and report generation. Data mining workflow tools usually orchestrate this process. The data analysis step in this process further consists a series of machine learning algorithms. There exists a variety of data mining tools and machine learning algorithms. Each tool or algorithm has its own set of features that become factors to affect both functional and nonfunctional attributes of the system of data analytics. Given domain-specific requirements of data analytics, understanding the effects of these factors and their combinations provide a guideline of selecting workflow tools and machine learning algorithms. In this paper, we develop an empirical evaluation method based on the principle of Design of Experiment. We apply this method to evaluate data mining tools and machine learning algorithms towards building big data analytics for telecommunication monitoring data. Two case studies are conducted to provide insights of relations between the requirements of data analytics and the choice of a tool or algorithm in the context of data analysis workflows. The demonstration also shows that our evaluation method can facilitate the replication of this evaluation study, and can conveniently be expanded for evaluating other tools and algorithms.
Since the emergence of mobile networks, the number of mobile subscriptions has continued to increase year after year. To efficiently assign mobile network resources such as spectrum (which is expensive), the network operator needs to critically process and analyze information and develop statistics about each base station and the traffic that passes through it. This paper presents an application of data analytics by focusing on processing and analyzing two datasets from a commercial trial mobile network. A detailed description that uses Apache Hadoop and the Mahout Machine learning library to process and analyze the datasets is presented. The analysis provides insights about the resource usage of network devices. This information is of great importance to network operators for efficient and effective management of resources and for supporting high-quality of user experience. Furthermore, an investigation has been conducted that evaluates the impact of executing the Mahout clustering algorithms with various system and workload parameters on a Hadoop cluster. The results demonstrate the value of performance data analysis. Specifically, the execution time can be significantly reduced using data pre-processing, some machine learning techniques, and Hadoop. The investigation provides useful information for the network operators for future real-time data analytics.
Big Data contains massive information, which are generating from heterogeneous, autonomous sources with distributed and anonymous platforms. Since, it raises extreme challenge to organizations to store and process these data. Conventional pathway of store and process is happening as collection of manual steps and it is consuming various resources. An automated real-time and online analytical process is the most cognitive solution. Therefore it needs state of the art approach to overcome barriers and concerns currently facing by the Big Data industry. In this paper we proposed a novel architecture to automate data analytics process using Nested Automatic Service Composition (NASC) and CRoss Industry Standard Platform for Data Mining (CRISP-DM) as main based technologies of the solution. NASC is well defined scalable technology to automate multi-disciplined problems domains. Since CRISP-DM also a well-known data science process which can be used as innovative accumulator of multi-dimensional data sets. CRISP-DM will be mapped with Big Data analytical process and NASC will automate the CRISP-DM process in an intelligent and innovative way.
Nowadays, streaming data overflows from a diversity of sources and technologies, making traditional data analytics technologies unsuitable to handle the latency of data processing relative to the growing demand for high processing speed and algorithmically scalability. Real-time streaming data analytics is needed to allow applications to analyze streaming data effectively and efficiently. The open source software Apache Storm, which is a distributed computation system for processing streaming data in real-time, has been widely used for building applications to analyze streaming data in real-time because it is fast, scalable, fault tolerant and reliable. This paper proposes a cloud-based architecture based on Apache Storm for supporting an entire streaming data analytics workflow, which consists of data ingestion, data processing, data visualization and data storing. The application on the proposed system is available at http://206.167.181.16:8000/map4.html.
Big Data is the paradigm that represents the ability to analyze and cross-reference large amounts of data generated by computational systems and turn them into useful knowledge. This potential is one solution organizations can use to answer the challenge of getting closer to their users. Organization managers face the challenge of understanding the Big Data concept and the business strategies inherent to its use. The high number of challenges that need to be addressed creates a high number of proposed technical solutions that most times only overlap existing ones. Frequently managers face these issues as their organizations race against the competitors for a market share, without having resources to embrace not only Big Data but also other options that can give competitive advantage. Therefore, organization owners and managers must be educated on deployed platforms that can make them understand the benefits that can be achieved on short term. In this paper we aim to provide an overview of using Big Data with Open Source tools. We explain the Big Data concept, the potential value and the organizational strategies that must be studied in order to determine which benefits organizations can win from it. We analyze the strengths and drawbacks of five open source frameworks for distributed data programming – Hadoop, Spark, Storm, Flink and H2O – and seven open source platforms for Big Data Analytics – Mahout, MOA, R Project, Vowpal Wabbit, Pegasus, GraphLab Create and MLLib. There is no single platform that truly embodies a one size fits all solution, so this paper aims to help decision makers by providing as much information as possible and quantifying some tradeoffs.
Predictive analytics in the big data era is taking on an ever increasingly important role. Issues related to choice on modeling technique, estimation procedure/algorithm and efficient execution can present significant challenges. For example, selection of appropriate/optimal models for big data analytics often requires careful investigation and considerable expertise which might not always be readily available. In this paper, we propose to use semantic technology to assist data analysts/data scientists in selecting appropriate modeling techniques and building specific models as well as the rationale for the techniques and models selected. To formally describe the modeling techniques, models and results, we developed the Analytics Ontology that supports inferencing for semi-automated model selection. The ScalaTion framework, which currently supports over thirty modeling techniques for predictive big data analytics is used as a test bed for evaluating the use of semantic technology.
In the era of big data, interest in analysis and extraction of information from large data graphs is increasing rapidly. This paper examines the field of graph analytics from somewhat of a query processing point of view. Whether it be determination of shortest paths or finding patterns in a data graph matching a query graph, the issue is to find interesting characteristics or information content from graphs. Many of the associated problems can be abstracted to problems on paths or problems on patterns. Unfortunately, seemingly simple problems, such as finding patterns in a data graph matching a query graph are surprisingly difficult. In addition, the iterative nature of algorithms in this field makes the simple MapReduce style of parallel and distributed processing less effective. Still, the need to provide answers even for very large graphs is driving the research. Progress, trends and directions for future research are presented.
The popularity of enterprise cloud storage is rapidly growing. A number of Internet service vendors and providers, such as Google, Baidu and Microsoft, entered this emerging market and released a variety of cloud storage services. These services allow people to access work documents and files all over the world at anytime. Interestingly, with the prevalence of mobile Internet, rich media becomes regular and popular. More and more people use cloud storage for keeping their personal photos, music and movies. Nevertheless, the size of the media files is often beyond the upper limit that normal form-based file upload service allows hence dedicated large-file upload services are required to be developed and deployed. Although many cloud vendors offer versatile cloud storage services, very little is known about the detailed development and deployment of the large-file upload services. This paper proposes a complete solution of large-file upload service, with the contributions in manifold: Firstly, we do not limit the maximum size of a large file that can be uploaded. This is extremely practical for storing huge database resource files generated from ERP tools. Secondly, we developed large-file upload service APIs that have very strict verification of correctness, to reduce the risk of data inconsistency, which has better safety. Thirdly, we extend the service developed recently for team collaboration with the capability of handling large files. Fourthly, this paper is arguably the first one that formalizes the testing and deployment procedures of large-file upload services with the help of Docker. In general, most largefile upload services are exposed to the public, facing security and performance issues, which brings much concern. With the proposed Docker-based deployment strategy, we can replicate the large-file upload service agilely and locally, to satisfy massive private or local deployment of KDrive. Finally, we evaluate and analyze the proposed strategies and technologies in accordance to the experimental results. This paper is an extension version of the SCC 2015 conference paper: On Developing and Deploying Large-File Upload Services of Personal Cloud Storage.
