BitTorrent is a typical peer-to-peer (P2P) file distribution application that has gained tremendous popularity in recent years. A considerable amount of research exists regarding BitTorrent’s choking algorithm, which has proved to be effective in preventing freeriders. However, the effect of the seeding strategy on the resistance to freeriders in BitTorrent has been largely overlooked. In addition to this, a category of selfish leechers (termed exploiters), who leave the overlay im-mediately after completion, has never been taken into account in previous research. In this paper two popular seeding strate-gies, the Original Seeding Strategy (OSS) and the Time-based Seeding Strategy (TSS), are chosen and we study, via mathe-matical models and simulation, their effects on freeriders and exploiters in BitTorrent networks. The mathematical model is verified and we discover that both freeriders and exploiters impact on system performance, despite the seeding strategy that is employed. However, a selfish-leecher’s threshold is identified; once this threshold is exceeded, we find that TSS outperforms OSS – that is, TSS reduces the negative impact of selfish lechers more effectively than OSS. Based on these results we discuss the choice of seeding strategy and speculate as to how more effective BitTorrent-based file distribution appli-cations can be built.

In Data Intensive Computing, properties of the data that are the input for an application decide running performance in most cases. Those properties include the size of the data, the relationships inside data, and so forth. There is a class of data intensive applications (BLAST, SETI@home, Folding@Home and so on so forth) whose performances solely depend on the amount of input data. Another important characteristic of those applications is that the input data can be split into units and these units are not related to each other during the runs of the applications. This characteristic helps this class of data intensive applications to be parallelised in the way where the input data is split into units and application runs on different computer nodes for certain portion of the units. SETI@home and Folding@Home have been successfully parallelised over peer-to-peer networks. However, they suffer from the problems of single point of failure and poor scalability. In order to solve these problems, we choose BLAST as our example data intensive applications and parallelise BLAST over a fully distributed peer-to-peer network. BLAST is a popular bioinformatics toolset which can be used to compare two DNA sequences. The major usage of BLAST is searching a query of sequences inside a database for their similarities so as to identify whether they are new. When comparing single pair of sequences, BLAST is efficient. However, due to growing size of the databases, executing BLAST jobs locally produces prohibitively poor performance. Thus, methods for parallelising BLAST are sought. Traditional BLAST parallelisation approaches are all based on clusters. Clusters employ a number of computing nodes and high bandwidth interlinks between nodes. Cluster-based BLAST exhibits higher performance; nevertheless, clusters suffer from limited resources and scalability problems. Clusters are expensive, prohibitively so when the growth of the sequence database are taken into account. It involves high cost and complication when increasing the number of nodes to adapt to the growth of BLAST databases. Hence a Peer-to-Peer-based BLAST service is required. This thesis demonstrates our parallelisation of BLAST over Peer-to-Peer networks (termed ppBLAST), which utilises the free storage and computing resources in the Peer-to-Peer networks to complete BLAST jobs in parallel. In order to achieve the goal, we build three layers in ppBLAST each of which is responsible for particular functions. The bottom layer is a DHT infrastructure with the support of range queries. It provides efficient range-based lookup service and storage for BLAST tasks. The middle layer is the BitTorrent-based database distribution. The upper layer is the core of ppBLAST which schedules and dispatches task to peers. For each layer, we conduct comprehensive research and the achievements are presented in this thesis. For the DHT layer, we design and implement our DAST-DHT. We analyse balancing, maximum number of children and the accuracy of the range query. We also compare the DAST with other range query methodology and state that if the number of children is adjusted to more two, the performance of DAST overcomes others. For the BitTorrent-like database distribution layer, we investigate the relationship between the seeding strategies and the selfish leechers (freeriders and exploiters). We conclude that OSS works better than TSS in a normal situation.

BLAST is a popular toolset for protein and nucleotide sequence comparisons. Recent estimates show that these databases of protein and nucleotide sequences are set to grow exponentially and as a result local BLAST searches on single PCs cannot satisfy recent data processing requirements. Traditional approaches for the parallelisation of BLAST are focused on utilising clusters. However, scaling clusters to adapt potential growing BLAST requests will significantly increase the cost and complexities of maintenance and administration. This paper introduces ppBLAST, a BLAST service utilising the free computing resources in the peer-to-peer overlay on the Inter-net. ppBLAST uses several peer-to-peer technologies such as DAST, DHT and BitTorrent-like distribution and layered de-sign. In this paper, the feasibility of ppBLAST is validated through a small-scale deployment. The performance of ppBLAST is evaluated; we show that the execution time of a BLAST job can be considerably shortened if the scale of the supporting peer-to-peer overlay is large enough.

This paper describes the design and implementation of DAST, a Distributed Arbitrary Segment Tree structure that gives support of range query for public Distributed Hash Table (DHT) services. DAST does not modify the underlying DHT infrastructure, instead it utilises the scalability and robustness of DHT while providing simplicity of implementation and deployment for applications. Compared with traditional segment trees, the arbitrary segment tree used by a DAST reduces the number of key-space segments that need to be maintained, which in turn results in fewer query operations and lower overheads. Moreover, considering that range queries often contain redundant entries that the clients do not need, we introduce the concept of Accuracy of Results (AoR) for range queries. We demonstrate that by adjusting AoR, the DHT operational overhead can be improved. DAST is implemented on a well-known public DHT service (OpenDHT) and validation through experimentation and supporting simulation is performed. The results demonstrate the effectiveness of DAST over exiting methods.

In this paper we define a Distributed Arbitrary Segment Tree (DAST), a distributed tree-like structure that layers the range query processing mechanism over public Distributed Hash Table (DHT) services. Compared with traditional segment trees, the arbitrary segment tree used by a DAST reduces the number of key-space segments that need to be maintained, which in turn results in fewer query operations and lower overheads. Moreover, considering that range queries often contain redundant entries that the clients do not need, we introduce the concept of accuracy of results (AoR) for range queries. We demonstrate that by adjusting AoR, the DHT operational overhead can be improved. DAST is implemented on a well-known public DHT service (OpenDHT) and validation through experimentation and supporting simulation is performed. The results demonstrate the effectiveness of DAST over exiting methods.

This paper addresses workload allocation techniques for clusters of computers. The workload in question is homogenous or heterogeneous. Homogeneous workload contains only QoS-demanding jobs (QDJ) or nonQoS jobs (NQJ) while heterogeneous workload is a mix of QDJs and NQJs. The processing platform used is a single cluster or multiple clusters of computers. Two workload allocation strategies (called ORT and OMR) are developed for homogeneous workloads by establishing and numerically solving optimisation equation sets. The ORT strategy achieves the optimised mean response time for homogeneous NQJ workload; while the OMR strategy achieves the optimised mean miss rate for homogeneous QDJ workload. Based on ORT and OMR, a heterogeneous workload allocation strategy is developed to dynamically partition the clusters into two parts. Each part is managed by ORT or OMR to exclusively process NQJs or QDJs. The judicial partitioning achieves an optimised comprehensive performance, which combines the mean response time and the mean miss rate. The effectiveness of these workload allocation techniques is demonstrated through queueing-theoretical analysis as well as through experimental studies. These techniques can be applied to e-business workload management to improve the distribution of different types of requests in clusters of servers.

This paper addresses the dynamic scheduling of parallel jobs with QoS demands (soft-deadlines) in multiclusters and grids. Three metrics (over-deadline, makespan and idle-time) are combined with variable weights to evaluate the scheduling performance. These three metrics are used to measure the extent of the jobs' QoS demand compliance, the resource throughput and the resource utilization. Two levels of performance optimisation are applied in the multicluster. At the multicluster level, a scheduler (which we call MUSCLE) allocates parallel jobs with high packing potential to the same cluster; it also takes the jobs' QoS requirements into account and employs a heuristic to allocate suitable workloads to each cluster to balance the overall system performance. At the single cluster level, an existing workload manager, called TITAN, utilizes a genetic algorithm to further improve the scheduling performance of the jobs previously allocated by MUSCLE. Extensive experimental studies are conducted to verify the effectiveness of the scheduling mechanism as well as the effect of the prediction accuracy on the scheduling performance. The results show that compared with traditional distributed workload allocation policies, the comprehensive scheduling performance of parallel jobs is significantly improved across the multicluster, and the presence of prediction errors does not dramatically weaken the performance advantage.

This paper addresses the dynamic scheduling of moldable jobs with QoS demands (soft-deadlines) in multiclusters. A moldable job can be run on a variable number of resources. Three metrics (over-deadline, makespan and idletime) are combined with weights to evaluate the scheduling performance. Two levels of performance optimisation are applied in the multicluster. At the multicluster level, a scheduler (which we call MUSCLE) allocates parallel jobs with high packing potential to the same cluster; MUSCLE also takes the jobs' QoS requirements into account and employs a heuristic to achieve performance balancing across the multicluster. At the single cluster level, an existing workload manager, called TITAN, utilizes a genetic algorithm to further improve the scheduling performance of the jobs allocated by MUSCLE. Extensive experimental studies are conducted to verify the effectiveness of the scheduling mechanism in MUSCLE. The results show that the comprehensive scheduling performance of parallel jobs is significantly improved across the multicluster.

The paper addresses the dynamic scheduling of parallel jobs with quality-of-service demands (soft-deadlines) in multiclusters and grids. Three performance metrics (over-deadline, makespan and idle-time) are combined with variable weights to evaluate the scheduling performance. These three metrics are used to measure the extent to which jobs comply with their QoS demands, the resource throughput and the resource utilisation. Therefore, clusters situated in different administrative organisations can utilise different weight combinations to represent their different performance requirements. Two levels of performance optimisation are applied in the multicluster. At the multicluster level, a scheduler (which we call MUSCLE) allocates parallel jobs with high packing potential to the same cluster; MUSCLE also takes the jobs' QoS requirements into account and employs a heuristic to allocate suitable workloads to each cluster to balance the overall system performance. At the local cluster level, an existing workload manager, called TITAN, utilises a genetic algorithm to further improve the scheduling performance of the jobs sent by MUSCLE. Extensive experimental studies are conducted to verify the effectiveness of the scheduling mechanism in MUSCLE. The results show that, compared with traditional distributed workload allocation policies, the comprehensive scheduling performance (in terms of over-deadline, makespan and idle-time) of parallel jobs is significantly improved across the multicluster.