High-performance scientific applications modeling natural phenomena are pushing the boundaries of modern parallel computing systems. These applications are computation and communication intensive. Continual scaling of these applications to even larger systems is key to gaining critical insights into natural phenomena. At the same time, performance optimization of large real-world scientific applications is becoming increasingly challenging due to increasing complexity of various components, such as computer architecture, message-passing library, network architecture and finally application algorithms. Effective optimization of applications increasingly requires careful understanding of cross-cutting issues. AWM-Olsen is a heavily used NSF Teragrid seismic modeling application which is communication intensive. In a previous paper we presented our design modifications to AWM-Olsen using MPI-2 RMA semantics. In this paper, we present a detailed performance analysis and further optimization techniques for the updated MPI-2 version of AWM-Olsen. We propose an improved design to increase overlap of computation and communication using MVAPICH2, a popular MPI-2 implementation on InfiniBand. Additionally, we propose using application logic aware loop fusion techniques to balance computation load within processes in a parallel job. Using the combination of our proposed optimizations, time spent in communication can be reduced by 72% over the previous MPI-2 version and by 89% over the unmodified MPI-1 version on 8K processes. This results in an overall speedup in the execution time by 8%, and 15% respectively. The experiments were carried out on the Ranger cluster at Texas Advanced Computing Center.

Interest points on the building facade are the basic element for 3-D building modeling and texturing. Grouping these points to the same or separate buildings is a fundamental process for establishing building models and detecting building boundaries. The grouping process is generally achieved by analyzing the geometric relation and the distances between the points in the object space, which requires precise interior and exterior orientation camera parameters. In this paper, we propose a method for grouping points on buildings in the image space and focus on the close-range stereo image problems. The highlight of this method is the capability to work with un-calibrated cameras and even with images gathered from the internet, such as Google Street View TM . Every plane in an image pair, such as building facade, has a unique homography matrix, which we exploit as the basis of grouping points lying on this plane. Three major steps in this method: Fundamental matrix estimation, grouping points to planes, and assigning planes to building facades. Projective geometry between the two images can be represented by the fundamental matrix, which can be solved using point correspondences. We apply 2D Delaunay triangulation to define a set of patches in image space. Each patch that reside on the building facade has a corresponding homography matrix can be estimated from the fundamental matrix. We group the patches based on the similarity of the estimated homography matrices for the patches and forming points into planes. Buildings are usually connected and obscured in the photograph taken in a building rich circumstance. Grouping planes into building facade require two additional steps: 1) Separating buildings that are connected in the image but not in the real world, and 2) connecting two or more planes that belong to the same building facade. These resulting building facade can then be transform to 3-D using the geometric constraint or GCPs.

Network congestion is an important factor affecting the performance of large scale jobs in supercomputing clusters, especially with the wide deployment of multi-core processors. The blocking nature of current day collectives makes such congestion a critical factor in their performance. On the other hand, modern interconnects like InfiniBand provide us with many novel features such as Virtual Lanes aimed at delivering better performance to end applications. Theoretical research in the field of network congestion indicate Head of Line (HoL) blocking as a common causes for congestion and the use of multiple virtual lanes as one of the ways to alleviate it. In this context, we make use of the multiple virtual lanes provided by the InfiniBand standard as a means to alleviate network congestion and thereby improve the performance of various high performance computing applications on modern multi-core clusters. We integrate our scheme into the MVAPICH2 MPI library. To the best of our knowledge, this is the first such implementation that takes advantage of the use of multiple virtual lanes at the MPI level. We perform various experiments at native InfiniBand, microbenchmark as well as at the application levels. The results of our experimental evaluation show that the use of multiple virtual lanes can improve the predictability of message arrival by up to 10 times in the presence of network congestion. Our microbenchmark level evaluation with multiple communication streams show that the use of multiple virtual lanes can improve the bandwidth / latency / message rate of medium sized messages by up to 13%. Through the use of multiple virtual lanes, we are also able to improve the performance of the Alltoall collective operation for medium message sizes by up to 20%. Performance improvement of up to 12% is also observed for Alltoall collective operation through segregation of traffic into multiple virtual lanes when multiple jobs compete for the same network resource. We also see that our scheme can improve the performance of collective operations used inside the CPMD application by 11% and the overall performance of the CPMD application itself by up to 6%.

AWM-Olsen is a widely used ground motion simulation code based on a parallel finite difference solution of the 3-D velocity-stress wave equation. This application runs on tens of thousands of cores and consumes several million CPU hours on the TeraGrid Clusters every year. A significant portion of its run-time (37% in a 4,096 process run), is spent in MPI communication routines. Hence, it demands an optimized communication design coupled with a low-latency, high-bandwidth network and an efficient communication subsystem for good performance. In this paper, we analyze the performance bottlenecks of the application with regard to the time spent in MPI communication calls. We find that much of this time can be overlapped with computation using MPI non-blocking calls. We use both two-sided and MPI-2 one-sided communication semantics to re-design the communication in AWM-Olsen. We find that with our new design, using MPI-2 one-sided communication semantics, the entire application can be sped up by 12% at 4K processes and by 10% at 8K processes on a state-of-the-art InfiniBand cluster, Ranger at the Texas Advanced Computing Center (TACC).

GridFTP, designed using the Globus XIO framework, is one of the most popular methods in use to perform data transfers in the grid environment. But the performance of GridFTP in WAN is limited by the relatively low communication bandwidth offered by the existing network protocols. On the other hand, modern interconnects such as InfiniBand, with many advanced communication features like zero-copy protocol and RDMA operations, can greatly improve communication efficiency. In this paper, we take on the challenge of combining the ease of use of the Globus XIO framework and the high performance achieved through InfiniBand communication, thereby natively sup-porting GridFTP over InfiniBand based networks. The Advanced Data Transfer Service (ADTS), designed in our previous work, provides the low level InfiniBand support to the Globus XIO layer. We introduce the concepts of I/Ostaging in the Globus XIO ADTS driver to achieve efficient disk based data transfers. We evaluate our designs in both LAN and WAN environments using micro benchmarks as well as communication traces from several real world applications. We also provide insights into the communication performance with some in-depth analysis. Our experimental evaluation shows a performance improvement of up to100% for ADTS based data transfers as opposed to TCP or UDP based ones in LAN and high delay WAN scenarios.

The increasing popularity of multi-core processors has made MPI intra-node communication, including the intra-node RMA (Remote Memory Access) communication, a critical component in high performance computing. MPI-2 RMA model includes one-sided data transfer and synchronization operations. Existing designs in popularly used MPI stacks do not provide truly one-sided intra-node RMA communication. They are built on top of two-sided send-receive operations, therefore suffering from overheads of two-sided communication and dependency on the remote side. In this paper, we enhance existing shared memory mechanisms to design truly one-sided synchronization. In addition, we design truly one-sided intra-node data transfer using two kernel based direct copy alternatives: basic kernel-assisted approach and I/OAT-assisted approach. Our new design eliminates the overhead of using two-sided operations and eliminates the involvement from the remote side. We also propose a series of benchmarks to evaluate various performance aspects over multi-core architectures (Intel Clovertown, Intel Nehalem and AMD Barcelona). The results show that the new design obtains up to 39% lower latency for small and medium messages and demonstrates 29% improvement in large message bandwidth. Moreover, it provides superior performance in terms of better scalability, reduced cache misses, higher resilience to process skew and increased computation and communication overlap. Finally, up to 10% performance benefits is demonstrated for a real scientific application AWM-Olsen.

High End Computing (HEC) systems are being deployed with eight to sixteen compute cores, with 64 to 128 cores/node being envisioned for exascale systems. \mbox{MVAPICH2} is a popular implementation of MPI-2 specifically designed and optimized for InfiniBand, iWARP and RDMA over Converged Ethernet (RoCE). MVAPICH2 is based on MPICH2 from ANL. Recently MPICH2 has been redesigned with an effort to optimize intra-node communication for future many-core systems. The new communication layer in MPICH2 is called Nemesis, which is very well optimized for shared memory message passing, with a modular design for various high-performance interconnects. In this paper we explore the challenges involved in designing the next-generation MVAPICH2 stack, leveraging the Nemesis communication layer. We observe that Nemesis does not provide abstractions for one-sided communication. We propose an extended Nemesis interface for optimized one-sided communication and provide design details. Our experimental evaluation shows that our proposed one-sided interface extensions are able to provide significantly better performance than the basic Nemesis interface. For example, inter-node MPI_Put bandwidth increased from 1,800 MB/s to 3,000 MB/s and latency for small messages went down by 13%. Additionally, with our proposed designs, we are able to demonstrate performance gains with small messages, when compared to the existing MVAPICH2 CH3 implementation. The designs proposed in this paper is a superset of currently available options to MVAPICH2 users and provides the best combination of performance and modularity.

With the emergence of multi-core-based processors, it is becoming significantly important to optimize both intra-node and inter-node communication in an MPI stack. MPICH2 group has recently introduced a new Nemesis-based MPI stack which provides highly optimized design for intra-node communication. It also provides modular design for different inter-node networks. Currently, the MPICH2/Nemesis stack has support for TCP/IP and Myrinet only. The TCP/IP interface allows this stack to run on the emerging InfiniBand network with IPoIB support. However, this approach does not deliver good performance and can not exploit the novel mechanisms and features provided by InfiniBand. In this paper, we take on the challenge of designing a portable InfiniBand network module (IB-netmod) for Nemesis. The IB-netmod is designed over the Verbs-level interface of InfiniBand and can take advantage of all features and mechanisms of InfiniBand. A complete design of the IB-netmod with the associated challenges are presented. A comprehensive performance evaluation (micro-benchmarks, collectives and applications) of the new Nemesis-IB design is carried out against the Nemesis TCP/IP (with IPoIB support on InfiniBand) and the native IB support of the MVAPICH2 stack. The new IB-netmod is able to deliver comparable performance to that of the native IB support of MVAPICH2. Compared to the MPICH2/IPoIB support for InfiniBand, the new design is able to deliver significant performance benefits. For NAMD application with 256 cores, the new IB-netmod is able to deliver 4% improvement compared to the latest MVAPICH2 release. To the best of our knowledge, this is the first IB-netmod design for the MPICH2/Nemesis framework. The next release of MVAPICH2 will be having this new IB-netmod support.

The rapid growth of InfiniBand, 10 Gigabit Ethernet/iWARP and IB WAN extensions is increasingly gaining momentum for designing high end computing clusters and data-centers. For typical applications such as data staging, content replication and remote site backup, FTP has been the most popular method to transfer data within and across these clusters. Although the existing sockets based FTP approaches can be transparently used in these systems through the protocols like IPoIB or SDP, their performance and scalability are limited due to the additional interaction overhead and unoptimized protocol processing. This leads to a challenge how to design more efficient FTP mechanisms by leveraging the advanced features of modern interconnects. In this paper we design a new Advanced Data Transfer Service (ADTS) with the capabilities such as zero-copy data-transfer, memory registration cache, persistent data sessions and pipelined data transfer etc. to enable efficient zero-copy data transfers over IB and iWARP equipped LAN and WAN. We then utilize ADTS to design a high performance FTP library (FTP-ADTS). From our experimental results, we observe that our design outperforms existing sockets based approaches by more that 95 in transferring large volumes of data over LAN. It also provides significantly better performance at much lower (by up to a factor of 6) CPU utilization in various IB WAN scenarios. These results present the promising future for designing high performance communication protocols to power the efficiency and scalability of next-generation parallel and distributed environments.

Though convergence has been a buzzword in the networking industry for sometime now, no vendor has successfully brought out a solution which combines the ubiquitous nature of Ethernet with the low latency and high performance capabilities that InfiniBand offers. Most of the overlay protocols introduced in the past have had to bear with some form of performance trade off or overhead. Recent advances in InfiniBand interconnect technology has allowed vendors to come out with a new model for network convergence — RDMA over Ethernet (RDMAoE). In this model, the IB packets are encapsulated into Ethernet frames thereby allowing us to transmit them seamlessly over an Ethernet network. The job of translating InfiniBand addresses to Ethernet addresses and back is taken care of by the InfiniBand HCA. This model, allows end users access to large computational clusters through the use of ubiquitous Ethernet interconnect technology while retaining the high performance, low latency guarantees that InfiniBand provides. In this paper, we present a detailed evaluation and analysis of the new RDMAoE protocol as opposed to the earlier overlay protocols as well as native-IB and socket based implementations. Through these evaluations, we also look at whether RDMAoE brings us closer the eventual goal of network convergence. The experimental results obtained with verbs, MPI, application and data center level evaluations show that RDMAoE is capable of providing performance comparable to Native-IB based applications on a standard 10GigE network.

Modern high-end computing systems utilize specialized offload engines to enhance various aspects of their processing. For example, high-speed networks such as InfiniBand, Quadrics and Myrinet utilize specialized hardware to offload network processing to help improve performance. However, such hardware units are expensive, and their manufacturing complexity increases exponentially depending on the number and complexity of tasks they offload. On the other hand, the proliferation of multi- and many-core processors into the general desktop and laptop markets is increasingly driving their cost down due to the economies of scale. To take advantage of the obvious benefits of multi/many-core architectures, we propose, design and evaluate ProOnE , a general purpose Protocol Onload Engine. ProOnE utilizes a small subset of the available cores on a multi-core CPU to ‘‘onload’’ various tasks in a dedicated manner instead of ‘‘offloading’’ them to specialized hardware. The general purpose processing capabilities of multi-core architectures allow ProOnE to be designed in a flexible, extensible and scalable manner, while benefiting from the reducing costs of general-purpose CPUs. In this paper, we onload onto ProOnE, several tasks relevant to communication sub-systems such as MPI that are too complex for current hardware offload engines to support, and demonstrate significant benefits in terms of overlap of computation and communication and improved application performance.

Efficient communication design is essential for HEC applications to scale to thousands and 10s of thousands of processors on current and upcoming systems. Traditionally, nonblocking two-sided semantics from MPI-1 have been used to achieve latency hiding when optimizing communication for HEC applications. The more recently proposed one-sided semantics of MPI-2 can also be used to achieve latency hiding but have not been evaluated at large scale. For this study, we investigate application level optimizations for latency hiding in conjunction with one-sided and non-blocking two-sided implementations of nearneighbor communication in a stencil-based earthquake-induced ground motion simulation code, AWM-OLSEN from the Southern California Earthquake Center. We implement reduced synchronization and communication-computation overlap, two key mechanisms for latency hiding. We use Post-Wait/Start-Complete calls or Fence calls in the one-sided versions and ISend/IRecv/Waitall calls in the two-sided version. Our experiments on TACCś Ranger system show that these mechanisms can provide similar benefits in both the onesided and two-sided models and demonstrate use of one-sided communication at large scale. We see a 7% improvement in the overall application performance on 4,096 cores.

Message Oriented Middleware (MOM) is a key technology in financial market data delivery In this context we study the Advanced Message Queuing Protocol (AMQP), an emerging open standard for MOM communication. We design a basic suite of benchmarks for AMQP's Direct, Fanout, and Topic Exchange types. We then evaluate these benchmarks with Apache Qpid, an open source implementation of AMQP In order to observe how AMQP performs in a real-life scenario, we also perform evaluations with a simulated stock exchange application. All our evaluations are performed over InfiniBand as well as 1 Gigabit Ethernet networks. Our results indicate that in order to achieve the high scalability requirements demanded by high performance computational finance applications, we need to use modern communication protocols, like RDMA, which place less processing load on the host. We also find that the centralized architecture of AMQP presents a considerable bottleneck as far as scalability is concerned.

FTP has been the most popular method to transfer large files for data-staging, replication, and the like. While existing FTP mechanisms have improved gradually with newer networks, they still inherit the fundamental limitations imposed by the underlying networking protocols (TCP/UDP) they use. These include limited network bandwidth utilization, high mem- ory bandwidth and CPU utilization that TCP/UDP cause on the end-nodes. Thus both the performance and scalability of such systems is limited. The advent of InfiniBand (IB) WAN has enabled the use of high performance transport protocols in the WAN scenarios, which can be leveraged for designing FTP mechanisms. Enabling IB-based FTP capabilities and providing good efficiency for such transfers presents considerable challenge. In this paper we present an Advanced Data Transfer Service (ADTS) to enable efficient data transfers over WAN. We leverage the ADTS's capabilities to design high performance file transfer mechanisms (FTP based on ADTS). Our ADTS layer improves data transfer performance by optimizing several aspects includ- ing efficient buffer management, memory registration cache, pipelining of data transfers, reducing TCP/IP related data copies, and maintaining persistent FTP data sessions. Further, we reduce the CPU utilization required for the data-transfers (by up to a factor of 6) and demonstrate a significantly higher FTP server scalability. In our experimental results, we observe that our FTP-ADTS design outperforms existing TCP and UDP based approaches by more that 80% in transferring large volumes of data. In addition, we utilize the WAN emulation capabilities of Obsidian InfiniBand WAN routers to study the impact of our designs in a wide range of WAN scenarios, leading to solutions that enable the design of highly capable WAN communication protocols required to power the next-generation high perfor- mance parallel and distributed environments.

Recovering the three-dimensional (3D) object shape lies as an unresolved and active research topic on the cross-section of computer vision, photogrammetry and bioinformatics. Although various techniques have been developed to tackle the shape recovery problems, the computational complexity and the constraints introduced by the other algorithms have limited the applicability of these methods in real world problems. In this paper, we propose a method that is based on the projective geometry between the object space and silhouette images taken from multiple viewing angles. The approach eliminates the requirements of dense feature matching and camera calibration that are generally adopted by other reconstruction method. The object is reconstructed by setting a set of hypothetical planes slicing the object volume and estimating the projective geometric relations between the images. The experimental results show that satisfactory 3D model can be generated by applying minimal constraints.

Distributed applications tend to have a complex design due to issues such as concurrency, synchronization and communication. Researchers in the past have proposed simpler abstractions to hide these complexities. However, many of the proposed techniques use messaging protocols which incur high overhead and are not very scalable. To address these limitations, in our previous work [20], we proposed an efficient Distributed Data Sharing Substrate (DDSS) using the features of high-speed networks. In this paper, we propose several design optimizations for DDSS in multi-core systems such as the combination of shared memory and message queues for inter-process communication, dedicated thread for communication progress and for onloading DDSS operations such as get and put. Our micro-benchmark results not only show a very low latency in DDSS operations but also demonstrate the scalability of DDSS with increasing number of processes. Application evaluations with R-Tree and B-Tree query processing and distributed STORM shows an improvement of up to 56%, 45% and 44%, respectively, as compared to traditional implementations. Evaluations with application checkpointing using DDSS demonstrate the scalability with increasing number of checkpointing applications. Further, in our evaluations, we demonstrate the portability of DDSS across multiple modern interconnects including InfiniBand and iWARP-capable 10-Gigabit Ethernet networks (applicable for both LAN/WAN environments).

Current data-centers employ admission control mechanism to maintain low response time and high throughput under overloaded scenarios. Existing mechanisms use internal (on the overloaded server) or external (on the front- end proxies) approaches. External admission control is preferred since it can be performed transparently without any modifications to the overloaded servers and global decisions can be made based on the load information of all the back-end servers. However, this external approach is bound to use TCP/IP communication protocol to get the load information from the back-end servers and rely on coarse-grained load monitoring due to the overheads associated with fine-grained monitoring. In this paper, we provide a fine-grained external admission control mechanism by leveraging the one-sided RDMA feature of modern interconnects and consequently provide response time guarantees and overload control in the multi-tiered data-center environment. Our design is implemented over InfiniBand-based clusters working in conjunction with Apache based servers. Experimental evaluations with single file, world cup and zipf traces show that our admission control can improve the response time by up to 28%, 17% and 23%, respectively, as compared to performing TCP/IP-based admission control. The improvement become 51%, 36% and 42%, respectively, when comparing to the base performance without any admission control. Further, our evaluations also show that RDMA-based admission control can provide better QoS guarantees than the TCP/IP-based admission control and no admission control approaches.

The emergence of multi-core processors has made MPI intra-node communication a critical component in high performance computing. In this paper, we use a three-stepmethodology to design an efficient MPI intra-node communication scheme from two popular approaches: shared memory and OS kernel-assisted direct copy. We use an Intel quad-core cluster for our study. We first run microbenchmarks to analyze the advantages and limitations of these two approaches, including the impacts of processor topology, communication buffer reuse, process skew effects, and L2 cache utilization. Based on the results and the analysis, we propose topology-aware and skew-aware thresholds to build an optimized hybrid approach. Finally, we evaluate the impact of the hybrid approach on MPI collective operations and applications using IMB, NAS, PSTSWM, and HPL benchmarks. We observe that the optimized hybrid approach can improve the performance of MPI collective operations by up to 60%, and applications by up to 17%.

As computing clusters grow in size and core count, congestion at the host as well as the network level is becoming a real problem in modern high performance computing. This is especially true in the popular cluster of cluster scenario where bottleneck links are readily available. With the advent of long range capable InfiniBand devices, we are now looking at Cluster of cluster scenarios that may span across continents. Providing Quality of Service for all applications in such scenarios pose a formidable challenge for cluster designers as well as administrators. On the other hand, modern high performance network interconnects like InfiniBand now provides us with advanced features like fine grained network level bandwidth provisioning. Such features give us the ability to create networks where we can minimize common network problems such as small message starvation and the like. The emerging trends in modern high performance computing mentioned above, gives us a strong motivation to explore QoS provisioning for high performance networks from a quantitative angle. Though current designs allow us to specify the priority for a job at the job launch time, there exists no way to ensure that this priority is enforced once the job is actually running on the network. Through our research, we aim to make the MPI Stack QoS-aware and ensure that the priority settings are adhered to throughout the life cycle of the job resulting in an overall improvement in application performance. To the best of our knowledge, this is the first such QoS-aware MPI design and evaluation for InfiniBand-based clusters. We explore various design alternatives such as Inter-Job and Intra-Job bandwidth partitioning by use of multiple service levels and effective use of NIC level buffers by utilizing multiple Virtual Lanes for data transmission. Our experiments show a 220% performance improvement in small message latency and 90% performance increase in large message latency can be achieved on giving increased priority

High performance interconnects such as InfiniBand (IB)have enabled large scale deployments of High Performance Computing (HPC) systems. High performance communication and IO middleware such as MPI and NFS over RDMA have also been redesigned to leverage the performance of these modern interconnects. With the advent of long haul InfiniBand (IB WAN), IB applications now have inter-cluster reaches. While this technology is intended to enable high performance network connectivity across WAN links,it is important to study and characterize the actual performance that the existing IB middleware achieve in these emerging IB WAN scenarios. In this paper, we study and analyze the performance characteristics of the following three HPC middleware: (i)IPoIB (IP traffic over IB), (ii) MPI and (iii) NFS over RDMA. We utilize the Obsidian IB WAN routers for inter-cluster connectivity. Our results show that many of the applications absorb smaller network delays fairly well. However, most approaches get severely impacted in high delay scenarios. Further, communication protocols need to be optimized in higher delay scenarios to improve the performance. In this paper, we propose several such optimizations to improve communication performance. Our experimental results show that techniques such as WAN-aware protocols, transferring data using large messages (message coalescing) and using parallel data streams can improve the communication performance (up to 50%) in high delay scenarios. Overall, these results demonstrate that IB WAN technologies can enable cluster-of-clusters architecture as a feasible platform for HPC systems.