The Analysis of Functional Neuroimaging (AFNI) suite [1], a set of C programs and auxiliary scripts for processing, analyzing, and displaying functional Magnetic Resonance Imaging (fMRI) data (a technique for mapping human brain activity), is a widely adopted open-source tool in the MRI data analysis community. For many types of analysis pipelines, a key step is to register a subject's image to a pre-defined template so different images can be compared within a normalized coordinate. Although a 12-point affine transformation that includes translation, rotation, scaling, and shear works fine for some standard cases, it is usually found to be insufficient for voxel-wise types of analyses. The need for some other approach is exacerbated if the subject has brain atrophy due to some kind of neurological conditions such as Parkinson's disease. The 3dQwarp code in AFNI is a non-linear image registration procedure that overcomes the drawbacks of a canonical affine transformation. However, the existing OpenMP parallelization in 3dQwarp does not scale well when warping at an ultra fine level, and the hard-coded number of iterations of the iterative algorithm also limits the accuracy. Based on profiling and benchmark analysis, we improve the parallel efficiency of 3dQwarp by the optimization of its OpenMP structure and obtain about a 2x speedup for a normalized workload. With the incorporation of convergence criteria, we are able to perform warping at a much finer resolution than before and achieve on average 20% improvement in accuracy with respect to Pearson correlation measure.

The Beacon Project is exploring the impact of the Intel Xeon Phi coprocessor on scientific supercomputing for a wide variety of computational science and engineering applications.

The near exponential growth in sequence data available to bioinformaticists, and the emergence of new fields of biological research, continue to fuel an incessant need for in- creases in sequence alignment performance. Today, more than ever before, bioinformatics researchers have access to a wide variety of HPC architectures including high core count Intel Xeon processors and the many-core Intel Xeon Phi.In this work, the implementation of a distributed, NCBI compliant, BLAST+ (C++ toolkit) code, targeted for multi- and many-core clusters, such as those containing the Intel Xeon Phi line of products is presented. The solution is robust: distributed BLAST runs can use the CPU only, the Xeon Phi processor or coprocessor, or both by utilizing the CPU or Xeon Phi processor plus a Xeon Phi coprocessor. The distributed BLAST implementation employs static load balancing, fault tolerance, and contention aware I/O. The distributed BLAST implementation, HPC-BLAST, maintains greater than 90% weak scaling efficiency on up to 160 Xeon Phi (Knights Landing) nodes.The source code and instructions, are available under the Apache License, Version 2.0 at https://github.com/UTennessee-JICS/HPC-BLAST.

In the study of Parkinson's disease (PD), substantial research has shown a variety of findings within brain MR images that differ on a group-wise level between individuals with PD and healthy adults, including resting brain activity and blood flow [1, 2], diffusion tensor fractional anisotropy [3], and brain iron deposition [4, 5]. Despite these promising findings, none of these putative biomarkers have become utilized in the field. We believe this is the case principally because it has been very difficulty to register findings visible at a group level to individual brains because of the vast amount of inter-subject variability within the human population. While specific brain structures are common, the precise location, distribution, and size of these structures can vary dramatically between individuals. MR images essentially represent vast unstructured datasets. In the United States in 2011, over 8.7 million MRI brain scans sessions (each comprised of multiple brain imaging sequences) occurred. It is clear that to make practical use of this massive amount of imaging data, not only for PD but also for analysis of other conditions of the brain, high performance computing solutions will be required. There are several software packages available for image registration. For non-linear image registration, the 3dQwarp code in the Analysis of Functional Neuroimaging (AFNI) package [6] designed by NIH provides the best result in terms of normalized cross correlation, mutual information and sum of squared differences [7]. The AFNI 3dQwarp code follows the h-method in the language of the finite element analysis, which divides the image into shrinking patches and warps the source image to the template image in incremental steps. In each step, the patches are optimized in at most four sweeps (one by default). We find that for some irregular images, the results are not satisfactory, related to insufficient convergence of the solution space at each patch size. One solution to insufficient convergence uses is a linear registration step. However, our dataset (Michael J. Fox Parkinson's Progressive Markers Initiative (PPMI), a well characterized multidimensional dataset with substantial imaging data.) indicates that the linear registration in AFNI is not always well performed for PD subjects and introduces unnecessary noises to the downstream analysis. We propose a new warping procedure, which starts with an image in original space, and applies non-linear registration directly. The important part is to make sure the solution has converged at each warping step before moving to the next finer optimization step. With the new procedure, our results show about 20% improvement compared to the original procedure in term of the Pearson correlation. The practical ramifications of this improvement were apparent in the end result; before adopting this process 58 of 161 subjects were judged insufficiently converged for analysis by the research user team (Skidmore, Anthony, Marstrander above). Subsequent to the institution of the new process, while computational load increased, only 3 of 158 subjects were rejected as insufficiently registered. Preliminary scaling studies demonstrate that the current implementation of 3dQwarp with OpenMP directives are not sufficient to take advantage of the number of cores available either on current day CPUs, or the number of cores available on the many-core Intel Xeon Phi. For both clinical and research utilization, registration code can be improved using high performance multi-thread computation. Therefore, a future direction will be to implement a version of 3dQwarp that does scale to a larger number of threads to assist in developing analytic turnaround times that will be more relevant for both research and clinical settings.

Given the growing popularity of accelerator-based supercomputing systems, it is beneficial for applications software programmers to have cognizance of the underlying platform and its workings while writing or porting their codes to a new architecture. In this work, the authors highlight experiences and knowledge gained from porting such codes as ENZO, H3D, GYRO, a BGK Boltzmann solver, HOMME-CAM, PSC, AWP-ODC, TRANSIMS, and ASCAPE to the Intel Xeon Phi architecture running on a Cray CS300ACTMCluster Supercomputer named Beacon. Beacon achieved 2.449 GFLOP/W in High Performance LINPACK (HPL) testing and a number one ranking on the November 2012 Green500 list [1]. Areas of optimization that bore the most performance gain are highlighted, and a set of metrics for comparison and lessons learned by the team at the National Institute for Computational Sciences Application Acceleration Center of Excellence is presented, with the intention that it can give new developers a head start in porting as well as a baseline for comparison of their own code’s exploitation of fine and medium-grained parallelism. Keywords-Intel Xeon Phi, MIC, accelerators, performance metrics, CS300-AC Disclaimer: Figures 1, 2, and 3 contain material that is copyrighted by Intel R © Corporation. Intel R ©, the Intel R © logo, XeonTM and XeonTM logo, Xeon PhiTM and Xeon PhiTM logo are trademarks or registered trademarks of Intel R © Corporation or its subsidiaries in the United States and other countries. Material used with permission.

To explore the potential impact of the Intel® Many Integrated Core (Intel® MIC) architecture and related programming models on the supercomputing community, several applications are ported to and subsequently studied on a 50-plus-core Intel® Xeon Phi™ coprocessor deployed at the National Institute for Computational Sciences (NICS). One of the studied applications -- a 2D Boltzmann-BGK solver -- employs the native execution model, in which the Intel Xeon Phi acts as a standalone compute node on which parallel code is executed directly via MPI or interactive login. Another application -- a Poisson solver -- employs both the native and offload execution models, the latter of which migrates computational kernels or sections of code that are flagged by special pragmas to the Intel Xeon Phi for execution while the primary code runs on the Intel® Xeon® processor. A final application -- a trapezoidal rule integration scheme -- is instrumented to run on the Intel Xeon Phi and the Intel Xeon processor simultaneously, and the results of different levels of heterogeneity in execution are studied. Information regarding the porting and scalability of each application code is presented.

High performance computing resources attract a wide range of computational users and corresponding job widths and lengths. For example, on the petaflop Cray XT5 machine, Kraken, users submit jobs ranging from a few hundred cores (capacity computing) to over hundred thousand cores (capability computing). Traditionally it has been difficult to maintain high utilization while juggling such a diverse job mix. This paper explores four unique approaches to achieve our scheduling goals of maximizing utilization on four distinct resources at the National Institute for Computational Sciences. The resources include the petaflop machine, Kraken, Athena — a 166 TF Cray XT4, a 4 TB shared memory NUMA machine called Nautilus, and a GPU cluster called Keeneland.

I/O performance in scientific applications is an often neglected area of concern during performance optimizations. However, various scientific applications have been identified which benefit from I/O improvements due to the volume of data or number of compute processes utilized. This work details the I/O patterns and data layouts of real scientific applications, discusses their impacts, and demonstrates pragmatic approaches to improve

In late 2009, the National Institute for Computational Sciences placed in production the world‘s fastest academic supercomputer (third overall, November 2009 Top500), a Cray XT5 named Kraken. Currently Kraken provides 1.17 Petaflops through 112,896 compute nodes accounting for over 60% of the total cycles available to the National Science Foundation users via the TeraGrid. Kraken has two missions that have proven difficult to simultaneously reconcile: providing the maximum number of total cycles to the community, while enabling full machine runs for “hero” users. Historically, this has been attempted by allowing schedulers to choose the time for the beginning of large jobs, with a concomitant reduction in utilization. This paper outlines a novel approach implemented at NICS, whereby the “draining” of the system is forced on a weekly basis, followed by consecutive full machine runs. As previous simulation results suggested, this led to utilization of over 90% (the equivalent of a 300+ Teraflop supercomputer, or several million dollars of compute time per year) with a 92% average over the

This work examines the accuracy of a parallel moving least squares algorithm for solving the governing equations of the hydrodynamic formulation of quantum mechanics. The algorithm solves the associated linear least squares problems using either normal equations or QR factorization. The accuracy of the algorithm is studied for both the free particle and the harmonic oscillator, and the results of a series of experiments designed to determine the spatial and temporal dependence of the accuracies are presented. In closing, a few qualitative observations concerning the performance of the algorithm are offered for consideration.

This article documents the first implementation of a parallel algorithm for solving the governing equations of the hydrodynamic formulation of quantum mechanics. The algorithm employs a quantum trajectory method (QTM) based on the serial algorithm introduced by Wyatt and Lopreore. The OpenMP API is employed to parallelize the code across the quantum trajectories in a shared memory environment. An outline of the parallel algorithm is provided; the analytical solution for a moving free particle is used to verify the solution obtained by the parallel algorithm. Further validation against several of the results obtained by Wyatt and Lopreore is also provided. The parallel speedups and runtimes are presented, and several performance issues are noted. Finally, the results of a preliminary accuracy study examining a moving free particle are provided. © 2001 John Wiley & Sons, Inc. Int J Quantum Chem, 2001