Hierarchical Data Format (HDF) is a data file format designed by the National Center for Supercomputing Applications (NCSA) to assist users in the storage and manipulation of scientific data across diverse operating systems and machines. NCSA developed a library of callable routines and a set of utility programs and tools for creating and using HDF files. This work is now performed by The HDF Group (THG).

Given a hyperedgeand vertex-weighted hypergraph H = (V, E), a k-way partitioning of V assigns the vertices to k disjoint nonempty partitions. The k-way partitioning problem seeks to minimize a given cost function c(P ) whose arguments are partitionings. A standard cost function is net cut,1 which is the sum of weights of hyperedges that are cut by the partitioning (a hyperedge is cut exactly when not all of its vertices are in one partition). Constraints are typically imposed on the partitioning solution, and make the problem difficult. For example, certain vertices can be fixed in particular partitions (fixed constraints). Or, the total vertex weight in each partition may be limited (balance constraints), which results in an NP-hard formulation [21]. Thus, the cost function c(P ) is minimized over the set of feasible solutions Sf , which is a subset of the set of all possible k-way partitionings. Effective move-based heuristics for k-way hypergraph partitioning have been pioneered in such works as [10], [6], [9], with refinements given by [38]. [43], [26], [40], [18], [4], [12], [25], [34], [19] and many others. A comprehensive survey of partitioning formulations and algorithms, centered on VLSI applications and covering move-based, spectral, flow-based, mathematical programming-based, etc. approaches, is given in [5]. A recent update on balanced partitioning in VLSI physical design is provided by [31].

In this paper we attempt to parallelise a commodity database for OLAP on a cluster of commodity PCs by using a distributed high-performance storage subsystem. By parallelising the underlying storage architecture we eliminate the need to make any changes to the database software. We look at two options that differ in their complexity and features: Distributed devices and distributed file systems. The former aggregates several single disks within the cluster into a RAID device across the network. The latter offers all the features of a real file system at the price of a considerably increased complexity. We configured a Linux version of ORACLE to run on various distributed devices or distributed file systems, respectively, and ran a TPC-D benchmark on our cluster of commodity PCs interconnected by a Gigabit Ethernet. While distributed devices achieve at least the performance of local disks, they offer the benefit of using all surplus storage in a cluster. The distributed file systems seem to run into performance problems due to their increased complexity. We explain the experimental results with an analytic model of the cluster architecture and include a comparison of the same workload on an architecture that distributes the TPC-D queries at a higher level (and not just the underlying storage system). We conclude with suggestions for higher performances in future clusters of commodity PCs.