We present a parallel AND/OR Branch-and-Bound scheme that uses the power of a computational grid to push the boundaries of feasibility for combinatorial optimization. Two variants of the scheme are described, one of which aims to use machine learning techniques for parallel load balancing. In-depth analysis identifies two inherent sources of parallel search space redundancies that, together with general parallel execution overhead, can impede parallelization and render the problem far from embarrassingly parallel. We conduct extensive empirical evaluation on hundreds of CPUs, the first of its kind, with overall positive results. In a significant number of cases parallel speedup is close to the theoretical maximum and we are able to solve many very complex problem instances orders of magnitude faster than before; yet analysis of certain results also serves to demonstrate the inherent limitations of the approach due to the aforementioned redundancies.

We present parallel AND/OR Branch-and-Bound which uses the power of a computational grid to push the boundaries of feasibility for combinatorial optimization. Two variants of the scheme are described, one of which aims to use machine learning techniques for parallel load balancing. Indepth analysis identifies two inherent sources of parallel search space redundancies that, together with general parallel execution overhead, can impede parallelization and render the problem far from embarrassingly parallel. We conduct extensive empirical evaluation on hundreds of CPUs, the first of its kind, with overall positive results. In a significant number of cases parallel speedup is close to the theoretical maximum and we are able to solve many very complex problem instances orders of magnitude faster than before; yet analysis of certain results also serves to demonstrate the inherent limitations of the approach due to the aforementioned redundancies.

In empirical studies we observed that caching can have very little impact in reducing the search effort in Branch and Bound search over context-minimal OR spaces. For example, in one of the problem domains used in our experiments we reduce only by 1% the number of nodes expanded when using caching in context-minimal OR spaces. By contrast, we reduce by 74% the number of nodes expanded when using caching in context-minimal AND/OR spaces on the same instances. In this work we document this unexpected empirical finding and provide explanations for the phenomenon.

One popular and efficient scheme for solving combinatorial optimization problems over graphical models exactly is depth-first Branch and Bound. However, when the algorithm exploits problem decomposition using AND/OR search spaces, its anytime behavior breaks down. This article 1) analyzes and demonstrates this inherent conflict between effective exploitation of problem decomposition (through AND/OR search spaces) and the anytime behavior of depthfirst search (DFS), 2) presents a new search scheme to address this issue while maintaining desirable DFS memory properties, and 3) analyzes and demonstrates its effectiveness through comprehensive empirical evaluation. Our work is applicable to any problem that can be cast as search over an AND/OR search space.

We address the problem of predicting the size of the search tree explored by Depth-First Branch and Bound (DFBnB) while solving optimization problems over graphical models. Building upon methodology introduced by Knuth and his student Chen, this paper presents a memory-efficient scheme called Retentive Stratified Sampling (RSS). Through empirical evaluation on probabilistic graphical models from various problem domains we show impressive prediction power that is far superior to recent competing schemes.

MOTIVATION:The use of dense single nucleotide polymorphism (SNP) data in genetic linkage analysis of large pedigrees is impeded by significant technical, methodological and computational challenges. Here we describe Superlink-Online SNP, a new powerful online system that streamlines the linkage analysis of SNP data. It features a fully integrated flexible processing workflow comprising both well-known and novel data analysis tools, including SNP clustering, erroneous data filtering, exact and approximate LOD calculations and maximum-likelihood haplotyping. The system draws its power from thousands of CPUs, performing data analysis tasks orders of magnitude faster than a single computer. By providing an intuitive interface to sophisticated state-of-the-art analysis tools coupled with high computing capacity, Superlink-Online SNP helps geneticists unleash the potential of SNP data for detecting disease genes.RESULTS:Computations performed by Superlink-Online SNP are automatically parallelized using novel paradigms, and executed on unlimited number of private or public CPUs. One novel service is large-scale approximate Markov Chain-Monte Carlo (MCMC) analysis. The accuracy of the results is reliably estimated by running the same computation on multiple CPUs and evaluating the Gelman-Rubin Score to set aside unreliable results. Another service within the workflow is a novel parallelized exact algorithm for inferring maximum-likelihood haplotyping. The reported system enables genetic analyses that were previously infeasible. We demonstrate the system capabilities through a study of a large complex pedigree affected with metabolic syndrome.AVAILABILITY:Superlink-Online SNP is freely available for researchers at http://cbl-hap.cs.technion.ac.il/superlink-snp. The system source code can also be downloaded from the system website.CONTACT:omerw@cs.technion.ac.ilSUPPLEMENTARY INFORMATION:Supplementary data are available at Bioinformatics online.

This thesis presents substantial enhancements to the state of the art in combinatorial optimization over graphical models. Our contributions are relevant in the context of both exact and approximate reasoning over Bayesian and Markov networks, weighted constraint satisfaction problems, and other related queries. While the focus of this work is on probabilistic and constraint inference, we also draw from the areas of distributed computing and statistical learning. Relevant practical applications we consider include genetic linkage analysis, protein side-chain prediction, medical diagnosis, resource scheduling, and signal processing. We extend AND/OR Branch-and-Bound (AOBB), a leading algorithm for optimization queries over graphical models. AOBB applies the principle of depth-first branch-and-bound to AND/OR search spaces, which exploit conditional independencies via problem decomposition and merge unifiable subproblems through caching of partial solutions. This thesis presents fundamental extensions to AOBB in three regards. First, we significantly improve the applicability of AOBB as an approximation scheme. We analyze and demonstrate the inherent conflict between problem decomposition (through AND/OR search spaces) and the anytime behavior of AOBB and depth-first search in general. We introduce a new algorithm, Breadth-Rotating AND/OR Branch-and-Bound (BRAOBB), which drastically improves upon AOBB with respect to its anytime performance while maintaining desirable depth-first complexity guarantees. Comprehensive analysis and experimental evaluation demonstrate the scheme's effectiveness. Furthermore, our entry based on BRAOBB placed first in all three optimization tracks of the PASCAL 2012 Probabilistic Inference Challenge. Second, we investigate the instance-based run-time complexity of AOBB. The asymptotic worst-case bounds are both time and space exponential in the problem's induced width, but often prove to be very loose due to the algorithm's powerful pruning, as we show empirically. We identify a range of (sub)problem features and develop learning schemes to estimate run-time complexity based on statistical regression analysis. We conduct extensive experimental evaluation within and across various problem classes and demonstrate convincing predictive performance. Third, we describe a parallel AND/OR Branch-and-Bound scheme that pushes the boundaries of feasibility for exact reasoning by orders of magnitude. We adapt the paradigm of parallel tree search to AND/OR search spaces; our implementation distributes conditioned subproblems on a grid of independent computers. In this context, we show how the pruning power of AOBB can cause large variance in subproblem complexity, which makes load balancing extremely elusive and impairs parallel performance. We thus propose load balancing based on the run-time estimation scheme presented earlier in the thesis, learning a complexity model offline from previously solved subproblems. Through experimental results using hundreds of computers on problem instances from a variety of classes we show convincing parallel performance with several orders of magnitude speedup over sequential AOBB, but we also highlight and analyze some inherent limitations. Our implementations of AOBB, BRAOBB and parallel AOBB are available online under an open-source license.

This paper provides algorithms for predicting the size of the Expanded Search Tree (EST) of Depth-first Branch and Bound algorithms (DFBnB) for optimization tasks. The prediction algorithm is implemented and evaluated in the context of solving combinatorial optimization problems over graphical models such as Bayesian and Markov networks. Our methods extend to DFBnB the approaches provided by Knuth-Chen schemes that were designed and applied for predicting the EST size of backtracking search algorithms. Our empirical results demonstrate good predictions which are superior to competing schemes.

We develop several algorithms taking advantage of two common approaches for bounding MPE queries in graphical models: mini-bucket elimination and message-passing updates for linear programming relaxations. Both methods are quite similar, and offer useful perspectives for the other; our hybrid approaches attempt to balance the advantages of each. We demonstrate the power of our hybrid algorithms through extensive empirical evaluation. Most notably, a Branch and Bound search guided by the heuristic function calculated by one of our new algorithms has recently won first place in the PASCAL2 inference challenge.

We describe a distributed version of an advanced branch and bound algorithm over graphical models. The crucial issue of load balancing is addressed by estimating subproblem complexity through learning, yielding impressive speedups on various hard problems using hundreds of parallel CPUs.

We study the problem of complexity estimation in the context of parallelizing an advanced Branch and Bound-type algorithm over graphical models. The algorithm's pruning power makes load balancing, one crucial element of every distributed system, very challenging. We propose using a statistical regression model to identify and tackle disproportionally complex parallel subproblems, the cause of load imbalance, ahead of time. The proposed model is evaluated and analyzed on various levels and shown to yield robust predictions. We then demonstrate its effectiveness for load balancing in practice.

We study iterative randomized greedy algorithms for generating (elimination) orderings with small induced width and state space size — two parameters known to bound the complexity of inference in graphical models. We propose and implement the Iterative Greedy Variable Ordering (IGVO) algorithm, a new variant within this algorithm class. An empirical evaluation using different ranking functions and conditions of randomness, demonstrates that IGVO finds significantly better orderings than standard greedy ordering implementations when evaluated within an anytime framework. Additional order of magnitude improvements are demonstrated on a multi-core system, thus further expanding the set of solvable graphical models. The experiments also confirm the superiority of the MinFill heuristic within the iterative scheme.

This paper describes our entry for the MAP/MPE track of the PA SC L 2011 Probabilistic Inference Challenge, which placed first in al l three time limit categories, 20 seconds, 20 minutes, and 1 hour. Our baseline is a b ranch-and-bound algorithm that explores the AND/OR context-minimal search graph of a graphical model guided by a mini-bucket heuristic. Augmented with rec ent advances that convert the algorithm into an anytime scheme, that improve t he heuristic power via cost-shifting schemes, and using enhanced variable ord ering schemes, it constitutes one of the most powerful MAP/MPE inference methods t date.

In the context of distributed Branch and Bound Search for Graphical Models, effective load balancing is crucial yet hard to achieve due to early pruning of search branches. This paper proposes learning a regression model over structural as well as cost function-based features to more accurately predict subproblem complexity ahead of time, thereby enabling more balanced parallel workloads. Early results show the promise of this approach.

We investigate a hybrid of two styles of algorithms for deriv ing bounds for optimization tasks over graphical models: non-iterative mes sage-passing schemes exploiting variable duplication to reduce cluster sizes (e .g. MBE) and iterative methods that re-parameterize the problem’s functions aimi ng to produce good bounds even if functions are processed independently (e.g. MPLP). In this work we combine both ideas, augmenting MBE with re-parameteriza tion, which we call MBE with Moment Matching (MBE-MM). The results of prelimina ry empirical evaluations show the clear promise of the hybrid scheme over its individual components (e.g., pure MBE and pure MPLP). Most significantly, w e demonstrate the potential of the new bounds in improving the power of mechani c lly generated heuristics for branch and bound search.

General pedigrees can be encoded as Bayesian networks, where the common MPE query corresponds to finding the most likely haplotype configuration. Based on this, a strategy for grid parallelization of a state-of-the-art Branch and Bound algorithm for MPE is introduced: independent worker nodes concurrently solve subproblems, managed by a Branch and Bound master node. The likelihood functions are used to predict subproblem complexity, enabling efficient automation of the parallelization process. Experimental evaluation on up to 20 parallel nodes yields very promising results and suggest the effectiveness of the scheme, solving several very hard problem instances. The system runs on loosely coupled commodity hardware, simplifying deployment on a larger scale in the future.

One popular and efficient scheme for solving exactly MPE/MAP and related problems over graphical models is depth-first Branch and Bound. However, when the algorithm exploits problem decomposition using AND/OR search spaces, its anytime behavior breaks down. This paper 1) analyzes and demonstrates this inherent conflict between effective exploitation of problem decomposition (through AND/OR search spaces) and the anytime behavior of depth-first search (DFS), 2) presents a first scheme to address this issue while maintaining desirable DFS memory properties, 3) analyzes and demonstrates its effectiveness. Our work is applicable to any problem that can be cast as search over an AND/OR search space.

A strategy for parallelization of a state-of-the-art Branch and Bound algorithm for weighted CSPs and other graphical model optimization tasks is introduced: independent worker nodes concurrently solve subproble ms, managed by a Branch and Bound master node; the problem cost functions are us ed to predict subproblem complexity, enabling efficient load balancing, which is cr ucial for the performance of the parallelization process. Experimental evalu ation on up to 20 nodes yields very promising results and suggests the effectivenes s of the scheme. The system runs on loosely coupled commodity hardware, simp lify ng deployment on a larger scale in the future.

We introduce a strategy for parallelizing a state-of-the-art sequential search algorithm for optimization on a grid of computers. Based on the AND/OR graph search framework, the procedure exploits the structure of the underlying problem graph. Worker nodes concurrently solve subproblems that are generated by a single master process. Subproblem generation is itself embedded into an AND/OR Branch and Bound algorithm and dynamically takes previous subproblem solutions into account. Drawing upon the underlying graph structure, we provide some theoretical analysis of the parallelization parameters. A prototype has been implemented and we present promising initial experimental results on genetic haplotyping and Mastermind problem instances, at the same time outlining several open questions.