In this chapter, we explore some naive approaches to solving BBO problems, and explain why they are not acceptable. This will lead to a better understanding of why DFO is preferred for real applications and provide some foundational material that is used throughout this book.

Model-based methods in DFO proceed from the idea that if it is possible to build a “good” model of the true objective function, then information from the model can be used to guide the optimization. In Chapter 9 , we studied several methods for constructing model functions using only objective function evaluations. We also defined fully linear, as a term to mean that the optimiser has access to an accuracy parameter $$\Delta $$ that can be used to drive the error in the model to 0 in a predictable way. We now turn our attention to how to employ model functions in unconstrained DFO algorithms.

Optical computed tomography scanners are powerful imaging tools used in three-dimensional radiation dosimetry to reconstruct dose distributions in gel dosimeters. Solid-tank scanners embed the gel in an acrylic block, replacing large fluid baths and simplifying setup. The solid-tank, fan-beam design is formulated as a multiobjective problem that balances imaging extent, uniformity and ray-crossover artifacts. A ray-tracing simulator is developed with new, objective-specific performance measures and a multifidelity setting controlled by ray count. To enable efficient search, the non-dominated sorting genetic algorithm (NSGA-II) is employed within a multifidelity framework that incrementally escalates simulation fidelity and uses the Pareto-optimal survivors from one fidelity stage to initialize the next stage. The authors' method illustrates the advantages of a multiobjective framework to reveal deeper insights into design trade-offs, avoid biased outcomes and support robust lens selection. Moreover, performance comparison indicates that the authors method achieves higher-quality solutions within a comparable runtime to the standard NSGA-II approach.

We study three quadratic models in model-based derivative-free optimization: the minimum norm (MN), minimum Frobenius norm (MFN), and quadratic generalized simplex derivative (QS) models. Despite their widespread use, their approximation accuracy and relationships have not been systematically explored. We establish fully linear error bounds for all three models, removing the uniformly bounded model Hessian assumption required in existing MN analyses and deriving the first such results for the QS model. We further analyze Hessian approximation accuracy via directional error bounds, showing that all three models achieve fully quadratic accuracy along sample directions under a mild condition on the sample set. This reveals a form of directional fully quadratic accuracy not captured by existing theory. Finally, we characterize the relationships among these models, identifying conditions under which they coincide and clarifying their structural connections.

Section 1.4 enumerated some common features of target applications of BBO. A recurring difficulty in such applications originates from the fact that the evaluation of the objective and constraint functions is computationally expensive. In some situations, a surrogate optimization problem is available. Surrogate That is, a problem that is considerably cheaper to evaluate and provides some insight on the true optimization problem. This chapter discusses ways of exploiting a surrogate problem as a copilot for an optimization algorithm.

In 1965, John Nelder and Roger Mead published a short (6 pages) note on a new method for unconstrained minimisation. The resulting algorithm has become one of the most widely applied and researched optimization algorithms in the world. Originally, Nelder and Mead titled their method the “Simplex Method”, as the method hinged around using function evaluations at the vertices of a simplex to seek a minimiser. Of course, in the world of optimization, the name “Simplex Method” was already in use – referring to Dantzig’s method algorithm for solving Linear Programs. Thus, in order to avoid confusion, the method became known as the Nelder-Mead (NM) method.

Benchmarking is essential for assessing the effectiveness of optimization algorithms. This paper reviews standard benchmarking methods, including convergence plots, performance profiles, data profiles, and accuracy profiles, which are widely used to evaluate optimization algorithms. The principal contribution of this work is to synthesize ideas and practical suggestions from the literature, and to articulate them within a coherent and unified framework applicable to three specific contexts: constrained optimization, multi-objective optimization, and surrogate-based optimization.

Chapter 3 introduced a first practical DFO algorithm for unconstrained optimization, the coordinate search (CS) algorithm. While it was proven to converge to the first order in some circumstance (see Theorem 3.4), it was also noted that the algorithm can fail on very simple nondifferentiable convex functions (see Example 3.3). The CS algorithm is an example of the subclass of DFO methods called direct search methods. Direct search methods are methods that work from an incumbent solution and examine a collection of trial points. If improvement is found, then the incumbent solution is updated; while if no improvement is found, then a step size parameter is decreased and a new collection of trial points is examined.

A polytope is inscribable if there is a realization where all vertices lie on the sphere. In this paper, we provide a necessary and sufficient condition for a polytope to be inscribable. Based on this condition, we characterize the problem of determining inscribability as a minimum rank optimization problem using slack matrices. We propose a semidefinite programming (SDP) approximation for the minimum rank optimization problem and prove that it is tight for certain classes of polytopes. Given a polytope, we provide three algorithms to determine its inscribability. All the optimization problems and algorithms we propose in this paper depend on the number of vertices and facets but are independent of the dimension of the polytope. Numerical results demonstrate our SDP approximation's efficiency, accuracy, and robustness for determining inscribability of simplicial polytopes of dimensions 4 \leq d \leq 8 with vertices n \leq 10, revealing its potential in high dimensions.

Throughout this book, we have considered the general problem of minimising a multivariate objective function f over a constraint set $$\Omega \subseteq \mathbb{R}^{n}$$ . Let us now be more specific about the nature of the variables and of the constraint set.

This textbook is suitable for self-learning, or for teaching an upper-year university course on derivative-free and blackbox optimization.

There are situations in which the optimization problem is driven by more than one objective function. Typically, these objectives are conflicting; for example, one may wish to maximise the solidity of a structure, while minimising its weight. In such a situation, one desires to take into account the relative tradeoffs between pairs of solutions.

While the line-search-based method of Chapter 10 is quick and easy to understand, it does not exploit the full power of a model function. In essence, the model-based descent algorithm for unconstrained optimization only uses gradient approximations to confirm descent directions. This can be seen in the convergence analysis of Chapter 10 , where only controllably accurate gradient approximations are used. In particular, convergence does not even require that the function values f(x k ) be estimated by a model.

Chapter 3 proposed the coordinate search (CS) algorithm for unconstrained optimization. The algorithm worked based on local exploration around the incumbent solution in the positive and negative orthogonal coordinate directions. In Chapter 7 , this algorithm was expanded to allow a broader use of search directions, resulting in the generalised pattern search (GPS) algorithm. However, the convergence analysis of the GPS algorithm falls a little short of what we would like. First, the main result states that if the objective function f is locally Lipschitz, then the generalised directional derivatives are nonnegative for only a finite set of directions. Second, and more importantly, the analysis only examined unconstrained optimization problems. In practice, there are very few problems in which the variables are free to take any values.