Markov decision processes (MDPs) have found success in many application areas that involve sequential decision making under uncertainty, including the evaluation and design of treatment and screening protocols for medical decision making. However, the data used to parameterize the model can influence what policies are recommended, and multiple competing data sources are common in many application areas, including medicine. In this article, we introduce the Multi-model Markov decision process (MMDP) which generalizes a standard MDP by allowing for multiple models of the rewards and transition probabilities. Solution of the MMDP generates a single policy that maximizes the weighted performance over all models. This approach allows the decision maker to explicitly trade-off conflicting sources of data while generating a policy of the same level of complexity for models that only consider a single source of data. We study the structural properties of this problem and show that it is at least NP-hard. We develop exact methods and fast approximation methods supported by error bounds. Finally, we illustrate the effectiveness and the scalability of our approach using a case study in preventative blood pressure and cholesterol management that accounts for conflicting published cardiovascular risk models.

Outputs from treatment units in an oil refinery are only semi-finished products. In this chapter, we discuss procedures for blending these products into finished products like gasoline, diesel oil, etc. meeting various specifications on them for selling in the marketplace.

This research creates an operations engineering and management methodology to optimize a complex operational planning and coordination challenge faced by sites that perform clinical research trials. The time-sensitive and resource-specific treatment sequences for each of the many trial protocols conducted at a site make it very difficult to capture the dynamics of this unusually complex system. Existing approaches for site planning and participant scheduling exhibit both excessively long and highly variable Time to First Available Visit (TFAV) waiting times and high staff overtime costs. We have created a new method, termed CApacity Planning Tool And INformatics (CAPTAIN) that provides decision support to identify the most valuable set of research trials to conduct within available resources and a plan for how to book their participants. Constraints include (i) the staff overtime costs, and/or (ii) the TFAV by trial. To estimate the site's metrics via a Mixed Integer Program, CAPTAIN combines a participant trajectory forecasting with an efficient visit booking reservation plan to allocate the date for the first visit of every participant's treatment sequence. It also plans a daily nursing staff schedule that is optimized together with the booking reservation plan to optimize each nurse's shift assignments in consideration of participants' requirements/needs.

Volunteer convergence refers to the influx of volunteers to affected areas after large‐scale disasters. There are not only many benefits to volunteer convergence, but it also creates significant logistical challenges that can impede relief efforts. This study examines polices for admitting volunteers into organized relief operations, and for assigning admitted volunteers to relief tasks. We represent this problem as a queueing system where, in addition to customer arrivals and departures, random server arrivals and abandonments are also present. Then, using a Markov decision process framework, we analyze server admission and assignment policies that seek to minimize relief tasks holding costs as well as volunteer holding and rejection costs. We show that the classic c μ rule, a server allocation policy that determines where to put servers based on relief tasks holding costs and processing requirements, is optimal under both collaborative and non‐collaborative service regimes and when batch server arrivals are allowed. Additionally, we find that the optimal server admission policy is a complex state‐dependent policy. As a result, we propose a class of admission heuristics that depend on the number of workers in the system and the remaining system workload. In a numerical study, we show that our heuristic policies perform well with respect to long‐run average costs, waiting times, number of volunteers in the system, and number of volunteers idling in the system over a range of parameter values and distributions that are based on real data from a case study. As such, they promise volunteer coordinators an effective and simple way to manage disaster volunteers.

An important decision-making problem in offshore drilling for crude oil is that of allocating various target locations in the field to drill, to available drilling rigs. In this chapter, we discuss an approach for solving this problem optimally.

Asymmetric information, investor optimism, and unbiased prices hypotheses are the main hypotheses proposed for explaining how investors’ difference of opinion may impact stock returns. We use a new measure for divergence in investor beliefs among sell-side analysts to test these three hypotheses. Our initial findings are not supportive of either the asymmetric information or the investor optimism hypotheses. However, since these two hypotheses predict opposing effects of divergence in opinion on stock returns, the effects could neutralize their respective impacts on stock prices. Our further empirical analysis though suggests that this is not the case. The weight of the evidence presented suggests that within the sell-side, the difference of opinion does not impose a bias on future stock returns.

The prevailing first-come-first-served approach to outpatient appointment scheduling ignores differing urgency levels, leading to unnecessarily long waits for urgent patients. In data from a partner healthcare organization, we found in some departments that urgent patients were inadvertently waiting longer for an appointment than non-urgent patients. This paper develops a capacity allocation optimization methodology that reserves appointment slots based on urgency in a complicated, integrated care environment where multiple specialties serve multiple types of patients. This optimization reallocates network capacity to limit access delays (indirect waiting times) for initial and downstream appointments differentiated by urgency. We formulate this problem as a queueing network optimization and approximate it via deterministic linear optimization to simultaneously smooth workloads and guarantee access delay targets. In a case study of our industry partner we demonstrate the ability to (1) reduce urgent patient mean access delay by 27% with only a 7% increase in mean access delay for non-urgent patients, and (2) increase throughput by 31% with the same service levels and overtime.

Markov decision process (MDP) models for the optimal time to initial a medical therapy, such as an organ transplantation, require the estimation of health state transition probabilities from physiological data. Such estimation may be a source of probabilistic ambiguity when, for example, some critical health states are seldom visited historically. For MDP models in general, robust dynamic programming has been proposed as an approach to mitigate the effects of ambiguity on optimal decisions. However, very few realworld studies examining the usefulness of robust MDP policies have been reported. We present a robust MDP model for medical therapy initiation in which worst-case transition probabilities are chosen from a set of probability measures constructed using relative entropy bounds. For this model, we prove that therapy is initiated sooner, in additional states, as the ambiguity increases. We apply the methodology to the problem of deciding when to undergo a living-donor liver transplantation, and present the results of a case study using clinical data. We propose a novel implied confidence level measure that maps the robust solutions to historical transplant decisions, and find that in some cases the robust policies are closer to decisions that have been made in practice.

This paper explores the use of the set of positional auxiliary verbs sit, stand, lie, and move in Biloxi. It offers a detailed analysis of these positionals and compares their use with other Siouan and some non-Siouan languages. While positional stance verbs also double as positional auxiliaries in many languages of the world, in Biloxi positional auxiliaries are grammaticized variants of what were once physical stance verbs, evidenced by the fact that the positional auxiliaries no longer take pronominal marking and sometimes occur side-by-side with their physical stance counterparts. This is a typologically interesting situation in which positional auxiliary verbs form a discrete series distinct from positional stance verbs. The proper use of positional verbs is a complex aspect of the grammar of Biloxi and other Siouan languages. While the choice of positional verbs is often logically motivated based on an object's salient axial extension or shape, this analysis demonstrates that, particularly in more abstract cases, the proper choice of positional auxiliary is not at all obvious. Discrepancies in choice of positional, even among closely related languages and even in the same language, are highlighted. [Keywords: Biloxi, Siouan, positional, auxiliary verb]

Robust dynamic programming robust DP mitigates the effects of ambiguity in transition probabilities on the solutions of Markov decision problems. We consider the computation of robust DP solutions for discrete-stage, infinite-horizon, discounted problems with finite state and action spaces. We present robust modified policy iteration RMPI and demonstrate its convergence. RMPI encompasses both of the previously known algorithms, robust value iteration and robust policy iteration. In addition to proposing exact RMPI, in which the “inner problem” is solved precisely, we propose inexact RMPI, in which the inner problem is solved to within a specified tolerance. We also introduce new stopping criteria based on the span seminorm. Finally, we demonstrate through some numerical studies that RMPI can significantly reduce computation time.

Ambiguous Health State Transition Probabilities – Extended Abstract David L. Kaufman Department of Industrial and Operations Engineering, University of Michigan 1205 Beal Avenue, Ann Arbor, MI 48109, USA davidlk@umich.edu Andrew J. Schaefer Department of Industrial Engineering, University of Pittsburgh 1048 Benedum Hall, Pittsburgh, PA 15261, USA schaefer@ie.pitt.edu Mark S. Roberts Division of General Internal Medicine, School of Medicine, University of Pittsburgh 200 Meyran Avenue, 2nd Floor, Pittsburgh, Pennsylvania 15213, USA robertsm@upmc.edu

Suppose that customers arrive at a service center (call center, web server, etc.) with two stations in accordance with independent Poisson processes. Service times at either station follow the same general distribution, are independent of each other and are independent of the arrival process. The system is charged station-dependent holding costs at each station per customer per unit time. At any point in time, a decision-maker may decide to move, at a cost, some number of jobs in one queue to the other. The goals of this paper are twofold. First, we are interested in providing insights into this decision-making scenario. We do so, in the important case that the service time distribution is highly variable or simply has a heavy tail. Secondly, we propose that the savvy use of Markov decision processes can lead to easily implementable heuristics when features of the service time distribution can be captured by introducing multiple customer classes. To this end, we consider a two-station proxy for the original system, where the service times are assumed to be exponential, but of one of two classes with different rates. We prove structural results for this proxy and show that these results lead to heuristics that perform well.

Machine maintenance is modeled in the setting of a single‐server queue. Machine deterioration corresponds to slower service rates and failure. This leads to higher congestion and an increase in customer holding costs. The decision‐maker decides when to perform maintenance, which may be done pre‐emptively; before catastrophic failures. Similar to classic maintenance control models, the information available to the decision‐maker includes the state of the server. Unlike classic models, the information also includes the number of customers in queue. Considered are both a repair model and a replacement model. In the repair model, with random replacement times, fixed costs are assumed to be constant in the server state. In the replacement model, both constant and variable fixed costs are considered. It is shown in general that the optimal maintenance policies have switching curve structure that is monotone in the server state. However, the switching curve policies for the repair model are not always monotone in the number of customers in the queue. Numerical examples and two heuristics are also presented. © 2007 Wiley Periodicals, Inc. Naval Research Logistics, 2007

We consider a two-station tandem queueing system where customers arrive according to a Poisson process and must receive service at both stations before leaving the system. Neither queue is equipped with dedicated servers. Instead, we consider three scenarios for the fluctuations of workforce level. In the first, a decision-maker can increase and decrease the capacity as is deemed appropriate; the unrestricted case. In the other two cases, workers arrive randomly and can be rejected or allocated to either station. In one case the number of workers can then be reduced (the controlled capacity reduction case). In the other they leave randomly (the uncontrolled capacity reduction case). All servers are capable of working collaboratively on a single job and can work at either station as long as they remain in the system. We show in each scenario that all workers should be allocated to one queue or the other (never split between queues) and that they should serve exhaustively at one of the queues depending on the direction of an inequality. This extends previous studies on flexible systems to the case where the capacity varies over time. We then show in the unrestricted case that the optimal number of workers to have in the system is non-decreasing in the number of customers in either queue.

We develop and numerically illustrate an exact solution of the multivariate, stochastic, differential equations that govern the velocity and position of a charged particle in a plane normal to a uniform, stationary, magnetic field, The equations self-consistently incorporate the Lorentz force into an Ornstein-Uhlenbeck collision model. Properties of the solution in the infinite dissipation limit are explored and the spectral energy density function is found.

A simple expansion method for numerically calculating the energy levels and the corresponding wave functions of a quantum particle in a two-dimensional infinite potential well with arbitrary shape (quantum billiard) is presented. The method permits the study of quantum billiards in an introductory quantum mechanics course. According to the method, wave functions inside the billiard are expressed in terms of an expansion of a complete set of orthonormal functions defined in a surrounding rectangle for which the Dirichlet boundary conditions apply, while approximating the billiard boundary by a potential energy step of a sufficiently large size. Numerical implementations of the method are described and applied to determine the energies and wave functions for quarter-circle, circle, and triangle billiards. Finally, the expansion method is applied to investigate the quantum signatures of chaos in a classically chaotic generic-triangle billiard.