Joint chance-constrained Markov decision processes

We consider a finite state-action uncertain constrained Markov decision process under discounted and average cost criteria. The running costs are defined by random variables and the transition probabilities are known. The uncertainties present in the objective function and the constraints are modelled using chance constraints. We assume that the random cost vectors follow multivariate elliptically symmetric distributions and dependence among the random constraints is driven by a Gumbel–Hougaard copula. We propose two second order cone programming problems whose optimal values give lower and upper bounds of the optimal value of the uncertain constrained Markov decision process. As an application, we study a stochastic version of a service and admission control problem in a queueing system. The proposed approximation methods are illustrated on randomly generated instances of queueing control problem as well as on well known class of Markov decision problems known as Garnets.