Efficient Sampling Procedure For Selecting The Largest Stationary Probability Of A Markov Chain

This study considers the problem of selecting the alternative with the largest stationary probability of a Markov chain. Specifically, we assume that a Markov chain is constructed in accordance with Google's PageRank. The stationary probabilities are determined by the transition probabilities of the Markov chain, and the transition probabilities are unknown but can be estimated by sampling (or collecting data through real-time web page monitoring). Sensitivity analysis is conducted to capture the marginal influence of transition probability estimation errors on the estimation of stationary probabilities. A dynamic sample allocation procedure is proposed, which uses not only posterior means and variances of the estimated transition probabilities but also scaling factors based on the sensitivity analysis. Numerical experiment results demonstrate that the proposed procedure is significantly more efficient than all compared methods.