Ranking by Dependence - A Fair Criteria

Estimating the dependences between random variables, and ranking them accordingly, is a prevalent problem in machine learning. Pur- suing frequentist and information-theoretic approaches, we first show that the p-value and the mutual information can fail even in simplistic situations. We then propose two conditions for regularizing an estimator of de- pendence, which leads to a simple yet eec- tive new measure. We discuss its advantages and compare it to well-established model- selection criteria. Apart from that, we derive a simple constraint for regularizing parame- ter estimates in a graphical model. This re- sults in an analytical approximation for the optimal value of the equivalent sample size, which agrees very well with the more involved Bayesian approach in our experiments.