Establishing Confidence in Average Metrics

This paper argues for the Bias-Corrected and Accelerated (BCa) bootstrapping method to add statistical rigor by computing confidence intervals when reporting average performance evaluation metrics in computer architecture and systems research and development. BCa can be computed for any average metric and for small data sets that follow a skewed, non Gaussian distribution, illustrating its broad applicability. The paper further demonstrates the importance and relevance of confidence intervals in concrete case studies.