Benchmarking Video Service Quality: Quantifying the Viewer Impact of Loss-Related Impairments

We present the first empirical study of the impact of loss-related errors on TV viewing engagement across disparate platforms, delivery technologies and performance measures. Our dataset comprises anonymized video viewing sessions and data about quality of delivery from a content service provider with a nationwide customer base. We study buffering events on streaming apps, mild and severe packet loss errors on multicast-delivered IPTV to a Set-Top-Box (STB) and signal strength errors on Digital Terrestrial TV. Since these metrics cannot be directly compared to each other, we use engagement as our proxy measure. We first characterize the relationship between each impairment and viewing engagement, investigating confounding factors such as type of content, asset length and connection type. We conclude that the loss of engagement due to poor quality delivery is incurred immediately for on-demand content and in the long-term for live content. We rank impairments across platforms by their impact on engagement.