Autohighlight: Highlight detection in League of Legends esports broadcasts via crowd-sourced data

Every minute, 500 h of footage is uploaded to Youtube.com, and ∼1900 h of footage is livestreamed on Twitch.tv. It can therefore be challenging for viewers to find the content they are most likely to enjoy. Highlight videos can entertain users who did not watch a broadcast, e.g. due to a lack of awareness, availability, or willingness. Furthermore, livestream content creators can grow their audiences by using highlights as advertisement, while also engaging casual followers who do not watch full broadcasts. However, hand-generating these videos is laborious, thus automatic highlight detection is an active research challenge. We examine automatic highlight detection by focusing on esports broadcasts. Esports are an emerging genre of sport played using a video games. We focus on League of Legends, a popular title with multiple professional leagues. Esports broadcasts are high-quality and professionally produced, mirroring traditional sports. We tackle the problem in a weakly supervised manner, utilising two datasets, one of ‘crowd-sourced’ highlight videos and one of unedited broadcasts. These datasets allow us to leverage massive data while hugely reducing the human cost of data curation and annotation. We propose two novel extensions to state-of-the-art rank-based highlight detection architectures. Firstly, a multimodal hybrid–fusion architecture that enables audio-visual highlight detection, and secondly, a smoothing step to incorporate context into decision making. Both extensions show significant improvement over state-of-the-art ranking models, in places performing nearly twice as well as competing architectures. Additionally, we examine the effectiveness of each modality and compare ranking models with classification based systems.