Multi-CDN strategies have become increasingly important in enhancing Quality of Experience (QoE) for adaptive video streaming. The recent development of the content steering standard (ETSI TS 103 998) aims to facilitate real-time decision-making about the best-performing CDN by gathering statistics from both players and CDNs. However, this task presents significant challenges. Existing solutions for CDN selection rely on heuristics, which often fail to adapt to diverse network conditions and suffer from issues such as prolonged CDN switching delays and/or complex implementation requirements, resulting in poor QoE. To address these limitations, we present StreamWise-a learning-based solution for real-time CDN selection that works effectively for both on-demand and live adaptive video streaming. StreamWise implements on the content steering standard, leveraging a deep reinforcement learning (DRL) framework to learn and predict in real-time the optimal CDN selection policy by continuously interacting with the environment. Our solution adapts in real-time to network conditions, content characteristics, and viewer requirements ensuring the delivery of high-quality content to users. We evaluate StreamWise through extensive trace-driven emulation-based experiments, demonstrating its superior performance compared to conventional CDN selection strategies. Our results demonstrate a significant improvement in VMAF by similar to 8.5% with similar to 1.5x higher average bitrate and QoE improvement of similar to 48% with minimal rebuffering events for video-on-demand streaming and an improvement in VMAF by similar to 7%, with similar to 1.2x higher average bitrate and a QoE improvement of similar to 22% while substantially reducing the rebuffering events by similar to 10x for live streaming.
We study the patch selection problem for efficient transformer-based visual understanding wherein a sampler can be used to drop less informative patches at inference in order to speed up model execution on resource-constrained devices. As no labels are available on the saliency of the patches, existing works either try to solve an auxiliary task of locating distinctive patches or learn a policy network through the global image/video-level supervision. The former approach could drop redundant but important patches while the latter suffers from the weak supervision of a single class label per image/video. In this work, we observe that the attention weights in trained transformer-based models clearly highlight the salient regions in images and videos. Therefore, we propose a learned patch sampling framework called SAMPL that utilizes the attention weights as fine-grained patch-level supervision to learn a lightweight policy network for patch selection. To train SAMPL end-to-end with the transformer-based models, we introduce a new loss function based on the REINFORCE algorithm to match the distribution of patch selection probabilities and the attention scores. Experimental results on ImageNet, UCF101, Something-Something v2 and Kinetics-400 show that SAMPL can effectively increase the throughput by at least 1.5x while achieving competitive classification accuracy.
3D Gaussian Splatting (3DGS) is an emerging approach for training and representing real-world 3D scenes. Due to its photorealistic novel view synthesis and fast rendering speed (e.g., over 100 FPS), it has the potential to transform how scenes that can be explored in 6 degrees-of-freedom (6-DoF) are represented. However, a limiting factor of 3DGS is its large size, which requires high network bandwidth for streaming reconstructed real-world 3D scenes. In this paper, we propose SGSS for optimizing the streaming transmission of 3DGS scenes during 6-DoF navigation. Since not all Gaussians in the full scene are needed for rendering a user's view, SGSS uses view-adaptive streaming, enabled by optimized spatial partitioning of the scene, for achieving network transmission savings. For each spatial partition, SGSS uses an importance-based Gaussians pre-sorting scheme to enhance the initial view quality and reduce the user-perceived scene loading time. We further design a client-side view-adaptive streaming algorithm that features lightweight visibility checking, prioritized streaming, incremental processing, and stream pausing/resuming schemes. We implement SGSS with JavaScript and WebGL2. Evaluation results show that the quality of views rendered with SGSS streaming is consistently higher than or on par with state-of-the-art approaches. Furthermore, the view-adaptive streaming of SGSS can result in high savings in network transmission without impacting the view quality.
Individuals who experience a significant loss of vision or blindness often face challenges navigating the world. In such scenarios, there is often a dependence on various types of support or aids to enable individuals to move through their surroundings. Examples of this include guide dogs and assistive white canes. Those who suffer from blindness also rely on a heightened sense of auditory processing. Despite these solutions, navigating the modern world is challenging from a safety perspective. In recent years, computer vision algorithms have been developed to interpret and report the contents of image streams. Applications of these algorithms have often focused on automated labeling of data (image analysis and audio processing etc.) within vast media databases; however, they can be leveraged to operate with live camera streams. In this research, the authors present an overview of a proof-of-concept technology stack to implement a danger detection system for individuals with significant visual impairment. The system integrates an advanced cloud driven approach employing a fine-tuned YOLO model for hazard detection, performs obstacle avoidance, segmentation processes for scene understanding, text recognition through optical character recognition (OCR), and visual question answering (VQA) for user queries. These technologies collectively address critical issues such as dynamic hazard recognition, spatial awareness, and personalized assistance.
The climate crisis has highlighted the environmental impact of information and communication technologies (ICT), underscoring the need for sustainable solutions to reduce carbon emissions from ICT. As video streaming continues to dominate Internet traffic, research in this field has become increasingly important. The energy consumption of OLED TVs relies on the video brightness. Thus, an approach to optimized the energy consumption is reducing the video brightness. In this work, we provide an open dataset for energy consumption in OLED TVs while playing videos with various brightness levels. The dataset comprises the energy data of four OLED TVs with different screen sizes and manufacturers in playing 176 videos in a range of dark and bright content. As results, 704 data traces of energy consumption are collected. We also conduct a subjective test to assess the quality of videos displayed in OLED TVs when they are reduced the brightness. Ten levels of brightness reduction are investigated four different video bitrates and video characteristics. The rating scores are attached to our dataset so that they can be used to develop QoE prediction models in the context of the tradeoff between energy consumption and video brightness. We collected opinion scores from 28 participants, resulting in 2240 responses in total. The dataset is available at https://github.com/minhkstn/OLED-EQ.
Extended reality (XR) devices must render high-quality 3D graphics at low latency to deliver truly immersive experiences. However, XR devices are severely power- and resource-constrained, limiting the quality of on-device (local) rendering. Offloading rendering to a powerful remote machine can enhance graphics quality, but network latency can degrade the overall experience. To mask latency, XR systems reproject the rendered frame to compensate for user motion since the rendered pose. Traditional reprojection, known as TimeWarp, uses a lightweight mechanism to compensate for latency in rotational motion, but not translational motion. Compensating for translational motion is more expensive, but is increasingly important at higher latencies. We present XRgo, the first open-source XR runtime that supports application-transparent offloading of rendering to a remote server with client-side six-degrees-of-freedom (6-DoF) reprojection. XRgo uses OpenWarp, the first open-source and OpenXR-compatible lightweight 6-DoF mesh-based reprojection technique. Compared to on-device rendering, XRgo consistently renders at the target frame rate while consuming less power. User studies show that on average, OpenWarp provides a superior XR experience than TimeWarp. Further, OpenWarp only consumes < 2% more power than TimeWarp. Our findings reveal that the quality of experience depends on both network average latency and variability, highlighting the need for end-to-end evaluations under live network conditions.
Power consumption, and the resulting limitation to computational load, is a first-order constraint in designing comfortable all-day-wear extended reality (XR) devices that can provide rich immersive experiences. This paper concerns reducing XR device power consumption by offloading head tracking, one of the top CPU and power consumers, to a remote server. We present RemoteVIO, the first open-source end-to-end XR system that offloads head tracking (visual inertial odometry or VIO) to a remote server. Our work distinguishes itself from past studies on computation offloading in XR by properly addressing two under-explored but critical aspects: 1) a comprehensive evaluation of user experience in a complete end-to-end XR system and 2) a quantification of the net power savings on real hardware. Through an Institutional Review Board (IRB) approved study, we find that RemoteVIO provides a satisfactory user experience under typical network conditions, but often degrades for network round trip time above 200 milliseconds (ms). We also demonstrate the first measured power savings from offloading head tracking on real hardware: compared with on-device tracking, RemoteVIO reduces CPU power by up to 52%, CPU+network power by up to 39%, and end-to-end full system power by up to 20%. Of equal importance, we examine the traditional approach of evaluating XR offloading techniques with datasets and quantitative metrics. Our results reveal that traditional head tracking metrics do not correlate with user experience, questioning the use of such metrics in XR systems research and underscoring the importance of using end-to-end systems that allow for user experience studies.
Precise mapping of ice hockey rinks is critical for applications such as player tracking, game strategy analysis, and broadcast enhancements. Traditional methods often rely on manual annotations or simplistic models that fail to account for the rink's complex geometry and dynamic in-game conditions. To address these limitations, we present HockeyRink, a novel dataset comprising 56 meticulously annotated keypoints corresponding to significant landmarks on a standard hockey rink, including face-off dots, goalposts, and blue lines. Leveraging the YOLOv8-Large pose estimation architecture, we adapted the model to treat the rink as a single 'pose' object, enabling accurate keypoint predictions tailored to the nuances of hockey rink imagery. Our dataset, derived from diverse hockey game footage from the Swedish Hockey League (SHL), facilitates applications in homography estimation, 2D/3D scene mapping, and tactical overlays. By detecting and mapping rink keypoints, one can compute precise transformations between the image plane and the rink's physical dimensions, enabling the overlay of player trajectories and tactical insights onto broadcast footage. This work addresses the unique challenges of ice hockey, such as occlusions, rapid camera movements, and varying lighting conditions, and offers a robust foundation for future research and innovation in sports analytics. The HockeyRink dataset and trained model are openly available at: https://huggingface.co/SimulaMet-HOST/HockeyRink.
We present a novel DASH-based streaming system for dynamic 3D Gaussian Splatting (3DGS) scenes, addressing the challenges of streaming large amounts of 3DGS data over diverse and dynamic networks. Our Layer, Tile, and Segment Adaptive streaming (LTS) system combines three key features: (i) multi-layer streaming, which adapts to diverse client capabilities while balancing visual quality and bandwidth usage, (ii) tiled streaming, which reduces unnecessary data transmission by focusing on the user's viewport, and (iii) segment streaming, which divides dynamic 3DGS scenes into segments, letting clients request them dynamically to handle network fluctuations. Our experimental results demonstrate that our LTS system achieves superior performance in both live and on-demand streaming of dynamic 3DGS scenes compared to the baselines. For example, in live streaming, LTS could achieve up to 99.70% reduction in missing frames on average and deliver a maximum PSNR (Peak Signal-to-Noise Ratio) improvement of 10.08 dB. In on-demand streaming, LTS could reduce the freeze time by up to 92.01%, and increase the synthesized view quality by up to 5.14 dB in PSNR and 0.11 in SSIM (Structural Similarity Index). Our source codes are available at: https://github.com/AIINS- NTHU/LTS-DASH-Streaming-System-for-3DGS.
The fast paced nature of ice hockey presents unique challenges for object detection, particularly in tracking the puck-a small, fast moving object that is critical to gameplay analysis. This paper introduces HockeyAI, a novel open source dataset specifically designed for multi-class object detection in ice hockey. The dataset includes 2,101 high resolution frames extracted from professional games in the Swedish Hockey League (SHL), annotated in the You Look Only Once (YOLO) format. Annotations span 7 classes, covering dynamic objects such as the players and the puck, as well as static rink elements such as goalposts and face-off circles. The dataset is derived from diverse SHL games across multiple seasons and teams, ensuring a rich variety of scenarios reflective of real world gameplay. A fine tuned YOLOv8 medium model is also provided, demonstrating high performance across all classes. Key comparisons highlight the dataset's advancements over existing resources, addressing their limitations such as incomplete class coverage, low resolution, and in-consistent annotations. The dataset, model, and an interactive demo are publicly available on Hugging Face under an open source license, fostering further collaboration in sports related computer vision applications (https://huggingface.co/SimulaMet-HOST/HockeyAI).
Point cloud video datasets enriched with user behavior traces are critical for advancing tile-based HTTP adaptive streaming (HAS) systems, particularly those reliant on large training data for player modules like Field of View (FoV) prediction. This paper presents MazeLab, a dynamic volumetric video dataset comprising a feature-rich point cloud representation of a large maze environment. The dataset captures navigation traces from 15 participants interacting with 15 distinct maze variants, categorized into seven classes designed to elicit specific behavioral characteristics such as navigation patterns, attention hotspots, and interaction dynamics. By incorporating diverse environmental designs, each targeting unique user responses, MazeLab provides a comprehensive behavioral repository, enabling robust testing and optimization of HAS-driven volumetric video systems in complex scenarios. The dataset is publicly available here.
Field of view (FoV) prediction is crucial for optimizing 6-DoF dynamic point cloud-based volumetric video (PCV) streaming. By accurately predicting which tiles fall within the viewer's region of interest, FoV prediction enables adaptive bitrate (ABR) algorithms to allocate higher bitrates to likely viewed tiles while assigning lower bitrates to less critical areas. This improves bandwidth efficiency and enhances the quality of experience (QoE) by aligning bitrate allocation with the viewer's focus. However, current 6-DoF salience-aware FoV prediction models face challenges related to high latency, computational costs, and a lack of complex datasets with detailed FoV traces, hindering the development of more effective real-time predictors. To address these challenges, we propose the LL-Sparse family, a suite of three solutions for direct tile salience score prediction: LL-Adapter, an extension of HMD-trajectory-based (HTB) models, such as GRUs, tailored for tile scoring; LL-PointNet, which integrates a GRU with PointNet to enhance salience-aware prediction; and LL-SparseConv, a scalable variant of LL-PointNet that employs sparse convolution in place of PointNet, serving as a proof of concept. These models strike a balance between practical performance and theoretical advancements in tile salience prediction. Furthermore, we introduce the MazeLab dataset, a novel, large-scale dynamic point cloud dataset that mimics real-world PCV scenarios to effectively benchmark FoV prediction models. Experimental results highlight the LL-Sparse family's exceptional scalability, reduced latency, and enhanced accuracy, establishing it as a promising solution for efficient real-time volumetric media applications.
Video-based Point Cloud Compression (V-PCC) standard offers state-of-the-art tools and efficiency for volumetric video compression. The V-PCC reference software, TMC2, is able to demonstrate the compression efficiency of V-PCC, but with computational complexity that is impractical for real-life applications. This paper introduces a novel open-source V-PCC encoder, named uvgVPCCenc, that brings V-PCC compression speed to a practical level. To reduce computational bottlenecks, uvgVPCCenc is implemented from the ground up in C++ with advanced multi-threading and streamlined algorithms. It also supports a flexible parameterization and different presets that facilitate its adaptation to various content and target bitrates. Our comparative evaluations show that uvgVPCCenc achieves substantial speedups over TMC2 while maintaining competitive rate-distortion-complexity tradeoff. To the best of our knowledge, uvgVPCCenc is the first open-source encoder explicitly designed for practical V-PCC compression, paving the way from theoretical research to practical volumetric video applications.
The Audiovisual Multimodal Interaction Suite (AMIS) is an open-source dataset and accompanying Unity-based demo implementation designed to aid research on immersive media communication and social XR environments. AMIS features synchronized audiovisual recordings of three actors performing monologues and participating in dyadic conversations across four modalities: talking-head videos, full-body videos, volumetric avatars, and personalized animated avatars. These recordings can be used to simulate scenarios such as traditional video conferences or XR meetings with 3D avatars in controlled and replicable environments. The limitations of existing datasets, which include a restricted number of audiovisual formats, a narrow application focus, and suboptimal inclusion of verbal and non-verbal cues, are addressed by AMIS. With AMIS Studio, a Unity-based demonstrator, researchers can explore the recordings and compare the different audiovisual formats in VR scenes. This paper outlines the creation of AMIS, its design considerations, and how it may be applied in interdisciplinary domains, including cognitive psychology, audiovisual quality assessment, and social XR research.
Learned point cloud compression methods have achieved rate-distortion performance, which is comparable to or higher than conventional approaches. However, this often comes at the cost of high hardware requirements and thus low throughput during encoding and decoding. In this paper, we present an adaptive bitrate point cloud streaming system utilizing learned compression. While other learned compression techniques require to split geometry and attributes, resulting in high encoding latency, we deploy a unified model to handle both modalities together, which drastically reduces the coding complexity. We explore the capabilities of the learned encoder to derive multiple quality representations with only re-running a fraction of the encoding steps, making it a suitable fit for adaptive bitrate streaming. Furthermore, we ablate the encoding latency of each component in the encoder and decoder stack, identifying bottlenecks in the process.
Real-time communication (RTC) systems, such as video conferencing and cloud gaming, depend on effective congestion control (CC) algorithms to manage diverse network conditions and access technologies like Wi-Fi, LTE/5G, and satellite networks. While tools like AlphaRTC and Pandia have significantly advanced CC algorithm development for WebRTC, there is an absence of a unified framework for systematic benchmarking and cross-platform evaluation. We introduce a practical test suite that offers a unified platform for evaluating algorithms developed with containerized tools such as AlphaRTC and Pandia. Leveraging Mahimahi and Containernet, our test suite supports trace-driven emulations of realistic heterogeneous network environments and incorporates diverse topologies with multiple RTC flows communicating simultaneously, enabling testing under complex and heterogeneous conditions. Beyond evaluation, our test suite promotes community engagement by enabling easy contribution and integration of algorithms. Researchers can add their own algorithms and test scenarios to the suite, allowing them to seamlessly compare their solutions with archived ones under reproducible evaluation conditions.
In recent years, the demand for large, diverse, and well-annotated firearm datasets has grown, particularly for advancing firearm recognition systems. However, existing datasets often lack sufficient variety in viewpoints, configurations, and model versions. To address these challenges, this paper introduces an extented version of the Ceasefire Hierarchical Firearm Dataset (eCHFD), a large image dataset of firearms consisting of over 93,000 images in 505 classes. It was constructed from more than 240 videos filmed at the Toulouse Forensics Laboratory (France) and further enriched with images from the existing CHFD dataset and additional downloaded images. Automated post-processing techniques were employed to efficiently build this large-scale dataset. In addition, a hierarchical annotation scheme validated by forensics experts was used to classify firearms at multiple levels, enabling its utility for fine-grained classification tasks. Experimental results was conducted using models such as HERBS and FGVC-PIM to demonstrate the effectiveness of the dataset.
Even as virtual reality devices and applications become more pervasive, our understanding of user behavior remains in its infancy. Typical studies involve some combination of post hoc summative evaluation (i.e., questionnaires after an experience is finished) and interaction log analysis (i.e., head position and clicks). Such methods do not illuminate the reasons, intentions, and causality underlying the user's actions. Our research aims to bridge this gap in the context of one specific application area: 360 degrees video viewing. The fundamental question we ask is why users pan when they do. We present a study of users (n = 24) viewing 360 degrees video and reporting their intentions each time they move their head to look at something new. By categorizing viewer behavior with detailed annotations, we established a structured framework for understanding viewer engagement. Additionally, we introduced a compiled dataset that merged pan-action data with corresponding user-reported motivations, presenting a valuable asset for subsequent research on VR viewer behavior. Through our approach, we facilitate the comprehension of human interaction within VR contexts, providing a foundational tool for future studies to incorporate viewer intentions with their movements.
Online distance learning emerged as a prominent means of education during the pandemic and is expected to continue as a long-lasting trend, enabling access to top education resources without the constraints of physical distance. However, conferencing software predominantly used for synchronous online teaching and learning is not suitable or sufficient for young students and teachers. The learning experiences of these students heavily rely on hands-on activities and interactions with peers and teachers that are challenging to replicate. Additionally, teachers face difficulties in effectively monitoring student progress. To address these challenges, we propose a Synchronous Accessible Immersive Online Learning System (SAILS). It offers an immersive and interactive platform for young learners by integrating real-life school activities with a virtual learning environment. Using the system, teachers can easily organize classes, assess students' work. We evaluated the effectiveness of SAILS through a user study involving 40 young learners from a kindergarten and several primary schools, along with their teachers. The results demonstrated a strong preference for our system among the participants, highlighting its ability to provide a more immersive and engaging online learning experience.
The recent introduction of the ETSI TS 103 998 content steering standard marks a significant milestone in the evolution of media delivery for content providers. This standard simplifies the process of utilizing multiple Content Delivery Networks (CDNs), a key requirement for managing large-scale client bases. It offers clear guidelines for real-time CDN state switching, which enhances the user experience by dynamically selecting the most suitable CDN based on changing network conditions. In contrast, current industry solutions often resemble load balancing techniques, relying on heuristic approaches that are manually crafted and unable to adapt effectively to varying network environments. These static solutions frequently fail to optimize the user experience in real-time, especially when faced with diverse and unpredictable network conditions. In this paper, we utilize our previous work StreamWise-a DRL-based multi-CDN solution that outperforms existing state-of-the-art methods, offering superior adaptability and efficiency. We integrate this solution into the Dash.js reference player. Through a comprehensive demonstration, we illustrate how StreamWise, or any other multi-CDN solution, can be seamlessly incorporated into Dash.js, providing a benchmark for future developments in multi-CDN switching solutions. Experimental results in the wild demonstrate the performance of various multi-CDN solutions for Dash.js in Video on Demand (VoD) streaming mode. On average, StreamWise provides a similar to 78% improvement in VMAF compared to other solutions, while also ensuring smooth quality transitions., while also ensuring smooth quality transitions.
