In TRECVID 2020 evaluation we participate in two tasks: instance search (INS) and activities in extended video (ActEV). In this paper, we describe the BUPT - MCPRL systems and evaluation results for TRECVID 2020[1] in INS and ActEV tasks. Activities in Extended Video (ActEV): Our proposed system framework for ActEV 2020 consists of three stages: spatial activity location, activity tracking, temporal activity detection and classification. p_baseline_4 : the baseline of our method. In this submission, we use 3D - RetinaNet and 3D - Cascade - RCNN with Focal Loss to generate activity tubes, then we use tracking algorithm to link the tubes and build efficient temporal localization systems to detect activities (activity instances). p_baseline_12: replace TPN with r(2+1)d and adjust the threshold.
In this report, the technical details of the VCL-team submission to the Disaster Scene Description and Indexing (DSDI) challenge, are presented. Although the modern Deep Learning (DL) concept detection schemes can be trained to detect, classify and index a huge variety of concepts under condition of high diversity, they rely solely on global-level information. However, the core part of LADI, i.e. the low altitude, oblique perspective of the imagery and disaster-related features, is quite unique in computer vision literature, therefore it raises challenges that need to be adequately addressed in the design of NN architecture. In order to evaluate the effect that objects have in scene understanding, a set of experiments/runs was conducted using different schemes for incorporating local-level information (e.g. objects, entities).
In this paper we present an overview of our participation in TRECVID 2020 Video to Text Description Challenge [1]. Specifically, we participated in the Description Generation subtask by extending of our recent paper [21]. We address the limitation of previous video captioning methods that have a strong dependency on the effectiveness of semantic representations learned from visual models, but often produce syntactically incorrect sentences which harms their performance on standard datasets. We consider syntactic representation learning as an essential component of video captioning. We construct a visual-syntactic embedding by mapping into a common vector space a visual representation, that depends only on the video, with a syntactic representation that depends only on Part-of-Speech (POS) tagging structures of the video description. We integrate this joint representation into an encoder-decoder architecture that we call Visual-Semantic-Syntactic Aligned Network (SemSynAN), which guides the decoder (text generation stage) by aligning temporal compositions of visual, semantic, and syntactic representations. Considering different datasets for training the model, such as VATEX and TGIF, our results represent third place by teams on the TRECVID 2020 Challenge for METEOR and CIDEr-D metrics. We also show that paying more attention to syntax improves the quality of generated descriptions. 1. Video to Text: Description Generation In this year’s TRECVID Challenge [1] we extended our work [20, 21] by training and validating the model in other datasets: VATEX, TGIF, and VTT20. We developed our method under the assumption that integrating semantic-concept representations with syntactic representations can improve the quality of generated sentences. Now, we briefly present the model and we refer the reader to Perez-Martin et al. [21] for details. For tasks like video retrieval from descriptions and video descriptions retrieval from videos [5, 6, 8, 15, 16], the joint visualsemantic embeddings have a successful application. These embeddings are constructed by combining two models: a language model that maps the captions to a language representation vector, and a visual model that obtains a visual representation vector from visual features. Both models are trained for projecting those representations into a joint space, minimizing a distance function. Dong et al. [6] obtain high-performance in retrieval tasks by using the same multi-level architecture for both models, and training with the triplet-ranking-loss function [7]. For video description generation, these embeddings have not been widely explored [10, 14, 18]. In LSTM-E [18], a joint embedding component is utilized to bridge the gap between visual content and sentence semantics. This embedding is trained by minimizing the relevance loss and coherence loss simultaneously. In SibNet [14], autoencoder for visual information, and a visual-semantic embedding for semantic information are exploited. These joint embeddings only consider the implicit contextual information of word vectors. To improve the perplexity and syntax correctness of generated sentences, we learn a new representation of videos with suitable syntactic information. We propose a model to create visual-syntactic embeddings by exploiting the Part-of-Speech (POS) templates of video descriptions. We do this by learning two functions: (·) that maps videos, and !(·) that maps (POS tags of) captions, ⇤This work was funded by ANID Millennium Science Initiative Program Code ICN17 002 and ANID/Doctorado Nacional/2018-21180648. visual-syntactic embedding concepts detector Visual Features Compositional Decoder CD NNS VBP IN IN DT NN two women get out of a taxi POS tags Figure 1. Video description generation with visual-syntactic embedding. The method computes high-level semantic and syntactic representations from the visual representation of the video. Next, the decoder generates a sentence from them. both into a common vector space. The learning process is based on a match and rank strategy, and ensures that videos and their corresponding captions are mapped close together in the common space. Then, when producing features for the decoder architecture (see the next point), we can use the function (·) to map the input video and generate our desired visual-syntactic embedding. To the best of our knowledge, this is the first approach to jointly learn embeddings from videos and (POS tags of) descriptions. Moreover, our proposal constitutes the first instance of an effective use of a ranking model to obtain syntactic representations of videos.
This paper presents the framework and results from the team “Florida International University-University of Miami (FIU-UM)” in the TRECVID 2020 Disaster Scene Description and Indexing (DSDI) task. We submitted four runs, each applying the same framework but using different score aggregation methods to rank each video shot. The score aggregation methods used in these runs are summarized as follows. • run1: the sum of the feature scores obtained from the video frames to rank the shots; • run2: the average of the feature scores obtained from the video frames to rank the shots; • run3: the maximum of the feature scores obtained from the video frames to rank the shots; • run4: the average of the top three features’ scores obtained from the video frames to rank the shots. Our framework includes the following processing steps: (1) pre-processing imagery from the provided LADI (Low Altitude Disaster Imagery) dataset; (2) generating soft labels for imagery in the LADI dataset through the fusion of annotations from both human and machine annotators as well as image time/location-based concept lookups of open datasets; (3) categorizing the frames in the LADI imagery by five Convolutional Neural Network (CNN) models (i.e., damage, environment, infrastructure, vehicles, and water), each focused on a subset of the 32 features; and (4) aggregating the predictive scores of the frame-level to the shot-level through sum, avg, max, and top. To improve the performance of the CNN models, we adopt various training strategies, including (1) the model pre-trained on ImageNet; (2) propagating the labels during training, following the sequence nature of the LADI dataset; and (3) retrieving more relevant data using an image crawler to enhance the training data. This year, the FIU-UM team achieved the first place among all the submitted runs, regardless of the training type. Among a total of four prioritized submitted runs with different relevancy sorting techniques, three of our runs ranked the top 3. The submission details are listed as follows. • Training type: LADI + Others (O) • Team ID: FIU-UM (Florida International University University of Miami) • Year: 2020
This paper presents the approach of the KsLab NUT team in the TRECVID 2020[1] VTT Task. We propose a method that focuses on reducing the processing time. By extracting only important frames from videos and using them for processing, we were able to drastically reduce the number of frames to be processed while achieving certain levels of accuracy. Furthermore, we also applied the methods used for text summarization to examine their performance.
In this paper we summarize our TRECVID 2020 [2] video retrieval experiments. We participated in the Ad-hoc Video Search (AVS) task with a fully deep learning based solution. Our solution is based on a newly developed model, which we term Sentence Encoder Assembly (SEA) [11]. The novelty of the SEA model is two-fold. First, di↵erent from the prior art that uses only a single common space, SEA supports text-video matching in multiple encoder-specific common spaces. Such a property prevents the matching from being dominated by a specific encoder that produces an encoding vector much longer than other encoders. Second, in order to explore complementarities among the individual common spaces, we propose to train SEA by multi-space multiloss learning. We exploit MSR-VTT and TGIF as training data. For video representation, we use pre-trained ResNet152 and ResNeXt-101 to extract frame-level features, and C3D to extract segment-level features. Video-level features are obtained by mean pooling. Using SEA alone obtains a mean infAP of 0.236 for the 2020 task. Having SEA pretrained on the Google’s Conceptual Captions dataset is helpful, obtaining a higher infAP of 0.251. We again find late average fusion of distinct models (consisting of SEA and W2VV++ trained in varied settings) beneficial, obtaining the best infAP of 0.269 among our four submissions, and ranked at the second place teamwise.
In this paper, we describe our approach for TRECVID 2020 DSDI task. This task requires to propose a system to output a ranked list of the top-k video clips that include the given disaster features. We treat this task as a multi-label multi-class classification problem by assigning predicted disaster features for the given frames and aggregating the frame-level labels to the video clip level labels for listing the ranked clip level result. LADI is used as our training dataset, that is a large scale disaster aerial image dataset with long-tail distribution, high-resolution and multiple noisy labels per image. To utilize LADI with consideration of handling the above characteristics, we propose a solution consisted of three parts: (1) Label encoding for smoothing the multiple annotations to reduce the noisy label propagation. (2) Incorporating a cost function based on Focal Loss for tackling the imbalanced data distribution. (3) Leveraging recently proposed efficient network architectures for dealing with high-resolution images as input. Furthermore, we combine these techniques with team NII-ICT AutoML solution, and also report the fusion results of ours with team NIIICT and NII UIT, that reached a top mAP with 0.383 under the evaluation setting as training data with LADI-only track.
We propose an action recognition system for surveillance scenarios, which wins TRECVID 2020 [1] Activities in Extended Video (ActEV 2 ) Challenge with a large advantage of 23.8% ahead the runner up system. Our system develops a dense spatial-temporal proposal generation model which collaborates with the state-of-the-art action classifiers. The proposed system utilizes multiple state-of-the-art modules and is trained on VIRAT Dataset with only released annotations. In this paper, we demonstrate the architecture and algorithms with technique details of the winner system.
This year, the PicSOM team participated only in the Video to Text Description (VTT), Description Generation subtask. In total, the PicSOM team submitted four runs. We had two goals in our submissions, first, to study the performance of our recent developments in the architectures of the captioning model, and second, to see the effect of using the VATEX dataset in model training. The submitted four runs are as follows: • PICSOM.1.PRIMARY: Our latest and best stacked attention model, trained with three datasets. • PICSOM.2: Model architecture similar to our best VTT 2019 submission, trained with three datasets. • PICSOM.3: Another well-performing stacked attention model, trained with two datasets. • PICSOM.4: Model architecture similar to our best VTT 2019 submission, trained with two datasets. The runs aim at comparing different implementations of stacked attention on the visual features and the benefit from using the VATEX dataset. Based on our results we can conclude that the use of the VATEX dataset had more effect on the improvement of the results than the stacked attention, which also produced small but noticeable improvement. Based on the results of the runs, it seems that our latest attention model combined with self-critical reinforcement learning was the best approach.
. The Waseda Meisei SoftBank team participated in the TRECVID 2020 Ad-hoc Video Search (AVS) task [1]. As with last year’s AVS task, we submitted both manually assisted and fully automatic runs this year. Our approach consisted of concept-based video retrieval for manually assisted runs and visual-semantic embedding for fully automatic runs. Our best manually assisted run achieved a mean average precision (mAP) of 25.2%, which ranked the highest among all the manually assisted systems. Our fully automatic run achieved an mAP of 20.0%, which ranked fifth among all participants.
Activity detection in surveillance videos is a challenging problem due to multiple factors such as large field of view, presence of multiple activities, varying scales and viewpoints, and its untrimmed nature. The requirement of processing the surveillance videos in real-time makes this more challenging. In this work, we propose a real-time online system to perform activity detection on untrimmed surveillance videos. The proposed system consists of three stages: first we detect tubelets with activities, then classify them, and finally merge them to generate spatio-temporal activity detections. We propose a localization network which takes a video clip as input and makes use of feature pyramid, multi-layer loss, and atrous convolutions to address the issue of multiple scales and detect small activities in terms of tubelets. The online processing of videos at a clip level drastically reduces the computation time in detecting activities. The detected tubelets are assigned activity class scores and merged together using our proposed Tubelet-Merge Action-Split (TMAS) algorithm to form action tubes. The TMAS algorithm efficiently connects the tubelets in an online fashion to generate spatiotemporal detections which are robust against varying length activities. We perform our experiments on the DIVA (Deep Intermodal Video Analytics) dataset and demonstrate the effectiveness of the proposed approach in terms of speed (∼100 fps) and performance with state-of-the-art results. The code and models will be made publicly available.
This paper describes a fan-driven and character-centered approach proposed by the MeMAD team for the 2020 TRECVID [Awad et al. 2020] Video Summarization Task. In terms of data, besides the provided videos, scripts and master shot boundaries, our approach relies on fan-made content, more pre-cisely on the BBC EastEnders episode synopses from its Fandom Wiki 1 . We also use BBC EastEnders characters’ images crawled from a search engine to train a face recognition system. All our runs use the same method, but with varying constraints of the number of shots and the maximum duration. The shots included in the summaries are the ones whose transcripts and visual content have the highest similarity with sentences from the synopsis. The runs submitted are as follows: Surprisingly, the scores obtained for each run were very similar for the questions answering part. Only for the character Ryan, one question more was answered by choosing 15 shots rather than less. For all our runs, the redundancy score improved with the number of shots included in the summary while the relation with the scores for tempo and contextuality seem more varying. The scores were lower for the question answering evaluation part. This is rather unsurprising to us as we realized while deciding on a similarity measure score that it was rather challenging for humans too to choose between two potentially interesting moments without knowing beforehand the questions included in the evaluation set. Overall, we con-sider that the results obtained speak in favour of using fan-made content as a starting point for such a task. As we did not try to optimize for tempo and contextuality, we there is some margin for improvement here, however the task of answering unknown questions remains challenging.
In the TRECVID 2020 Instance Search task, the participants are required to retrieve specific persons doing specific actions from 468,533 shots. The retrieval target is divided into two parts: person retrieval and action retrieval. As for person retrieval, a face detection and recognition model pretrained on wilder face dataset is adopted to compute person retrieval scores. As for action retrieval, we utilize a common action recognition model(TSM) pre-trained on kinetics dataset and a humanand-object-interaction model(PPDM) pre-trained on HICO-DET dataset to compute action retrieval scores. We obtain the final results by fusing person scores and action scores. The approaches of person retrieval in two runs we submitted are the same, however there is something different in action retrieval: • F_M_E_A_WHU_NERCMS.20_2: We adopt PPDM for some specific actions and C3D that we used last year for the others. • F_M_E_A_WHU_NERCMS.20_1: We replace C3D with TSM. The MAP increases by a percentage of 5.2 after C3D is replaced, which means that TSM works much better than C3D in action retrieval. Our best result achieved 0.151 mAP in automatic task according to the evaluation.
