Utilizing vision and language models (VLMs) pre-trained on large-scale image-text pairs is becoming a promising paradigm for open-vocabulary visual recognition. In this work, we extend this paradigm by leveraging motion and audio that naturally exist in video. We present \textbf{MOV}, a simple yet effective method for \textbf{M}ultimodal \textbf{O}pen-\textbf{V}ocabulary video classification. In MOV, we directly use the vision encoder from pre-trained VLMs with minimal modifications to encode video, optical flow and audio spectrogram. We design a cross-modal fusion mechanism to aggregate complimentary multimodal information. Experiments on Kinetics-700 and VGGSound show that introducing flow or audio modality brings large performance gains over the pre-trained VLM and existing methods. Specifically, MOV greatly improves the accuracy on base classes, while generalizes better on novel classes. MOV achieves state-of-the-art results on UCF and HMDB zero-shot video classification benchmarks, significantly outperforming both traditional zero-shot methods and recent methods based on VLMs. Code and models will be released.

Network-on-Chip (NoC), using a packetized communication model presents a scalable interconnect infrastructure for System-on-Chip (SoC) architectures that meets its Performance, Power and Area (PPA) objectives. A typical NoC consists of building blocks such as routers, resizers and Power and Clock Domain Converters (PCDC). Hand crafting a NoC that meets PPA requirements within Time-to-Market (TTM) constraints is difficult if not intractable for real world systems. In this paper, we present an automated NoC synthesis tool that generates PPA optimized, customized NoC for any system from its behavioral specification. The tool provides solution based on multiple isolated communicating trees with fixed points of inter-communication amongst them. It models a variety of requirements like deadlock-avoidance, quality of service etc. as conflicts represented in a Traffic Conflict Graph (TCG) and uses combinatorial optimization techniques to minimize the conflicts resulting in a better overall design. This is in contrast with traditional approaches which focus only on reducing communication overheads. Implementation and evaluation of the tool in production-grade designs shows that it achieves better topologies as compared to hand-crafted NoCs in only a fraction of time. Across several multi-million gate SoCs, the tool has reduced latency, buffer-size and area by 40%, 50% and 8% on average respectively w.r.t hand-crafted NoCs, while meeting user-specified performance requirements. Use of this tool has brought down the typical NoC design time from several months to less than two weeks thereby considerably reducing design effort and TTM targets.

Functionally safe control logic design without full duplication is difficult due to the complexity of random control logic. The Reorder buffer (ROB) is a control logic function commonly used in high performance computing systems. In this study, we focus on a safe ROB design used in an industry quality Network-on-Chip (NoC) Advanced eXtensible Interface (AXI) Network Interface (NI) block. We developed and applied area efficient safe design techniques including partial duplication, Error Detection Code (EDC) and invariance checking with formal proofs and showed that we can achieve a desired safe Diagnostic Coverage (DC) requirement with small area and power overheads and no performance degradation.

Recently neural network accelerators have grown into prominence with significant power and performance efficiency improvements over CPU and GPU. In this paper, we proposed two safety design techniques include Algorithm Based Atomic Error Checking-1 (ABAEC-1) and ABAEC-2 for a Weight Stationary (WS) Convolutional Neural Network (CNN) accelerator focusing on low latency and low overhead error detection and correction with no performance degradation. The proposed design techniques not only detect the errors on-thefly but also perform error diagnosis to localize the errors to a Processing Element (PE) for on-line fault management and recovery. We applied the design techniques on an industry quality CNN accelerator and demonstrated that we could achieve the required Diagnostic Coverage (DC) goal with minimal area and power overhead for selected configurations. Furthermore, we discussed methods to extend the proposed techniques to other dataflow architecture.

Video is one of the richest sources of information available online but extracting deep insights from video content at internet scale is still an open problem, both in terms of depth and breadth of understanding, as well as scale. Over the last few years, the field of video understanding has made great strides due to the availability of large-scale video datasets and core advances in image, audio, and video modeling architectures. However, the state-of-the-art architectures on small scale datasets are frequently impractical to deploy at internet scale, both in terms of the ability to train such deep networks on hundreds of millions of videos, and to deploy them for inference on billions of videos. In this paper, we present a MapReduce-based training framework, which exploits both data parallelism and model parallelism to scale training of complex video models. The proposed framework uses alternating optimization and full-batch fine-tuning, and supports large Mixture-of-Experts classifiers with hundreds of thousands of mixtures, which enables a trade-off between model depth and breadth, and the ability to shift model capacity between shared (generalization) layers and per-class (specialization) layers. We demonstrate that the proposed framework is able to reach state-of-the-art performance on the largest public video datasets, YouTube-8M and Sports-1M, and can scale to 100 times larger datasets.

Functional safety is the top priority for the design of automotive and other mission-critical systems. We proposed Functional Safe NoC (FSNoC) with a new Packet Level Lock Stepping (PLLS) concept for Concurrent Error Detection (CED) of Network-on-Chip (NoC) with high Diagnostic Coverage (DC) and reduced area overhead. Furthermore, we proposed to divide the NoC network of a System-On-Chip(SOC) design into partitions with different performance requirements and apply separate but inter-operable safety mechanisms based on Performance Power Area (PPA) trade-off given the design meet safety requirement. The proposed techniques were used on an industry NoC design to achieve over 99% DC coverage with 11-33% of area, 12-29% power overhead and 5-22% of wiring overhead depending on partition choices.

In this paper, Chinese adjective-noun combinations ([A N]) are shown to be words instead of phrases. Construction Morphology (Booij G, Construction Morphology. Oxford University Press, Oxford, 2010) can describe the various properties of Chinese [A N], but has difficulty accounting for the alternation and competition between [A N] and [A de N]. To solve this problem, I propose an approach that combines Construction Morphology with Realization Optimality Theory (Xu Z, Inflectional morphology in Optimality Theory. Stony Brook University dissertation, Stony Brook, 2007, Lang Ling Compass 5(7):466–484, 2011, The role of morphology in Optimality Theory. In: Hippisley A, Stump GT (eds) The Cambridge handbook of morphology. Cambridge University Press, Cambridge, pp 550–587, 2016; Aronoff M, Xu Z, Morphology 20(2):381–411, 2010; Xu Z, Aronoff M, J Ling 47(3):673–707, 2011a, A Realization Optimality-Theoretic approach to full and partial identity of forms. In: Maiden M, Smith JC, Goldbach M, Hinzelin MO (eds) Morphological autonomy: perspectives from Romance inflectional morphology. Oxford University Press, Oxford, pp 257–286, 2011b).

The process of laying out the various interconnect components and configuring them, is termed as interconnect synthesis. A Network-on-Chip (NoC), has various building blocks such as Routers, Resizers, Power and Clock domain converters (PCDCs), Pipeline elements etc. A software tool is needed to connect these various components (topology) and then configure them (including routing) so that the user constraints are met and the overall logic and wiring cost along with power is kept low. In this paper we present a tool which generates Power, Performance and Area (PPA) optimized NoCs. The input is a behavioral specification which consists of a rough floor-plan, bridge parameters, available clock, power and voltage domains, address spaces, stochastic traffic (including classes and latency criticality), traffic dependency and any partial topology for the locked down portions of the NoC. The output is an optimized NoC, with instantiation and placement of components (routers, Resizers etc.), Virtual Channel (VC) assignments, clock-domain assignments, routing, bridge parameter tuning, FIFO sizes etc. Using this flow, we are able to generate NoCs which are within 15% of the hand-tuned designs (optimized over several months), for various metrics and exceed critical metrics by as much as 30%.

Recently Machine Learning (ML) accelerators have grown into prominence with significant power and performance efficiency improvements over CPU and GPU. In this paper, we developed an Algorithm Based Error Checker (ABEC) for Concurrent Error Detection (CED) based on an industry quality Convolution Neural Network (CNN) accelerator with priority to meet high safety Diagnostic Coverage (DC) requirement and enhanced area and power efficiency. Furthermore, we developed an Algorithm Based Cluster Checker (ABCC) with coarse-grained error localization to improve run-time availability. Experimental results showed that we could achieve above 99% DC with only 30% area and power overhead for a selected configuration.

With the development of image/video based 3D pose estimation techniques, service robots, human-computer interaction, and 3D somatosensory games have been developed rapidly. However, 3D pose estimation is still one of the most challenging tasks in computer vision. On the one hand, diversity of poses, occlusion and self-occlusion, change in illumination, and complex background increase the complexity of human pose estimation. On the other hand, many application scenarios require high real-time performance for 3D pose estimation. Therefore, we present a 3D pose estimation method based on binocular vision in this paper. For each frame of the binocular videos, the human body is detected firstly; Then Stacked-Hourglass network is used to detect the human joints, and the pixel coordinates of the key joints of all the human bodies in the binocular images are obtained. Finally, with the calibrated camera internal parameters and external parameters, the 3D coordinates of the major joints in the world coordinate system are estimated. This method does not rely on 3D data sets for training. It only requires binocular cameras to perform 3D pose estimation. The experimental results show that the method can locate key joints precisely and the real-time performance is achieved in complex background.

Surgical tool presence detection is one of the key problems in automatic surgical video content analysis. Solving this problem benefits many applications such as the evaluation of surgical instrument usage and automatic surgical report generation. Given the fact that each video is only sparsely labeled at the frame level, meaning that only a small portion of video frames will be properly labeled, existing approaches only model this problem as an image (frame) classification problem without considering temporal information in surgical videos. In this paper, we propose a deep neural network model utilizing both spatial and temporal information from surgical videos for surgical tool presence detection. The proposed model uses Graph Convolutional Networks (GCNs) along the temporal dimension to learn better features by considering the relationship between continuous video frames. To the best of our knowledge, this is the first work taking videos as input to solve the surgical tool presence detection problem. Our experiments demonstrate the employment of temporal information offers a significant improvement to this problem, and the proposed approach achieves better performance than all state-of-the-art methods.

We propose a simple primal-dual method for nuclear norm plus total variation minimization problems. A predictor-corrector scheme to the dual variable is used in our algorithm. Convergence of the method is proved and convergence rate which is O(1/N) in the ergodic sense is also discussed, where N denotes the iteration number. Numerical results including tensor completion, parallel magnetic resonance imaging and dynamic magnetic resonance imaging demonstrate the efficiency of the new algorithm.

Observing the recent progress in Deep Learning, the employment of AI is surging to accelerate drug discovery and cut R&D costs in the last few years. However, the success of deep learning is attributed to large-scale clean high-quality labeled data, which is generally unavailable in drug discovery practices. In this paper, we address this issue by proposing an end-to-end deep learning framework in a semi-supervised learning fashion. That is said, the proposed deep learning approach can utilize both labeled and unlabeled data. While labeled data is of very limited availability, the amount of available unlabeled data is generally huge. The proposed framework, named as seq3seq fingerprint , automatically learns a strong representation of each molecule in an unsupervised way from a huge training data pool containing a mixture of both unlabeled and labeled molecules. In the meantime, the representation is also adjusted to further help predictive tasks, e.g., acidity, alkalinity or solubility classification. The entire framework is trained end-to-end and simultaneously learn the representation and inference results. Extensive experiments support the superiority of the proposed framework.

The most recent history of parallel Magnetic Resonance Imaging (pMRI) has in large part been devoted to finding ways to reduce acquisition time. While joint total variation (JTV) regularized model has been demonstrated as a powerful tool in increasing sampling speed for pMRI, however, the major bottleneck is the inefficiency of the optimization method. While all present state-of-the-art optimizations for the JTV model could only reach a sublinear convergence rate, in this paper, we squeeze the performance by proposing a linear-convergent optimization method for the JTV model. The proposed method is based on the Iterative Reweighted Least Squares algorithm. Due to the complexity of the tangled JTV objective, we design a novel preconditioner to further accelerate the proposed method. Extensive experiments demonstrate the superior performance of the proposed algorithm for pMRI regarding both accuracy and efficiency compared with state-of-the-art methods.

Due to the popularity of smartphones and wearable devices nowadays, mobile health (mHealth) technologies are promising to bring positive and wide impacts on people's health. State-of-the-art decision-making methods for mHealth rely on some ideal assumptions. Those methods either assume that the users are completely homogenous or completely heterogeneous. However, in reality, a user might be similar with some, but not all, users. In this paper, we propose a novel group-driven reinforcement learning method for the mHealth. We aim to understand how to share information among similar users to better convert the limited user information into sharper learned RL policies. Specifically, we employ the K-means clustering method to group users based on their trajectory information similarity and learn a shared RL policy for each group. Extensive experiment results have shown that our method can achieve clear gains over the state-of-the-art RL methods for mHealth.

Chest X-rays is one of the most commonly available and affordable radiological examinations in clinical practice. While detecting thoracic diseases on chest X-rays is still a challenging task for machine intelligence, due to 1) the highly varied appearance of lesion areas on X-rays from patients of different thoracic disease and 2) the shortage of accurate pixel-level annotations by radiologists for model training. Existing machine learning methods are unable to deal with the challenge that thoracic diseases usually happen in localized disease-specific areas. In this article, we propose a weakly supervised deep learning framework equipped with squeeze-and-excitation blocks, multi-map transfer, and max-min pooling for classifying thoracic diseases as well as localizing suspicious lesion regions. The comprehensive experiments and discussions are performed on the ChestX-ray14 dataset. Both numerical and visual results have demonstrated the effectiveness of the proposed model and its better performance against the state-of-the-art pipelines.

In this paper, we propose an efficient algorithm for dynamic magnetic resonance (MR) image reconstruction. With the total variation (TV) and the nuclear norm (NN) regularization, the TVNNR model can utilize both spatial and temporal redundancy in dynamic MR images. Such prior knowledge can help model dynamic MRI data significantly better than a low-rank or a sparse model alone. However, it is very challenging to efficiently minimize the energy function due to the non-smoothness and non-separability of both TV and NN terms. To address this issue, we propose an efficient algorithm by solving a primal-dual form of the original problem. We theoretically prove that the proposed algorithm achieves a convergence rate of O(1/N) for N iterations. In comparison with state-of-the-art methods, extensive experiments on single-coil and multi-coil dynamic MR data demonstrate the superior performance of the proposed method in terms of both reconstruction accuracy and time complexity.

Enterprise systems (ES) are a priority in China. The main literature on ES is dominated by survey-based research on large Western firms, which typically focus on the user firm only and on the before-and-after effects of implementation. Analyses using critical success factors and technology adoption theories feature widely. China-based research on ES although growing is relatively limited and is significantly influenced by Western-based approaches. In contrast, this research examines ES adoption and implementation in Chinese SMEs from a continuous, process perspective using four case studies. It argues that ES adoption and implementation is a complex process that involves multiple stakeholders, and exploring the interplay among them explains why and how an ES is adopted and implemented. Actor-network theory (ANT) informs the main analysis. Importantly, the unit of analysis (UOA) is the user-SME and the technology provider combined. The findings challenge the dominant user-centred adoption theories and the importance of the technology provider in the sociotechnical role.

Nowadays, wireless sensor networks (WSNs) are based on techniques more and more oriented towards image, video and sound processing, hence the recent need of wireless multimedia sensor networks (WMSNs). One of the important challenges for real-time surveillance system is end-to end delay QoS for packet deliveries. Providing end-to-end QoS is difficult due to two reasons. As wireless sensor nodes may require multichip transmissions to reach the sink and some of the wireless transmissions may be not successful. Multimedia data are characterised by their large volume, and have strict requirements in terms of quality of service (QoS) such as bandwidth, delay, packet loss, delay jitter, etc. In this paper, we are interested in routing protocols based on clusters that aim to reduce congestion in order to have reliable data transmission and a reduced loss rate. This is achieved by balancing the traffic load, which results into a balanced energy consumption within the network.