Deep Learning (DL), especially with Large Language Models (LLMs), brings benefits to various areas. However, DL training systems usually yield prominent idling GPU resources due to many factors, such as resource allocation and collective communication. To improve GPU utilization, we present SpecInF, which adopts a Speculative Inference Filling method to exploit idle GPU resources. It collocates each primary training instance with additional inference instances on the same GPU, detects the training bubbles and adaptively fills with online or offline inference workloads. Our results show that SpecInF can effectively enhance GPU utilization under mainstream parallel training modes, delivering additional up to 14 × offline inference throughputs than TGS and 67

Serverless computing, with its ease of management, auto-scaling, and cost-effectiveness, is widely adopted by deep learning (DL) applications. DL workloads, especially with large language models, require substantial GPU resources to ensure QoS. However, it is prone to produce GPU fragments (e.g., 15%-94%) in serverless DL systems due to the dynamicity of workloads and coarse-grained static GPU allocation mechanisms, gradually eroding the profits offered by serverless elasticity. Different from classical serverless systems that only scale horizontally, we present introspective elasticity (IE), a fine-grained and adaptive two-dimensional co-scaling mechanism to support GPU resourcing-on-demand for serverless DL tasks. Based on this insight, we build Dilu, a cross-layer and GPU-based serverless DL system with IE support. First, Dilu provides multi-factor profiling for DL tasks with efficient pruning search methods. Second, Dilu adheres to the resourcing-complementary principles in scheduling to improve GPU utilization with QoS guarantees. Third, Dilu adopts an adaptive 2D co-scaling method to enhance the elasticity of GPU provisioning in real time. Evaluations show that it can dynamically adjust the resourcing of various DL functions with low GPU fragmentation (10%-46% GPU defragmentation), high throughput (up to 1.8× inference and 1.1× training throughput increment) and QoS guarantees (11%-71% violation rate reduction), compared to the SOTA baselines.

With the explosive data growth on the web, there are massive textual data without class labels such that zero-shot text classification has attracted much research attention. However, existing zero-shot text classification models still take the class labels as the weakly supervised signal, which are usually unavailable in the open domain. In this paper, we study the text classification problem in a fully zero-shot setting, in which not only are we not given any training samples for unseen classes, but also the label names and the total number of unseen classes are unknown. We propose a fully zero-shot text classification model (FZeroTC) in a semi-supervised learning framework to simultaneously discover and label unseen classes. In the FZeroTC model, a pairwise loss and a Kullback–Leibler divergence-based regularization term are specially designed for unseen class discovery, and a faraway loss is specially designed for class labeling. We propose three different kinds of learning strategies based on the pretrained language model and prompt learning to train FZeroTC. From extensive experiments on four public text classification datasets, FZeroTC outperforms the state-of-the-art zero-shot text classification models in terms of unseen class discovery performance and can provide high-quality labels for unseen classes.

As deep learning technology continues to advance, image generation models, especially models like Stable Diffusion, are finding increasingly widespread application in visual arts creation. However, these models often face challenges such as overfitting, lack of stability in generated results, and difficulties in accurately capturing the features desired by creators during the fine-tuning process. In response to these challenges, we propose an innovative method that integrates Singular Value Decomposition (SVD) into the Low-Rank Adaptation (LoRA) parameter update strategy, aimed at enhancing the fine-tuning efficiency and output quality of image generation models. By incorporating SVD within the LoRA framework, our method not only effectively reduces the risk of overfitting but also enhances the stability of model outputs, and captures subtle, creator-desired feature adjustments more accurately. We evaluated our method on multiple datasets, and the results show that, compared to traditional fine-tuning methods, our approach significantly improves the model's generalization ability and creative flexibility while maintaining the quality of generation. Moreover, this method maintains LoRA's excellent performance under resource-constrained conditions, allowing for significant improvements in image generation quality without sacrificing the original efficiency and resource advantages.

Geo-distributed ML training can benefit many emerging ML scenarios (e.g., large model training, federated learning) with multi-regional cloud resources and wide area network. However, its efficiency is limited due to 2 challenges. First, efficient elastic scheduling of multi-regional cloud resources is usually missing, affecting resource utilization and performance of training. Second, training communication on WAN is still the main overhead, easily subjected to low bandwidth and high fluctuations of WAN. In this paper, we propose a framework, Cloudless-Training, to realize efficient PS-based geo-distributed ML training in 3 aspects. First, it uses a two-layer architecture with control and physical training planes to support elastic scheduling and communication for multi-regional clouds in a serverless maner.Second, it provides an elastic scheduling strategy that can deploy training workflows adaptively according to the heterogeneity of available cloud resources and distribution of pre-existing training datasets. Third, it provides 2 new synchronization strategies for training partitions among clouds, including asynchronous SGD with gradient accumulation (ASGD-GA) and inter-PS model averaging (MA). It is implemented with OpenFaaS and evaluated on Tencent Cloud. Experiments show that Cloudless-Training can support general ML training in a geo-distributed way, greatly improve resource utilization (e.g., 9.2%-24.0% training cost reduction) and synchronization efficiency (e.g., 1.7x training speedup over baseline at most) with model correctness guarantees.

Semi-supervised learning (SSL), which leverages limited labeled data and a large amount of unlabeled data for model training, has been widely studied to mitigate the requirement for expensive and time-consuming annotations. Recently proposed methods have achieved promising yet unstable results, which presume that initial samples are randomly selected and labeled. For improving the fluctuated performance while saving annotation overhead, effective prior labeling for SSL on the source cluttered unlabeled dataset is challenging but significant. In this paper, we propose a novel criterion and a distribution balance strategy to automatically achieve targeted labeling without access to the test set and any labels. Comprehensive experiments are conducted on commonly-used datasets to demonstrate the effectiveness of our method. Furthermore, targeted labeling is orthogonal to existing framework-centric SSL methods and can achieve state-of-the-art performance.

Most existing network embedding based anomalous link detection methods regard network embedding and anomalous link detection as two independent tasks. However, removing anomalous links from the original network can reduce the data noise, thus hopefully improving the performance of network embedding models and anomalous link detection. In this paper, we propose an Anomaly Aware Network Embedding (AANE) framework by simultaneously learning node embedding and detecting anomalous links in a unified way. To instantiate the AANE framework, we propose a heuristic anomalous link selection based model AANE-H and an embedding disentangling based model AANE-D on Graph Auto-Encoder (GAE). In AANE-H, we adopt an anomalous link selector to iteratively select significant anomalous links based on a heuristic rule during model training, while in AANE-D the normal and anomalous links are generated by disentangled normal and anomalous embedding respectively. For the evaluation purpose, we propose a heuristic anomalous link generation algorithm to inject synthetic anomalous links into six real-world network datasets used in our experiments. Experimental results show that AANE outperforms both the state-of-the-art network embedding models and anomalous node detection models in terms of anomalous link detection performance. As a general network embedding model, AANE can also improve other downstream tasks like node classification.

事件检测旨在识别文本中提到的事件及其类型.基于触发词的事件检测方法需要额外的人工成本标注事件触发词.本文从无触发词的文本语义提取出发,提出了一种基于对比学习增强句子语义的事件检测方法.该方法首先在事件检测数据集上通过自监督学习对预训练的语言模型基于转换器的双向编码表示器(BERT)调优,提高语言模型的领域适应性.然后利用掩码(mask)操作和丢弃(dropout)操作构建自监督对比样例,增加监督对比样例,实现了 自监督对比和监督对比2种句子语义增强的方法.此外在训练过程中自动调整对比损失和事件分类的交叉熵损失的权重,以降低人工调参的成本,同时提高模型收敛速度.在自动内容抽取(ACE)2005中英文语料上的实验结果表明,本文方法比先前无触发词事件检测方法取得更好的结果,与利用预训练BERT模型微调的事件检测方法相比也具有优势.

Serverless workflows are characterized as multi-stage computing, while downstream functions require accessing intermediate states or the output of upstream functions for running. The workflow's performance can be easily affected due to the inefficiency of data access. Studies accelerate data access with various policies, such as direct and indirect methods. However, these methods may fail due to various limitations such as resource availability. In this paper, we propose asynchronous state replication pipelines (ASRP) to speed up workflows for general applications, replacing the sequential computing pattern of current workflows. Chitu is built based on the insight with three main points. First, differentiable data types (DDT) are provided at the programming model level to support incremental state sharing and computation. Second, ASRP works by continuously delivering changes of DDT objects in real-time so that downstream functions can consume the objects without waiting for the ending of upstream functions. Third, we make a systematic design to support DDT and ASRP in Chitu framework, including direct communication and change propagation. We implement Chitu atop OpenFaaS, compare it with popular serverless workflow frameworks, and evaluate it with three commonly seen cases. The results show that Chitu accelerates data transmission in general serverless workflows up to 1.7x, and speeds up end-to-end applications by up to 57%.

Semi-supervised object detection (SSOD) is a significant application of Semi-supervised learning to further improve object detectors but suffers more seriously from confirmation bias and error accumulation caused by the classes imbalance. Existing SSOD approaches have attempted to address this issue but fails to consider dynamically changed detection difficulties of different classes for detectors. In this paper, we propose adaptively focus-aware confidence, which treats object classes differently. Predictions generated from the teacher and student models are stored in a memory dictionary, and the differences between them are utilized to adaptively perceive learning statuses. Based on this, confidence thresholds are flexibly assigned and adjusted for different classes. Extensive experiments are conducted on MS-COCO benchmark dataset with multiple protocols and our SSOD framework out-performs the state-of-the-art competitors by a large margin.

Dropout regularization has been widely used in deep learning but performs less effective for convolutional neural networks since the spatially correlated features allow dropped information to still flow through the networks. Some structured forms of dropout have been proposed to address this but are prone to result in over or under regularization as features are dropped randomly. In this paper, we propose a targeted regularization method, TargetDrop, which incorporates the attention mechanism to drop several discriminative feature units. Specifically, it masks out the target regions in the feature maps corresponding to the target channels. We conduct comprehensive experiments and demonstrate that TargetDrop outperforms the other dropout-based regularization methods.

序列到序列(seq2seq)方法在开放域对话生成领域中备受研究学者的关注.然而,标准的序列到序列模型容易产生语义冲突和不连贯的对话回复,这种不一致性是现有系统生成的回复显著有别于人类真实对话的重要原因之一.对话生成中的一致性既包括回复内部的语义一致性,也包括上文与其回复之间的外部关联性.本文提出了一个新的对话生成框架,称为基于张量匹配的生成式对抗网络(MatchGAN),以提高对话回复与其上文之间的外部关联性.与传统的基于最大似然估计的方法不同,该框架通过基于序列到序列模型的生成器和基于张量匹配网络的判别器之间的对抗学习来生成与上文相关的回复.通过使用匹配网络对上文与回复之间的多维关系进行建模,该模型所产生的回复更加符合人类对话的特点.此外,本研究进一步引入了目标侧注意力机制来增强所产生回复的内部语义一致性.实验结果表明,本文提出的框架能够产生高质量的对话回复,在量化指标评价和人工评测方面均优于其他基线方法.

Learning unlabeled samples from unseen categories given limited labeled data is a challenging problem. Existing few-shot learning methods fail to generate satisfactory feature representations due to tackling the informative and interference information without distinction. In this paper, we propose an attention-guided two-stream convolutional neural network (AGTSNet), which highlights the salient and discriminative features of the main object while alleviating the background interference to address this indiscriminate treatment. Comprehensive experiments are conducted on few-shot image classification with four standard benchmark datasets to demonstrate the effectiveness of our method.

We focus on the problem of how to boost greater confidence in computations outsourced to an untrusted cloud server. Naturally, we aim for solutions that cater for the following two security requirements: data confidentiality/privacy and verifiability of computations. This problem is addressed by the research community through an intriguing paradigm combining a verifiable computation (VC) and a fully homomorphic encryption (FHE). Despite continuing advances in achieving efficient VC and FHE, existing approaches that meet both requirements simultaneously are expensive and impractical for servers and clients. In this work, we first introduce an interesting primitive, called versatile message authenticators, to achieve a verifiable outsourced computation on encrypted data and then give a concrete construction. Without the involvement of FHE, our scheme mainly depends on a pseudo-random function and a adapted Homomorphic MAC scheme. Our solution is proven secure in a strong security model where the adversary is allowed to issue any verification queries. Furthermore, it allows for arbitrary composition and generates constant-size tags. However, our construction just supports linear functions. Finally, our security proof shows its semantic security and unforgeability.

Distributed ML training is widely used to improve training performance. However, current distributed training frameworks bring undesirable burdens to application-oriented users due to its server-centric design. It is also difficult for users to customize training (e.g., with adaptive policies) to guarantee performance in dynamic environments. Thus, it is meaningful to make training framework lightweight and programmable. We argue that serverless paradigm can effectively help meet the demands. In this paper, we propose TrainFlow, adopting serverless paradigm to simplify and extend programmability of data-parallel training. First, the basic framework is built with a novel serverless process model, providing a high-level view and various state sharing. Then training can be divided into 2 processes with specific workflows. Second, TrainFlow provides an event-driven hook mechanism, allowing users to customize training workflow. We implement and evaluate TrainFlow with OpenFaaS. Experiments demonstrate its availability and programmability. For availability, TrainFlow can support various training patterns, and shows advantages of performance (e.g., 1.6× higher speedup ratio than baseline) and resource consuming (e.g., at most 41.0% less memory consuming than baseline). For programmability, TrainFlow can work with adaptive policies as expected (e.g., at most 1.48× higher throughput in a case).

Bitcoin transactions are pseudo-anonymous, which can be exploited to reveal a user' s private information. To eliminate this threat, this paper presents FairMixer, a highly secure and efficient Bitcoin mixing system using the trusted execution environments ( TEEs) . With the TEE' s confiden-tiality and integrity guarantees for code and data, FairMixer enables a correct and privacy-preserving mixing process. However, a TEE-based implementation cannot prevent the manipulation of inputs to the mixer, such as mixing request submissions and blockchain feeds. Against this background, FairMixer captures users' mixing requests via Bitcoin transactions for deterring a malicious service provider from dropping benign participants. To constrain misbehavior during a mixing mission, a misconduct monitoring mechanism and a penalty mechanism are introduced. The proposed scheme is fully compatible with Bitcoin and forces mixes to be accountable. Finally, a prototype of FairMixer is provided using Intel Software Guard Extensions ( SGX) and its performance is evaluated in the Bitc-oin Testnet. FairMixer mixes 700 inputs in just 8. 39 s, which outperforms most existing decentral-ized mixers.

面向大电网的实时仿真计算被建模为协同运行的多进程并行计算任务的集合,任务调度与资源分配的优劣是其能否实现实时性目标的关键因素之一。本文针对大规模电网机电-电磁混合仿真计算的应用场景,根据任务进行了特性分析,总结了资源利用率规律,提出了一种通信敏感的组调度框架（CGS）。该框架提出了一种集中式两阶段调度架构,以在不中断在线流运行过程中对任务进行主动采样和调度,达到精准预测需求的目标;提出了基于通信图的图划分策略与基于调度模型的匹配策略相结合的CGS调度算法,实现了进程组调度,降低了任务跨节点的通信开销。实验从任务平均划分数、平均割边成本、有负载节点数、集群资源利用率和资源碎片率指标方面对5种基线算法与应用了CGS的算法进行对比测试,结果表明CGS与基本策略相比至少降低了37%的进程间通信开销,减少了19%的资源碎片,平均提高了34%的集群资源利用率。

Semi-supervised learning (SSL), which leverages a small number of labeled data that rely on expert knowledge and a large number of easily accessible unlabeled data, has made rapid progress recently. However, the information comes from a single modality and the corresponding labels are in form of one-hot in pre-existing SSL approaches, which can easily lead to deficiency supervision, omission of information and unsatisfactory results, especially when more categories and less labeled samples are covered. In this paper, we propose a novel method to further enhance SSL by introducing semantic modal knowledge, which contains the word embeddings of class labels and the semantic hierarchy structure among classes. The former helps retain more potential information and almost quantitatively reflects the similarities and differences between categories. The later encourages the model to construct the classification edge from simple to complex, and thus improves the generalization ability of the model. Comprehensive experiments and ablation studies are conducted on commonly-used datasets to demonstrate the effectiveness of our method.

Running HPC in the cloud has gained more and more practice. As a new cloud paradigm, serverless is highly attractive for HPC service providers due to its distinctive benefits such as scalability. However, it is difficult for serverless to meet the demands of MPI programming and running, resulting in that MPI programs cannot scale with serverless functions. We introduce a serverless parallel function model to solve the problems. It divides parallelism at function and worker levels to bridge gaps in programming and running between serverless and MPI. Then we present the SMPI framework atop the model. For programming, SMPI redefines the function generation pipeline for parallel functions to prepare metadata for MPI parallel functions. For running, SMPI employs the parallel function gateway and scheduler to realize parallel function invocation and instantiation for MPI parallel functions. It is implemented and evaluated with OpenFaaS. Experiments show that SMPI supports MPI programming and running in a complete serverless manner. Compared to server-centric methods, it reduces efforts on cluster maintenance, provides scalable serverless MPI computing with competitive performance (0.559–1.048s slower of start-up time, and 0.145s–0.945s slower of computing time than best-behaved baseline), and is potential to scale on multiple clusters for higher scalability.