The LION(evoLved sIgn mOmeNtum)optimizer was found by Google via program search,making it a unique learning-based optimization algorithm.With the incorporation of two distinct interpolations between the previous step momentum and the current step gradient,as well as the integration of decoupled weight decay,LION successfully outperforms many tradi-tional sign-based gradient descent algorithms,showing impressive performance in solving large scale deep learning problems.Although previous studies have investigated its convergence prop-erties,no research has yet provided a comprehensive analysis of its convergence rate,which is more practically relevant.Recognizing that LION can be regarded as solving a specific box-con-strained problem,this paper focuses on demonstrating its convergence to the Karush-Kuhn-Tuck-er(KKT)point at the rate ofO(√dK-1/4),where d is the problem dimension and K is the num-ber of iteration steps.Step further,we remove the constraint and establish that LION converges to the critical point of the general unconstrained problem at the same rate.This rate not only de-livers the currently optimal dependence on the problem dimension d among existing studies but also tightly matches the theoretical lower bound for nonconvex stochastic optimization algorithms with respect to the number of iterations K.Additionally,the lower bound is typically measured by the gradientℓ2norm,while the LION optimizer,as a member of the SignSGD family,usually measures the largerℓ1 norm.Since different gradient norm measures may lead to different conver-gence rate dependencies on d,to verify that our convergence rate is also optimal with respect to d,we conduct extensive experiments across various deep learning tasks.Through these experi-ments,we not only demonstrate that LION achieves lower loss and higher performance compared to standard SGD,but also empirically confirm that the gradient ℓ1/ℓ2norm ratio aligns withΘ(√d),thus proving that our convergence rate matches the theoretical lower bound with respect to d in the empirical sense.
The performance of object detection networks is often limited by the depth of the feature extraction network. Increasing network parameters may yield limited improvements in the detection system's performance. Additional careful designs of network details are necessary, but they can significantly increase training difficulty. This paper introduces a novel object detection method that utilizes generative adversarial training. Our approach focuses on minimizing the EM distance (Wasserstein distance) of the feature distribution as the primary training objective. We enhance the image features so that training of GAN (Generative Adversarial Networks) yields a feature distribution that exceeds that of the original dataset, obtaining a better feature extraction network. A new loss function is also added to the adversarial training process to ensure stable improvement of the detector. A comparative experiment conducted with the original CenterNet network on MS COCO (Microsoft Common Objects in COntext) 2017 reveals that the generative adversarial training method significantly improves the average precision for most of the examined backbone networks. Among the four backbone networks employed in the experiments, the mean improvement in network AP (Average Precision) values ranged from 0.3 to 0.9, demonstrating their success with minimal training efforts. Moreover, none of the four backbone networks experienced an increase in network parameters during inference. Experimental results indicate that the proposed architecture effectively enhances the network's feature extraction capability without compromising speed during inference.
The implementation of large-scale software systems usually depends on low-level frameworks,or third-party libraries.However,the evolution of these frameworks or libraries is independent of the upper-level applications,which brings challenges in upper-level code quality assurance.For instance,changes in framework/library code,such as adding or removing APIs and altering API semantics,may result in inconsistencies among different versions of the frame-work/library code.These inconsistencies can impact the quality of higher-level apps when devel-opers update frameworks/libraries.To address this issue,analyzing the evolution process of framework/library code APIs is essential.This analysis helps upper-level app developers swiftly choose compatible versions or adjust their code.In this context,analyzing the evolution process corresponds to constructing a framework API lifecycle model.Nowadays,existing works propose the API existence-changing model for defect detection,while not considering the influence of se-mantic changes in APIs,especially exception-related code evolution.To fill this gap,this paper adopts static analysis techniques to extract exception summary information in the framework API code,proposes a multi-step matching strategy to obtain the changing process of exceptions,and finally generates exception-aware API lifecycle models for the given framework/library project.Our approach:(1)adopts control-dependency slicing analysis to extract the conditions of the ex-ception-thrown statements;uses a parameter tracing strategy to transform exception-throwing conditions into external-variable-related preconditions;and performs inter-procedure precondition construction by a bottom-up summary-based analysis.(2)proposes the exact-matching and adap-tive-matching strategies to analyze the framework/library code changes including additions,dele-tions and modifications of APIs;generates exception-aware API lifecycle models which cover sev-en API changing types.With this approach,the API lifecycle extraction tool,JavaExP,is imple-mented,which is based on Java bytecode analysis.Compared to the state-of-the-art tool,the F1 score of exception summary information extracted by JavaExP has increased by 67％,with a re-duction in processing time by 87％.The evaluation of real-world projects shows that,compared to the exception-unaware API lifecycle modeling,JavaExP can identify 18％times more API changes.Among the 75,433 APIs under analysis,20％of APIs have changed their exception-throwing behavior at least once after API introduction.These APIs involve a total of more than 7K independent exception changes.The overall results show that the exception-aware lifecycle modeling can describe the evolution process of APIs more accurately.
Superpixel segmentation has recently seen important progress benefiting from the advances in differentiable deep learning. However, the very high-resolution superpixel segmentation still remains challenging due to the expensive memory and computation cost, making the current advanced superpixel networks fail to process. In this paper, we devise Patch Calibration Networks (PCNet), aiming to efficiently and accurately implement high-resolution superpixel segmentation. PCNet follows the principle of producing high-resolution output from low-resolution input for saving GPU memory and relieving computation cost. To recall the fine details destroyed by the down-sampling operation, we propose a novel Decoupled Patch Calibration (DPC) branch for collaboratively augment the main superpixel generation branch. In particular, DPC takes a local patch from the high-resolution images and dynamically generates a binary mask to impose the network to focus on region boundaries. By sharing the parameters of DPC and main branches, the fine-detailed knowledge learned from high-resolution patches will be transferred to help calibrate the destroyed information. To the best of our knowledge, we make the first attempt to consider the deep-learning-based superpixel generation for high-resolution cases. To facilitate this research, we build evaluation benchmarks from two public datasets and one new constructed one, covering a wide range of diversities from fine-grained human parts to cityscapes. Extensive experiments demonstrate that our PCNet can not only perform favorably against the state-of-the-arts in the quantitative results but also improve the resolution upper bound from 3K to 5K on 1080Ti GPUs.
Privacy-preserving clustering has received increasing research attention in recent years. Local differential privacy (LDP) is a privacy model without relying on trusted third parties. It plays a crucial role in distributed privacy-preserving clustering. Most previous methods focus on partition-based clustering (e.g., k-means), which necessitates many iterative interactions. Massive iterations cause a division of the privacy budget and increase the amount of individual noise, affecting the clustering utility. To address this issue, we turn to grid-based clustering and design the GC-LDP algorithm to balance the privacy and clustering utility with only three rounds of interactions. In GC-LDP, our core contribution is to develop a non-uniform grid division method via the coefficient of variation (CV), which can generate a grid structure approximating the global data distribution within two rounds of interactions. Besides, we designed a new perturbation mechanism to reduce the amount of individual noise injection. Further, a cell aggregation method is developed by exploring the relative density difference among grids to achieve multi-density clustering. Theoretical analysis and experiments on real-world datasets show that GC-LDP can obtain high-quality clustering results while satisfying local differential privacy.
