Data-Driven Bayesian-Guided Activation Functions for Multi-Task Pattern Recognition

The performance of deep learning (DL) models is highly dependent on the design of activation functions. However, traditional fixed-shape activation functions often suffer encounter issues such as gradient vanishing and local optimisation, particularly when dealing with complex tasks, thereby limiting their adaptability to diverse task requirements. Although trainable activation functions enhance the flexibility of DL models by incorporating learnable parameters, their optimisation process predominantly relies on gradient descent, which is prone to local optima. To address these limitations, this study proposes a data-driven, prior distributions-based optimisation framework for trainable activation functions. The proposed framework integrates a two-stage optimisation strategy, combining gradient descent and Bayesian inference, to significantly enhance neural network performance across multiple tasks. The core contributions of this paper are threefold: 1) The design of a generalised gated composite activation function that adaptively adjusts its shape according to task requirements by dynamically integrating multiple underlying activation functions. 2) The proposal of a two-stage optimisation framework that effectively alleviates the issue of local optima inherent in traditional optimisation methods. and 3) Comprehensive experimental validation on tasks such as image classification, regression, denoising, segmentation, and super-resolution, demonstrating that the proposed method delivers substantial performance gains across various tasks and datasets, surpassing existing classical activation functions and their variants. This study offers novel insights into the selection and optimisation of activation functions for DL models, holding significant academic and practical implications. Our dataset is available at https://github.com/hellorjb/GCAF.