Genetic Algorithm-Based Hyperparameter Tuning for Transformer Models in Biomedical Text Mining

In biomedical text mining, it is imperative to be able to get accurate clinical insights from large, varied datasets. Transformer-based models like BERT and BioBERT have gotten top-notch results. However, the effectiveness of these models largely depends on the proper hyperparameter settings, which are typically determined by hand or through grid search, both of which are computationally expensive. The authors of this study recommend employing a Genetic Algorithm (GA) to autonomously modify transformer architecture hyperparameters such as learning rate, batch size, dropout rate, and attention heads when handling biomedical datasets. The GA uses selection, crossover, and mutation to change hyperparameters that are stored as chromosomes. It uses the validation F1-score fitness as a guide. The experimental results on benchmark biomedical datasets show a 6.8% improvement in F1-score and a 23% reduction in training time compared to baseline tuning methods. The results show that GA can efficiently search large search spaces and find high-performance configurations. This makes it possible to deploy transformer models in clinical text mining applications more quickly.