To address the issue of compromised detection accuracy due to noise interference in collected leakage signals during oil and gas pipeline monitoring, this study proposes a denoising method integrating Variational Mode Decomposition (VMD) and an Improved Red-Tailed Hawk Optimization algorithm(IRTH). The conventional Red-Tailed Hawk Optimization algorithm tends to converge to local optima when addressing complex structural optimization problems, making it challenging to locate global optimal solutions. To overcome this limitation, several improvements are implemented: the Halton sequence is introduced to achieve uniform population initialization, thereby enhancing initial population diversity; the simplex method is integrated during iterative updates to refine the positions of the worst-performing individuals and optimize overall population performance; finally, a tangent flight strategy is employed to expand the search space by simulating flight behavior, leveraging stochastic update rules to effectively prevent convergence to local optima and ensure practical applicability. Comparative experiments conducted on nine benchmark functions validate that the enhanced RTH algorithm exhibits significant improvements in both convergence speed and optimization accuracy. To mitigate the issues of over-decomposition and mode deficiency commonly encountered in VMD, the proposed enhanced algorithm is applied to optimize the parameter selection for VMD decomposition. The combined denoising approach is subsequently implemented in practical pipeline leakage signal processing, demonstrating effective noise reduction performance and offering robust technical support for improving the precision of oil and gas pipeline leak detection.