Functionally graded materials (FGMs) are widely used in high-end fields like aerospace and energy for their customizable gradient properties, yet accurate detection of subtle defects in their inhomogeneous structures remains a key challenge for conventional non-destructive testing (NDT) techniques. To address this, this study proposes a multi-task learning-based phased array ultrasonic testing (PAUT) system for FGM defect inspection, featuring a multi-task neural network integrating CNN, RNN, and ensemble learning, plus gradient-corrected acoustic modeling, multi-scale feature extraction, and 3D reconstruction. A physic dataset was built based on Ti6Al4V-ZrO2 FGM acoustic properties, incorporating gradient-induced wave distortion and Gaussian/speckle synthetic noise. The system’s CNN extracts B-scan spatial features and LSTM captures A-scan temporal dependencies, enabling synergistic defect localization and quantification via a combined loss function optimized by Pareto multi-objective strategy. Experimental results show high detection accuracy for different size defects. Transfer learning adapts it to Al2O3-Ni FGMs and trained/validated on 5 defect types with Bayesian uncertainty quantification ensuring reliability. This work provides a physics-informed solution for FGM inspection, overcoming single-modal NDT and homogeneous-material model limitations, and supports intelligent testing system generalization in FGM-based high-end manufacturing.