Physics-informed neural networks (PINNs) have shown promising potential for solving partial differential equations (PDEs) by integrating physical laws into the training process. However, their application to multi-material neutron diffusion problems in nuclear reactor physics presents significant challenges due to discontinuities in material properties at interfaces, where abrupt changes in diffusion coefficients lead to non-smooth solutions that are difficult for neural networks to approximate accurately. To address these limitations, we propose an enhanced Physics-Specialized Neural Network (PSNN) approach, integrated with a Source Iteration (SI) technique. This method introduces improved construction methods for specialized functions, utilizing both low-order and high-order polynomial approaches, with specific implementations for one-dimensional and two-dimensional multi-material problems. The specialized functions are engineered to inherently satisfy value and flux continuity conditions at material interfaces as hard constraints, rather than as soft constraints incorporated into the loss function. The SI-PSNN method combines the interface handling capabilities of PSNN with the efficiency of source iteration for eigenvalue problems. We demonstrate the effectiveness of this approach through comprehensive numerical experiments. For multi-group eigenvalue problems, SI-PSNN achieves validation on classical reactor physics benchmarks, producing L-2 errors of 8.22x10(-4) and 1.34x10(-3) for two energy groups on the TWIGL benchmark, with a k(eff) error of only 10.20 pcm, significantly outperforming the SI-PINN, which records errors of 2.48 x 10-2 and 3.01 x 10-2 with a k(eff) error of 922.28 pcm. Additionally, high accuracy is maintained on the IAEA benchmark with 69 subdomains, yielding L-2 errors of 3.30 x 10(-3) and 3.10 x 10(-2), along with a k(eff) error of 124.54 pcm. The proposed method alleviates the limitations of PINNs in multi-material neutron diffusion problems, explores the application of PSNN in neutron diffusion eigenvalue problems, and enhances the capability of physics-informed neural networks in nuclear reactor physics calculations.