All-optical computing offers ultra-high speed, low-power consumption, and parallel processing capabilities, crucial for overcoming Moore's Law limitations. However, conventional single-wavelength diffractive deep neural networks (D2NN) face significant challenges in achieving synergistic optimization between high-precision optical edge-feature extraction and classification tasks. Here, an edge-detecting spin-differential diffractive neural network (ESD-DNN) is proposed for single-wavelength all-optical object classification. The network architecture is implemented through a Pancharatnam-Berry phase gradient metasurface to achieve rapid edge-feature extraction, while classification inference is accomplished by utilizing a spin-differential mechanism based on left-/right-handed circularly polarized (LCP/RCP) components. Through end-to-end optimization of the diffractive layers, the ESD-DNN achieves co-optimization of edge-feature extraction and classification, significantly improving accuracy while reducing computational costs. Numerical validations reveal that the single-layer ESD-DNN attains 97.5% (MNIST) and 87.5% (Fashion-MNIST) classification accuracy, surpassing traditional single-wavelength D2NN by 10.2% and 5.2%, respectively. Meanwhile, it achieves 5-fold higher computational efficiency while reducing time complexity by 80% compared to a four-layer D2NN. Remarkably, under extreme conditions such as moderate turbulence intensity or thermal lensing effects, the network maintains >90% classification accuracy (MNIST), demonstrating its exceptional environmental robustness. These findings pave the way for applications in artificial intelligence, satellite remote sensing, intelligent industrial inspection, and space optical communications.