Fabric defect detection is essential for ensuring the quality of textiles, particularly when addressing lustre issues in silk materials and tiny, inconspicuous flaws. This paper introduces a saliency detection framework for silk surface defects, named EFRNet, which aims to overcome the limitations of existing technologies in handling such fabric imperfections. Firstly, a multilayered spatial domain edge perception method was designed. This method enhances the perception of defect edge details through edge‐aware units and integrates these detailed features with high‐level semantic information using cross‐layer connection strategies, thereby achieving precise detection of defect edges. Secondly, a context‐aware multilayer feature fusion technique is proposed, which includes attention‐induced feature aggregation units and global context differential units, aimed at optimising feature selection, reducing background interference and emphasising the salient features of fabric defects through differential analysis. Experiments have demonstrated that EFRNet excels in improving the accuracy of small defect detection and local detail recognition. Compared with existing methods, it significantly enhances the quality of saliency maps, proving its effectiveness in enhancing the adaptability and precision of fabric defect detection.