One of the foremost earlier sign of breast cancer is architectural distortion. This research work proposes the fish swam optimization based detection of architectural distortion in mammograms acquired prior to the diagnosis of breast cancer in the interval between scheduled screening sessions. The potential sites are obtained using node maps through the implementation of Gabor filter and portrait modeling namely linear phase for detecting its presence. The ROI is extracted after pre-processing and is characterized with entropy measures such as angle, coherence and orientation strength. The ROI is also represented using Fourier spectrum that includes Shannon’s entropy and Renyi entropy. The outcome of the experiment reveals that, using the entropy measures with fish swarm optimization for feature selection performs better than the particle swarm optimization and ant colony optimization. The experimental results are proved by performing classification using Artificial Neural Network.