Polymer-based responsive structural color materials are polymers with ordered microstructures that can change colors. They offer unique advantages and present a novel solution for many industries. Despite the presence of classic review papers on responsive photonic materials, a comprehensive review of polymer-based responsive structural color materials, including fundamental research and applications, is still lacking. This study provides a detailed overview of polymer-based responsive structural color materials, including polymer selection, structure fabrication, response mechanisms, and applications. The current challenges and future developments are also presented. It is hoped that this review will encourage interdisciplinary communications across physics, chemistry, biology, medicine, materials science, and engineering.