Development of a novel online chatter monitoring system for flexible milling process

•An online chatter monitoring system consisting of chatter detection and chatter frequency estimation is developed.•Matrix notch filters are designed to remove the spindle-related periodic components.•A normalized cyclostationarity energy ratio indicator is proposed for chatter detection.•An adaptive TLS-ESPRIT algorithm is proposed to estimate the dominant chatter frequency accurately.•The developed chatter monitoring system is successfully verified by the simulation signal and milling experiments.