A Novel Intelligent Assessment Based on Audio-Visual Data for Chinese Zither Fingerings

In this paper, we make a novel study on the intelligent assessment for Chinese zither (Zheng) fingerings in the cross field of art AI. Due to the gaps between science and art, there is a lack of deep combination of musical instrument expertise with computer aided technology. This paper attempts to explore the inheritance and innovation of Chinese classical musical instrument in this interdisciplinary study. We integrate video understanding, audio analysis with zither professional knowledge to design a pipeline for Zheng intelligent evaluation. Firstly, we establish a Zheng fingering Dataset based on video and audio (ZF-VA-Dataset). Then, according to zither teaching experience and accumulated research on machine perception, we formulate a Zheng fingering assessment scale, which is served as a bridge between Zheng knowledge and audio-visual technology. Finally, an intelligent evaluation method based on dual modes is put forward for Chinese zither fingerings. The experimental results show that the proposed scheme is feasible and effective in achieving intelligent assessment of Zheng fingerings.