In this work, we investigate the physical mechanisms of endurance degradation of SiOx:Ag-based memristors with electrical and material characterizations. The results indicate that the density of Si dangling bonds will influence the diffusion direction of Ag atoms during the relaxation process. High density of Si dangling bonds lead to Ag diffusion to the SiOx layer and endurance degradation, which is supported by the temperature and frequency dependent conduction mechanism. Finally, excellent endurance above 10(7) cycles is achieved by using oxygen annealing according to the mechanism.