State Gradients for Analyzing Memory in LSTM Language Models

•Decreasing state gradient SVs correspond to decreasing importance of the input.•The average memory of LSTM LMs decays quickly but not exponentially over time.•Average memory depends on type of data, selectiveness on the size of the dataset•Infrequent words are remembered better and longer by LSTM LMs.•LSTM LMs remember on average (syntactic differences between) nouns best.