Modeling Document Lengths

In some applications, very approximate modeling of the distribution of document lengths will suffice. However, there are many scenarios in which accurate modeling is desirable. For example, significant gains in retrieval effectiveness were achieved at TREC-3 through better normalization of document length [74, 81]. Those effects could not have been observed on a simulated collection in which all documents had the same length. Furthermore, distributions of term co-occurrences, distributions of TFs, and topic mixtures are all affected by the distribution of document lengths.