Reply to Hausner et al. Re: use of text mining tools in the development of search strategies—comparison of different approaches

What is new?Key findings• TMT searches reduced strategy development time compared to UP searches.• TMT searches were slightly less sensitive for simple and complex topics.What this adds to what is known?• Studied freely available TMTs to increase real world SR applicability.• TMTs useful adjuncts when timeliness is more important than comprehensiveness.What should change now?• Comparative TMT functionality studies are needed to develop “best practices.” Key findings• TMT searches reduced strategy development time compared to UP searches.• TMT searches were slightly less sensitive for simple and complex topics.What this adds to what is known?• Studied freely available TMTs to increase real world SR applicability.• TMTs useful adjuncts when timeliness is more important than comprehensiveness.What should change now?• Comparative TMT functionality studies are needed to develop “best practices.” We thank Hausner et al. for their commentary and reply to each of their points below. We are happy to agree it is entirely possible that our team would have achieved similar results if we had access to either of the fee-based software packages (i.e., SimStat/WordStat or the R Text Mining Package) used by Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen (Institute for Quality and Efficiency in Health Care) information specialists in their research along with their training and support [1Hausner E. Guddat C. Hermanns T. Lampert U. Waffenschmidt S. Prospective comparison of search strategies for systematic reviews: an objective approach yielded higher sensitivity than a conceptual one.J Clin Epidemiol. 2016; 77: 118-124Abstract Full Text Full Text PDF PubMed Scopus (28) Google Scholar, 2Hausner E. Guddat C. Hermanns T. Lampert U. Waffenschmidt S. Development of search strategies for systematic reviews: validation showed the noninferiority of the objective approach.J Clin Epidemiol. 2015; 68: 191-199Abstract Full Text Full Text PDF PubMed Scopus (45) Google Scholar, 3Hausner E. Waffenschmidt S. Kaiser T. Simon M. Routine development of objectively derived search strategies.Syst Rev. 2012; 1: 19Crossref PubMed Scopus (78) Google Scholar]. Our interest lay in expanding the research base in this field by exploring the use of free text-mining tools (TMTs) available to evidence synthesis searchers and testing their performance on simple and complex topics (simple = one indication, one treatment clinical topics; complex = everything else) in a ‘real world’ environment, that is, in which searchers only have access to free tools for which no best-practice guidance currently exists [[4]Paynter R.A. Featherstone R. Stoeger E. Fiordalisi C. Voisin C. Adam G.P. A prospective comparison of evidence synthesis search strategies developed with and without text-mining tools.J Clin Epidemiol. 2021; 139: 350-360Abstract Full Text Full Text PDF PubMed Scopus (9) Google Scholar,[5]Paynter R.A. Fiordalisi C. Stoeger E. Erinoff E. Featherstone R. Voisin C. et al.A prospective comparison of evidence synthesis search strategies developed with and without text-mining tools. Agency for Healthcare Research and Quality (US), Rockville (MD)2021Crossref Google Scholar]. In our study, for both processes, the recall was generally very high (i.e., we are not saying the technology does not work) and perhaps with standardization, such as Hausner et al. describe, it could be even higher than our results. This initial foray into a different set of tools and using the creativity of information specialists to develop novel methods we hoped would lay some groundwork for future studies that may indeed establish best practices for these individual tools. Please observe our recently published short pros and cons article for more details on using TMTs [[6]Adam G.P. Paynter R. Development of literature search strategies for evidence syntheses: pros and cons of incorporating text mining tools and objective approaches.BMJ Evid Based Med. 2022; 28 (bmjebm-2021-111892)Google Scholar]. Our current research project is creating a user-friendly, comparative guide to search tools listed on the SR Toolbox website. Using standardized criteria we are evaluating availability, usability, and performance, along with providing some usage tips and tricks with the results presented in an easy-to-understand graphic. With so many tools available but often too little time to try them out, we hope our fellow information specialists will find this guide useful. We agree that the experience with TMTs and ongoing professional support are important ingredients in their successful use. We count ourselves among those evidence synthesis searchers who are not located in such an environment and thus wanted to investigate the performance of tools available to us and whether the time to learn a new tool paid off (even with no experience and no best-practices guidance available). We hope our brief comments have been useful to others regarding what works and what does not work so well. We agree that a representative seed set of citations is critical for generating a more accurate analysis. To our knowledge, there has not been any research on the effect of an unrepresentative (biased) seed set of citations on the development of searches. With the likely increase in the use of search tools, attention to this issue is critical in our opinion and the development of a risk of bias tool would be most welcomed. The requested information would indeed make an interesting additional study; however, we did not systematically collect these, so are unable to provide them. This area of research would potentially benefit from the use of a standard dataset of search strategies and relevant citations, such as that created by Scells et al. [[7]Scells H. Zuccon G. Koopman B. Deacon A. Azzopardi L. Geva S. A test collection for evaluating retrieval of studies for inclusion in systematic reviews.SIGIR. 2017; 17: 1237-1240Google Scholar], which could be used across evaluations to give more consistent results.