NLPDedup: Using Natural Language Processing for Data Deduplication

With the increasing use of technology, the amount of data has been grown rapidly. The data can be reduced by using data deduplication functions. It finds the redundant data stored in a file system, and it deletes the data; hence, the amount of data can be reduced. However, the function affects the performance because of the many read I/O and comparing data. To mitigate the performance penalty, it is effective to narrow down the files processed. Conventional methods use file metadata, such as file size, and a hash value generated from file contents as indicators. However, if the many files are stored in a file system, then the methods cannot narrow files down efficiently. Moreover, if the large files are stored in the file system, then the calculation of hash value becomes a high CPU load. The indicators are recently insufficient. Therefore, a new indicator is necessary for the method. This paper proposes a novel method, called NLPDedup, to narrow files down for data deduplication functions. NLPDedup uses the file names as an indicator for narrowing down target files. This paper describes the overview of NLPDedup, how NLPDedup determines the target files, and the evaluation results. From the evaluation results, the thresholds of indicator need to be set in terms of the algorithms and use cases.