With the continuous development of Web technology, many Internet issues evolve into Big Data problems, characterized by volume, variety, velocity and variability. Among them, how to organize plenty of web pages and retrieval information needed is a critical one. An important notion is document classification, in which nearest neighbors query is the key issue to be solved. Most parallel nearest neighbors query methods adopt Cartesian Product between training set and testing set resulting in poor time efficiency. In this paper, two methods are proposed on document nearest neighbor query based on pairwise similarity, i.e. brute-force and pre-filtering. brute-force is constituted by two phases (i.e. copying and filtering) and one map-reduce procedure is conducted. In order to obtain nearest neighbors for each document, each document pair is copied twice and all records generated are shuffled. However, time efficiency of shuffle is sensitive to the number of the intermediate results. For the purpose of intermediate results reduction, pre-filtering is proposed for nearest neighbor query based on pairwise similarity. Since only first top-k neighbors are output for each document, the size of records shuffled is kept in the same magnitude as input size in pre-filtering. Additionally, detailed theoretical analysis is provided. The performance of the algorithms is demonstrated by experiments on real world dataset.

The K Nearest Neighbor (KNN) technique is very simple, highly efficient and effective in the field of text categorization, pattern recognition, object recognition etc. However, its efficiency is challenged by Big Data, which is characterized with volume, variety and velocity. All these characters require a new KNN algorithm with better time complexity, which is lower than O(n) at least. Additionally, the characters witness that centralized variants executed in a single thread are no longer reasonable. In this paper, we propose an efficient and scalable strategy, which we called as BALLKNN, for KNN classifier. In BALLKNN, since computation of some samples are reduced by replacing complicated quadratic sum with simple subduction and comparison, time efficiency improvement is obtained; the disadvantage of dependency on global data structure is overcome with the help of temporary variable(s), BALLKNN can be implemented with mUltiple threads easily, and it scales well with the size of the dataset. Meanwhile, as all samples are handled, the classilication accuracy is guaranteed different from traditional sample reduction methods. Furthermore, detailed theoretical analysis on relationship between dimensionality of input and time complexity is provided. Finally, extensive experiments in large real datasets, with tens or hundreds of thousands of records and up to 618 dimensions, have demonstrated the efficiency and scalability of our methods.