Information is shared more extensively when a user can confidently classify all his information according to its desired degree of disclosure prior to transmission. While high quality classification is relatively straightforward for structured data (e.g., credit card numbers, cookies, confidential reports), most consumer and business information is unstructured (e.g., Facebook posts, corporate email). All current technological approaches to classifying unstructured information seek to identify only that information having the desired characteristics (i.e., to maximize the percentage of filtered content that requires privacy protection). Such focus on boosting classifier Precision (P) causes technology solutions to miss sensitive information [i.e., Recall (R) is compromised for the sake of P improvement]. Such privacy protection will fall short of user expectations no matter how intelligent the technology may be in extending beyond keywords to user meaning. Systems must simultaneously optimize both P and R in order to protect privacy sufficiently to encourage the free flow of personal and corporate information. This requires a socio-technical methodology wherein the user is intimately involved in iterative privacy improvement. The approach is a general one in which the classifier can be modified as necessary at any time when sampling measures of P and R deem it appropriate. Matching the ever-evolving user privacy model to the technology solution (e.g., active machine learning) affords a technique for building and maintaining user trust.

Retrieval of relevant unstructured information from the ever-increasing textual communications of individuals and businesses has become a major barrier to effective litigation/defense, mergers/acquisitions, and regulatory compliance. Such e-discovery requires simultaneously high precision with high recall (high-P/R) and is therefore a prototype for many legal reasoning tasks. The requisite exhaustive information retrieval (IR) system must employ very different techniques than those applicable in the hyper-precise, consumer search task where insignificant recall is the accepted norm. We apply Russell, et al.’s cognitive task analysis of sensemaking by intelligence analysts to develop a semi-autonomous system that achieves high IR accuracy of F1 ≥ 0.8 compared to F1 < 0.4 typical of computer-assisted human-assessment (CAHA) or alternative approaches such as Roitblat, et al.’s. By understanding the ‘Learning Loop Complexes’ of lawyers engaged in successful small-scale document review, we have used socio-technical design principles to create roles, processes, and technologies for scalable human-assisted computer-assessment (HACA). Results from the NIST-TREC Legal Track’s interactive task from both 2008 and 2009 validate the efficacy of this sensemaking approach to the high-P/R IR task.

Treating the information retrieval (IR) task as one of classification has been shown to be the most effective way to achieve high performance. In real-world Systems, a human is the ultimate determinant of relevance and must be integrated symbiotically into the control structures. We report on a hybrid, Human-Assisted Computer Classification system that opportunistically pairs processes of Active Learning and User Modeling to produce a high-Q computational engine. Top-down human goals are impedance-matched with bottom-up corpus analysis utilizing critical control loops. The System contributions of humans and machines as 'Proxy,' 'Assessor,' and 'Classifier' elements are blended through inter-related 'Model,' 'Match,' and 'Measure' processes (M3) to achieve consistently high precision IR with high recall. We report results for over a dozen topics, with confirmation of internal measures from topic 103 of the 2008 TREC legal track's interactive task.

Throughout its history, AI researchers have alternatively seen their mission as producing computer behavior that is indistinguishable from that of humans or as providing computational tools to augment human intelligence. The legal context provides a particularly rich domain for exploring the benefits of both approaches because human judgments are central to the delivery of just outcomes with formal mechanisms for challenging any and all judgments within its core practices and institutions. Through exploration of legal e-discovery, we explicitly address these alternative paradigms for applying interdisciplinary sciences to knowledge-based systems. We demonstrate through a series of quantitative studies that a system architecture for Human-Aided Computer Cognition automates and replicates judgments significantly better than the ubiquitous, traditional Computer-Aided Human Cognition of senior attorney relevance as is defined in legal practice.

Active learning algorithms actively select training examples to acquire labels from domain experts, which are very effective to reduce human labeling effort in the context of supervised learning. To reduce computational time in training, as well as provide more convenient user interaction environment, it is necessary to select batches of new training examples instead of a single example. Batch mode active learning algorithms incorporate a diversity measure to construct a batch of diversified candidate examples. Existing approaches use greedy algorithms to make it feasible to the scale of thousands of data. Greedy algorithms, however, are not efficient enough to scale to even larger real world classification applications, which contain millions of data. In this paper, we present an extremely efficient active learning algorithm. This new active learning algorithm achieves the same results as the traditional greedy algorithm, while the run time is reduced by a factor of several hundred times. We prove that the objective function of the algorithm is submodular, which guarantees to find the same solution as the greedy algorithm. We evaluate our approach on several largescale real-world text classification problems, and show that our new approach achieves substantial speedups, while obtaining the same classification accuracy.

Introduction: In a perfect world, discovery would ideally be conducted by the senior litigator who is responsible for developing and fully understanding all nuances of their client’s legal strategy. Of course today we must deal with the explosion of electronically stored information (ESI) that never is less than tens-of-thousands of documents in small cases and now increasingly involves multi-million-document populations for internal corporate investigations and litigations. Therefore scalable processes and technologies are required as a substitute for the authority’s judgment. The approaches taken have typically either substituted large teams of surrogate human reviewers using vastly simplified issue coding reference materials or employed increasingly sophisticated computational resources with little focus on quality metrics to insure retrieval consistent with the legal goal. What is required is a system (people, process, and technology) that replicates and automates the senior litigator’s human judgment. In this paper we utilize 15 years of sensemaking research to establish the minimum acceptable basis for conducting a document review that meets the needs of a legal proceeding. There is no substitute for a rigorous characterization of the explicit and tacit goals of the senior litigator. Once a process has been established for capturing the authority’s relevance criteria, we argue that literal translation of requirements into technical specifications does not properly account for the activities or states-of-affairs of interest. Having only a data warehouse of written records, it is also necessary to discover the intentions of actors involved in textual communications. We present quantitative results for a process and technology approach that automates effective legal sensemaking.