Most techniques for relating textual information rely on intellectually created links such as author-chosen keywords and titles, authority indexing terms, or bibliographic citations. Similarity of the semantic content of whole documents, rather than just titles, abstracts, or overlap of keywords, offers an attractive alternative. Latent semantic analysis provides an effective dimension reduction method for the purpose that reflects synonymy and the sense of arbitrary word combinations. However, latent semantic analysis correlations with human text-to-text similarity judgments are often empirically highest at ≈300 dimensions. Thus, two- or three-dimensional visualizations are severely limited in what they can show, and the first and/or second automatically discovered principal component, or any three such for that matter, rarely capture all of the relations that might be of interest. It is our conjecture that linguistic meaning is intrinsically and irreducibly very high dimensional. Thus, some method to explore a high dimensional similarity space is needed. But the 2.7 × 107 projections and infinite rotations of, for example, a 300-dimensional pattern are impossible to examine. We suggest, however, that the use of a high dimensional dynamic viewer with an effective projection pursuit routine and user control, coupled with the exquisite abilities of the human visual system to extract information about objects and from moving patterns, can often succeed in discovering multiple revealing views that are missed by current computational algorithms. We show some examples of the use of latent semantic analysis to support such visualizations and offer views on future needs.

While team tasks provide a wealth of data on individual and team performance, techniques for modeling team communication can be quite effortful and time-consuming. Automated techniques of analyzing team discourse provide the promise of quickly judging team performance and permitting feedback to teams both in training and in operations. In previous research, techniques using Latent Semantic Analysis (LSA) have proven successful for analyzing team transcripts. However, converting the audio discourse into transcripts often requires hand transcription. In this work, we describe applying automated speech recognition (ASR) to team transcripts and using the output of the ASR to predict overall team performance. Results indicate that ASR can be used in conjunction with semantic methods of modeling team communication to provide accurate predictions of performance. The work has potential for assisting operators in the performance of their tasks because it can “listen” and in real-time evaluate free-form verbal communication from a variety of sources.

This study examines the hypothesis that the ability of a reader to learn from text depends on the match between the background knowledge of the reader and the difficulty of the text information. Latent Semantic Analysis (LSA), a statistical technique that represents the content of a document as a vector in high‐dimensional semantic space based on a large text corpus, is used to predict how much readers will learn from texts based on the estimated conceptual match between their topic knowledge and the text information. Participants completed tests to assess their knowledge of the human heart and circulatory system, then read one of four texts that ranged in difficulty from elementary to medical school level, then completed the tests again. Results show a nonmonotonic relation in which learning was greatest for texts that were neither too easy nor too difficult. LSA proved as effective at predicting learning from these texts as traditional knowledge assessment measures. For these texts, optimal assignment of text on the basis of either prereading measure would have increased the amount learned significantly.

This study examines the hypothesis that the ability of a reader to learn from text depends on the match between the background knowledge of the reader and the difficulty of the text information. Latent Semantic Analysis (LSA), a statistical technique that represents the content of a document as a vector in high-dimensional semantic space based on a large text corpus, is used to predict how much readers will learn from texts based on the estimated conceptual match between their topic knowledge and the text information. Participants completed tests to assess their knowledge of the human heart and circulatory system, then read one of four texts that ranged in difficulty from elementary to medical school level, then completed the tests again. Results show a nonmonotonic relation in which learning was greatest for texts that were neither too easy nor too difficult. LSA proved as effective at predicting learning from these texts as traditional knowledge assessment measures. For these texts, optimal assignment of text on the basis of either prereading measure would have increased the amount learned significantly.

LSA, a mathematical modeling technique, captures the essential relationships between text documents and word meaning, or semantics, the knowledge base which must be accessed to evaluate the quality of content. Several educational applications that employ LSA have been developed: (1) selecting the most appropriate text for learners with variable levels of background knowledge, (2) automatically grading the content of an essay, and (3) helping students effectively summarize material. Introduction Latent Semantic Analysis (LSA) is a mathematical/statistical technique for extracting and representing the similarity of meaning of words and passages by analysis of large bodies of text. It uses singular value decomposition, a general form of factor analysis, to condense a very large matrix of word-by-context data into a much smaller, but still largeÑtypically 100-500 dimensionalÑrepresentation (Deerwester, Dumais, Furnas, Landauer & Harshman, 1990). The right number of dimensions appears to be crucial; the best values yield up to four times as accurate simulation of human judgments as ordinary co-occurence measures. The similarity between resulting vectors for words and contexts, as measured by the cosine of their contained angle, has been shown to closely mimic human judgments of meaning similarity and human performance based on such similarity in a variety of ways. For example, after training on about 2,000 pages of English text it scored as well as average test-takers on the synonym portion of TOEFLÑthe ETS Test of English as a Foreign Language (Landauer & Dumais, 1997). After training on an introductory psychology textbook it achieved a passing score on a multiple-choice exam (Landauer, Foltz & Laham, in prep). LSA significantly improves automatic information retrieval by allowing user requests to find relevant text on a desired topic even when the text contains none of the words used in the query (Dumais, 1991, 1994). ¥For mathematical and computational details of the LSA method see Deerwester, et. al, 1990; Landauer & Dumais, 1997; Landauer, Foltz, & Laham, (in press). HCIC Ô98 BoasterÑEducational LSA Page 2 Text Selection ¥This section is extracted from Wolfe, Schreiner, Rehder, Laham, Foltz, Kintsch, & Landauer (in press). For additional information see Rehder, Schreiner, Wolfe, Laham, Landauer, & Kintsch (in press); Schreiner, Rehder, Landauer, & Laham (1997). This application is a result of an empirical examination of a theoretical relationship proposed by Kintsch (1994) in which the ability of a reader to learn from text is proposed to be dependent on the match between the background knowledge of the reader and the difficulty of the text information. LSA is used as a means of automatically predicting how much readers will learn from texts based on the estimated conceptual match between their knowledge of the topic and the information in the text they read. Participants in this study were given tests to assess their knowledge of the human heart and circulatory system, including questionnaires and open-ended essay questions before and after reading one of four relevant texts that ranged in difficulty from elementary (A) to medical school (D) level. Results show a nonmonotonic relationship in which learning was greatest for texts that were neither too easy nor too difficult. We call this the zone-of-learnability, or the ÒGoldilocks principleÓ. LSA proved just about as effective at predicting learning from these texts as traditional knowledge assessment measures. For these texts, optimal assessment of text to student on the basis of either pre-reading measure would have increased the amount learned significantly. Over all texts, the three measures of knowledge used hereÑthe pre-scores on the questionnaire (pre-questionnaire) and the grade on the pre-essay (pre-essay) and the cosine between the participants essay vector and Text C (cos essay.standard) were all quite highly correlated: r(pre-questionnaire : pre-essay) = .74, r(pre-questionnaire : cos essay.standard) = .68, and r(pre-essay : cos essay.standard) = .63, all p < .01. For comparison, the correlation between the two professional graders who scored the essays was r = .77. Thus, one can say that the LSA measure of knowledge is about as good as our questionnaire measures, and correlates with human graders almost as well as the human graders correlate among themselves. Amount of learning was operationally defined in two ways: as the proportion of possible improvement in the scores on the questionnaire from before to after reading (Learn-questionnaire), and as the proportion of possible improvement in the grades the student's essays received before and after reading (Learn-essay). The average cosine between the students' essays and the text they read can be used to predict the proportion improvement scores for the general knowledge test and the essay grades, Learn-questionnaire and Learn-essay. The data are shown in Figure 1 for the four groups of college students who read Texts A, B, C, and D, respectively, as well as for the medical students who read only Text A. The latter were included in this analysis because none of the texts was obviously too easy for the college students; to test the zone-of-learnability hypothesis we needed a group of learners who read a text that was clearly too easy for them. The curves fitted to the points in Figure 1 are second order polynomials. The zone-of-learnability hypothesis is supported most clearly by a plot of the average learning scores for the HCIC Ô98 BoasterÑEducational LSA Page 3 four texts as a function of the average cosine between the students' essays and the text they read. 0 0.1 0.2 0.3 0.4 0.5 Le ar ni ng S co re s

In another article (Wolfe et al., 1998/this issue) we showed how Latent Semantic Analysis (LSA) can be used to assess student knowledge—how essays can be graded by LSA and how LSA can match students with appropriate instructional texts. We did this by comparing an essay written by a student with one or more target instructional texts in terms of the cosine between the vector representation of the student's essay and the instructional text in question. This simple method was effective for the purpose, but questions remain about how LSA achieves its results and how the results might be improved. Here, we address four such questions: (a) What role does the use of technical vocabulary play? (b) how long should the student essays be? (c) is the cosine the optimal measure of semantic relatedness? and (d) how does one deal with the directionality of knowledge in the high‐dimensional space?

Singular value decomposition (SVD) can be viewed as a method for unsupervised training of a network that associates two classes of events reciprocally by linear connections through a single hidden layer. SVD was used to learn and represent relations among very large numbers of words (20k-60k) and very large numbers of natural text passages (1k-70k) in which they occurred. The result was 100-350 dimensional "semantic spaces" in which any trained or newly added word or passage could be represented as a vector, and similarities were measured by the cosine of the contained angle between vectors. Good accuracy in simulating human judgments and behaviors has been demonstrated by performance on multiple-choice vocabulary and domain knowledge tests, emulation of expert essay evaluations, and in several other ways. Examples are also given of how the kind of knowledge extracted by this method can be applied.

Latent Semantic Analysis (LSA) is a theory and method for extracting and representing the contextual‐usage meaning of words by statistical computations applied to a large corpus of text (Landauer & Dumais, 1997). The underlying idea is that the aggregate of all the word contexts in which a given word does and does not appear provides a set of mutual constraints that largely determines the similarity of meaning of words and sets of words to each other. The adequacy of LSA's reflection of human knowledge has been established in a variety of ways. For example, its scores overlap those of humans on standard vocabulary and subject matter tests; it mimics human word sorting and category judgments; it simulates word‐word and passage‐word lexical priming data; and, as reported in 3 following articles in this issue, it accurately estimates passage coherence, learnability of passages by individual students, and the quality and quantity of knowledge contained in an essay.

Many computational models of semantic memory rely on vector representations of concepts based on explicit encoding of arbitrary feature sets. Latent Semantic Analysis (LSA) creates high dimensional (n = 300+) vectors for concepts in semantic memory through statistical analysis of a large representative corpus of text rather than subjective feature sets linked to object names (for details see Landauer & Dumais, 1997; Landauer, Foltz, & Laham, in press). Concepts can be compared in the semantic space and their similarity indexed by the cosine of the angle between vectors. Computational models of concept relations using LSA representations demonstrate that categories can be emergent and self-organizing based exclusively on the way language is used in the corpus without explicit hand-coding of category membership or semantic features. LSA categorization is context dependent and occurs through a dynamic process of induction. Semantic “meaning” is not encapsulated within an object representation, but emerges as the set of relationships between selected objects in a context-based sub-space. Neuropsychological studies (e.g. Warrington & Shallice, 1984) point to a class of patients who exhibit disnomias for specific categories of objects (natural kinds) while retaining the ability to name other objects (man-made artifacts). The objects from natural kind categories tend to be significantly more clustered in LSA space than are those from artifact categories. If brain structure corresponds to LSA structure, the identification of concepts belonging to strongly clustered categories should suffer more than weakly clustered concepts when their representations are partially damaged. Three types of modeling experiments were conducted: matching base concept names to superordinate categories in forced-choice testing, correlating LSA similarity measures to human judgments of typicality, and multivariate analyses of similarity matrices to capture category boundaries. For the forced-choice matching of concept names to superordinate categories, a selection of 140 objects (rated as most typical in their category) from 14 categories was used. Each object name was compared to each of the 14 category names (apple—flower, apple—mammal, etc.). The LSA match was considered correct when the highest cosine comparison in the set was between an object and its relevant superordinate (apple—fruit). The results show that in all 14 categories, LSA predicts membership well above chance (chance = 7%), however, there are differences in the degree of clustering: the percent correct for animate natural kinds (flowers, mammals, fruit, trees, vegetables, and birds) = 92%; for inanimate natural kinds with observed deficits in neuropsychological patients (gemstones, musical instruments) = 100%; and for man-made artifacts (furniture, vehicles, weapons, tools, toys, and clothing) = 53%. Correlations between LSA similarity judgments and human typicality judgments were consistently better for the natural kinds than for the artifacts. For natural categories, LSA similarities (cosine between concept and either superordinate name, most typical member, or centroid of all members) showed high correlations with human judgments (e.g. fruit: r = .82), while artifact similarities showed low to near-zero correlations with human judgments. As illustrated in Figure 1, multivariate analyses of LSAbased similarity matrices show more cohesive structure for natural kinds than for artifacts. Factors 4-6 in this analysis load high on concepts in the bird category—additional factors (7-15) load on specific artifact concepts (not shown).

How much of the meaning of a naturally occurring English passage is derivable from its combination of words without considering their order? An exploratory approach to this question was provided by asking humans to judge the quality and quantity of knowledge conveyed by short student essays on scientific topics and comparing the interrater reliability and predictive accuracy of their estimates with the performance of a corpus-based statistical model that takes no account of word order within an essay. There was surprisingly little difference between the human judges and the model. In the studies reported here, experts were asked to read short student essays about scientific topics with the goal of determining how much knowledge was accurately reflected in a given essay. We measured the readers’ success by how well their ratings agreed with each other and how well they predicted scores on an objective test on the same subject. All current accounts of human discourse understanding

This paper reviews two current Air Force Research Laboratory / Human Effectiveness Directorate (AFRL/HEA) efforts that are maturing Latent Semantic Analysis (LSA) tools for the Air Force. The first effort is developing new LSA-based agent software that helps decision makers to identify required job knowledge, determine which members of the workforce have the knowledge, pinpoint needed retraining content, and maximize training and retraining efficiency. Modern organizations are increasingly faced with rapid changes in technology and missions and need constantly changing mixes of competencies and skills. Assembling personnel with the right knowledge and experience for a task is especially difficult when there are few experts, unfamiliar devices, redefined goals, and short lead-times for training and deployment. LSA is being used to analyze course content and materials from current training pipelines and to identify appropriate places in alternative structures where that content can be reused. This saves time for training developers since the preexisting content has already been validated as a part of its earlier application. AFRL/HEA's second research effort involves a demonstration of a combined speech-to-text and LSA-based software agent for embedding automatic, continuous, and cumulative analysis of verbal interactions in individual and team operational environments. The agent will systematically parse and evaluate verbal communication to identify critical information and content required of many of today's AF operators. LSA is promising new technology that has significant potential for assisting operators in the performance of their tasks because it can "listen" and in almost real-time evaluate free-form verbal communication from a variety of sources and match content to stored language dictionaries. One application of this technology being explored is tracking and scoring the tactical communications that occur between the members of a four-ship air combat flight and their weapons director to identify areas of training need and as an additional tool for assessing the efficacy of DMT scenarios and missions.