The evaluation of scientific collaboratories has lagged behind their development, and fundamental questions have yet to be answered: Can distributed scientific research produce high quality results? Do the capabilities afforded by collaboratories outweigh their disadvantages from scientists’ perspectives? Are there system features and performance characteristics that are common to successful collaboratory systems? Our goal is to help answer such fundamental questions by evaluating a specific scientific collaboratory system called the nanoManipulator Collaboratory System. The system is a set of tools that provide collaborative interactive access to a specialized scientific instrument and office applications. To evaluate the system, we conducted a repeated-measures controlled experiment that compared the outcomes and process of scientific work completed by 20 pairs of participants (upper level undergraduate science students) working face-to-face and remotely. We collected scientific outcomes (graded lab reports) to investigate the quality of scientific work, post-questionnaire data to measure intentions to adopt the system, and post-interviews to understand the participants’ views of doing science under both conditions. We hypothesized that study participants would be less effective, report more difficulty, and be less favorably inclined to adopt the system when collaborating remotely. However, the quantitative data showed no statistically significant differences with respect to effectiveness and adoption. Furthermore, in post-interviews participants reported advantages and disadvantages working under both conditions but developed work-arounds to cope with the perceived disadvantages of collaborating remotely. A theoretical explanation for the results can be found in the theory of the life-world (Schutz & Luckman, 1973, 1989). Considered as a whole, the analysis leads us to conclude there is positive potential for the development and adoption of scientific collaboratory systems.

Interdisciplinary collaboration occurs when people with different educational and research backgrounds bring complementary skills to bear on a problem or task. The strength of interdisciplinary scientific research collaboration is its capacity to bring together diverse scientific knowledge to address complex problems and questions. However, interdisciplinary scientific research can be difficult to initiate and sustain. We do not yet fully understand factors that impact interdisciplinary scientific research collaboration. This study synthesizes empirical data from two empirical studies to provide a more comprehensive understanding of interdisciplinary scientific research collaboration within the natural sciences in ac ademia. Data analysis confirmed factors previously identified in various literatures and yielded new factors. A total of twenty factors were identified, and classified into four categories: personal, resources, motivation and common ground. These categorie s and their factors are described, and implications for academic policies and practices to facilitate and sustain interdisciplinary collaboration are discussed.

Interdisciplinary collaboration occurs when people with different educational and research backgrounds bring complementary skills to bear on a problem or task. The strength of interdisciplinary scientific research collaboration is its capacity to bring together diverse scientific knowledge to address complex problems and questions. However, interdisciplinary scientific research can be difficult to initiate and sustain. We do not yet fully understand factors that impact interdisciplinary scientific research collaboration. This study synthesizes empirical data from two empirical studies to provide a more comprehensive understanding of interdisciplinary scientific research collaboration within the natural sciences in ac ademia. Data analysis confirmed factors previously identified in various literatures and yielded new factors. A total of twenty factors were identified, and classified into four categories: personal, resources, motivation and common ground. These categorie s and their factors are described, and implications for academic policies and practices to facilitate and sustain interdisciplinary collaboration are discussed.

The goal of the Panel is to explore the nature and role of scientific collaboration from differing perspectives, for example, from scientometrics-informetrics, sociology of science, and social network analysis. The panel presents two separate but linked sessions with papers describing and advocating new methods and approaches to research design followed by papers that have applied these methods to good effect. Scientific collaboration is frequent and an important aspect of the ongoing development of science. Scientometric methods alone cannot provide the full picture; they must be combined with other approaches particularly when causal mechanisms are to be investigated. To understand the contribution of collaboration, it is crucial to find ways to link data about social networks across fields, organizations and countries with scientometric and bibliometric indicators of collaboration, such as co-authorships, citations, and institutional affiliations. Data relating to differences among disciplines, about the influence of research policies, research support and funding mechanisms for science are important aspects that should be taken into account and linked to scientometric and social indicators. Social relationships among peers and colleagues, the building of social networks within institutions and their staff, also need to be integrated into analyses. To cover this breadth of work, the Panel is divided into two separate sessions. The first includes four papers advocating new methods and approaches to research design with discussion focussed on how far quantitative methods can take us in the exploration of scientific collaboration and on ways of integrating qualitative evidence from the social aspects of collaborative work. The second follows up that discussion by presenting a further four research papers that have used innovative designs and which provide new insights into collaboration from various standpoints. ASIST's overall theme of managing and enhancing conflict and culture is encapsulated in this Panel, with its focus on the culture and ambiguities between disciplines and fields, between quantitative and qualitative approaches, and between individual and institutional factors mat drive cooperation as well as competition in research. Participants engaged in research on collaboration, whether in science or other arenas, would learn from this dual session format. First, participants will learn by hearing about the range of methods currently being applied to the study scientific collaboration, and secondly, from discussions on issues related to application of various integrative approaches and of specific methods between panel speakers and members of the audience. Part 1: Methodology for Investigating Collaboration Methodological Issues and Problems in Investigating Scientific Collaboration Henry Small, Keynote Speaker The collaboration between public science and technology (S&T) organizations and industrial companies has been the subject of considerable research. Although we now have a good understanding of the process of collaboration and technology transfer between these two types of institutions, we lack a solid and acceptable methodology to measure the outputs and the performance of such collaboration. A consistent problem with the methodology seems to be the lack of adequate quantitative measures that could provide a good assessment of the success of the collaboration. The usual metrics of scientific and technical performance, such as cooperation agreements, bibliometrics and patents provide a partial and sometimes even a biased evaluation of the collaborative effort. This paper explores the current state of metrics used in the evaluation of government-industry collaboration, and suggests a method and the metrics to improve the state of this effort. The collaboration and technology transfer or commercialization of the vast network of American public S&T organizations with private industry, are used as an illustration of the proposed metrics. The link to national goals and mission objectives of the S&T organizations is also considered as a force in determining the adequacy of metrics of the collaboration with industry. This paper will focus on the nature of scientific collaboration as structural-hole filling agents in knowledge diffusion and the evolution of scientific collaboration networks, as primarily manifested as co-author networks. Progressive knowledge domain visualization techniques provide one of the promising means to improve our understanding of some of the underlying factors that may influence scientific collaboration from this particular perspective. The central hypothesis is drawn upon the social structure of competition in social network research, namely Hurt's theory of structural holes. The hypothesis states that the greater the structural gap in the knowledge base, the more significant the outcome is likely to be. One can derive a number of auxiliary hypotheses that can be tested with the help of visualizing the progression of temporal patterns in scientific collaboration networks associated with structural holes of high-risk and high-return. The cost of initiating and maintaining scientific collaboration can be measured in many ways, for example, in terms of how far apart the collaborators are geographically, for example, same institution, same town, same country, and whether they are in the same discipline. The effect of scientific collaboration, similarly, can be measured in a wide variety of ways, such as the total number of publications, citations of joint publications, the amount of research grants awarded, the number of patents issued, the number new researchers recruited, and national and international research ranking. Studying scientific collaboration in this way can help us to address both theoretical and methodological issues. Mapping the evolution of various factors involved in scientific collaboration from a structural whole point of view is expected to contribute to the study of scientific collaboration in general. As a potentially widely usable method, an easy understanding and interpreting dynamical properties of structural holes may foster further collaboration. Open access journals are gaining popularity in recent years despite uncertainties in funding for many of them. Currently, there are 702 open access journals according to the Directory of Open Access Journals (http://www.doaj.org), among which 431 (61%) titles are in scientific and engineering disciplines. Advocates strongly believe that open access journals promise a whole new venue for scientific and scholarly communication and will have profound impact on the way scientists communicate and collaborate in research. However, what are the appropriate ways to measure the impact of open access journals? Will the traditional citation data alone be sufficient for such assessment? If not, what other data are needed and how can they be collected? This presentation attempts to address these questions from a methodological perspective. Research collaboration in science has been measured by citation mapping and co-authorship, and the impact of journals by the ISI JIF. To assess the impact of open access journals on research collaboration would require more than the sum of citation mapping, co-authorship, and journal impact factor. The forms and output of collaborative scientific research in an open access journal environment are no longer limited only to published research papers; scientists communicate and interact via comment areas, author email accessibility, research data and results, and many other channels made available within open access journal systems. By examining the dimensions and relationships of scientific collaboration and open access journals, the speakers discuss the scientometric and non-scientometric measures for open access journals' quality and accessibility and the variables for assessing the impact of open access journals on scientific collaboration, the data collection challenges, and the pitfalls in data interpretation. Our presentation addresses problems of measuring social processes that underpin scientific collaboration. We start by briefly reviewing the social processes underlying group formation and cohesion as an introduction to our study on the qualitative dimensions of collaboration related to group productivity. Cohesiveness represents the degree of attraction of members to a group and its coherence. Ridgeway (1983) defined it as ‘the extent to which features of the group bind the members to it’. We assume when group size exceeds two that social processes are more complex and that larger groups will have clear demarcation of specialized tasks, with specific tasks within the group. We also assume that no more than two authors are likely to be responsible for intellectual functions associated with the research endeavor while the rest of the team will perform technical and other specific functions. The fact that integration of science groups involves both consensus and cohesion could also lead us to determine the direction of influence within each group, and a clarification of the decisions made with respect to name ordering in published papers. Consensus is the convergence of a set of agreements and commitments between members of a group during interaction resulting in a process of identity transformation. Convergence of agreements and commitments is established by the perception of covariation of different properties, relations and positions from the members of the group, between them and in relation to the group task. (Liberman, 1983). In order to test our assumptions we take Web of Science co-authorship data as our starting point, for the selection of groups of more than two co-authors working at the National University of Mexico (UNAM) in two areas: Physics, and Biotechnology, because, in these disciplines, it is assumed that more specialized tasks are performed by the research group. The members of the selected groups in the present study will be subjected to a semi-structured group interview to clarify the distinct tasks and roles they fulfill within the group. Results will undergo content analysis. Our recent work has established a relationship between co-authorship, as a measure of collaboration in scientific groups, and group cohesiveness in these same groups of scientists. Cohesion was achieved more as a result of the identification of group members with the task assigned to the group rather than with the group itself or with the other members. (Lima, Liberman and Russell 2004). The ultimate goal of this research, which falls within the new field of the Social Psychology of Science, is to define the tasks and roles of the different members of scientific groups and relate this to name order of co-authors and to the frequency of their co-authored publications. Part 2: Research Papers —Collaboration in Action Six case studies of international cooperation at the subfield level are presented and compared. The cases examine international collaboration by detailing co-authorship links among researchers by field, evidenced at the level of the nation. Cases are offered based on possible drivers for collaboration: sharing theory, cooperating around equipment, cooperating around resources, and sharing data. Scientometric and network analysis of linkages are presented and discussed for each of the six cases: astrophysics, geophysics, mathematical logic, polymers, soil science, and virology. Visualizations of the cosine networks within each field are compared for 1990 and 2000. International collaboration grew in all the fields at rates higher than the international average. The possibility that rapid increases in international collaboration in science can be attributed in part to certain drivers related to access to resources or shared equipment is not upheld by the data. Other possible explanations for the rapid growth of collaboration are offered, including the possibility that weak ties evidenced by geographically remote collaboration can better promote new knowledge creation than side-by-side collaboration. Interdisciplinary scientific collaborations, that include scientists from two or more disciplines, are increasingly necessary to address complex problems. Furthermore, as problems increase in complexity, interdisciplinary collaborations may become increasingly diverse. To encourage and facilitate interdisciplinary collaboration, there is a need to better understand which factors are most critical to its success. This talk will discuss factors that appear to impact academic inter-disciplinary scientific collaboration and the different roles the factors play in inter- and intradisciplinary collaboration. Interviews were conducted with 24 scientists who had extensive experience in both inter- and intra-disciplinary collaboration in academic settings. Analysis of the interview data suggests factors that impact collaboration can be grouped into five categories based on theoretical frameworks proposed by Ranganathan (1957) and Sonnenwald and livonen (1999). The categories are: personality (participants' abilities, preferences, feelings, perceptions and relationships); matter (the tangible such as physical objects or output, including funding agency support, institutional support; existing publications; opportunities for publication; and students/human resources); energy (action and factors that cause action, including communication and motivations); space/location (where work or communications may take place); and time (period in which the collaboration occurred.) Interview data suggests that these factors are both facilitators and barriers to collaboration, and they have a greater impact on interdisciplinary collaborations than on intradisciplinary collaborations. For example, a majority of the participants indicated that, unlike for intra-disciplinary collaboration, their institutions do not provide funding support for interdisciplinary collaboration and institutional practices, such as inter-departmental politics, made interdisciplinary collaboration more difficult than intradisciplinary collaboration. These results may help inform science policy as well as scientific practice and education. The extent of collaborative papers will increase by some ratio in each subsequent five-year period. That in each of the five-year periods, the number of collaborative authors increases by some ratio. That there will always be a gap between the many national (internal to a country or state) collaborations to the fewer international (external to state or country). The number of papers and authors increases by some proportion, yet each country's overall percentage contribution to world literature hardly shifts. The project will examine time-dependent aspects of science activities as expressed in papers from scholarly research journals on the subject of ophthalmology for the past 30 years (1974 to 2003). Data for the study come from the ISI Science Citation Index. The issue for research collaborators is how to share and/or pool knowledge toward a common goal among people with diverse backgrounds. It is not sufficient to say that exchange must take place; the question is also what kind of exchanges forms the basis for collaborative groups, and with whom. Is there something different about interdisciplinary than intra-disciplinary collaboration? Do needs and exchanges differ by the role of individuals in collaborative teams, e.g., novices and experts, students and senior researchers, principal investigators and team members? Do different teams develop their own patterns of interaction, or is there some commonality across teams? To explore these questions, social network data on work and learning activities were collected from members of three interdisciplinary teams. Team members were asked what they learned from the 5–7 others with whom they worked most closely, and what they thought others learned from them. Answers to these questions describe social networks of learning among the interdisciplinary researchers through a mapping of both in-team and out-team interactions with closest sets of co-workers. Qualitative answers to the question of “What sorts of things did you learn from them?” (or “from you”) provide a more in-depth look at what kinds of exchanges support collaborative teams. Results show that exchange of factual knowledge, accounting for only one-quarter of exchanges, is only one of a number of learning exchanges that support such teams. Other important exchanges include: learning the process of doing something; information about methods; engaging jointly in research; learning about a technology; generating new ideas; socialization into the profession; access to a network of contacts; administration work; and social support. Knowledge about process and methods figured prominently in what is reported as learned. Examining the networks of learning exchanges reveal patterns of learning exchanges and show how knowledge circulates in the network, and between whom, For example, in one group a triad of theory-simulation-data is evident: theory from one team member is used in creating models and computer programs that run simulations by others, while those who run simulations also build ties with those who collect data in order to evaluate the simulations. The data indicate the range and multiplexity of exchanges maintained by individuals, and in particular, give information about the nature and distribution of interactions with other team members as well as with people outside the team. Networks of “typical” team members and/or members with particular roles are shown also by the data.

When collaborating, individuals rely on situation awareness (the gathering, incorporation and utilization of environmental information) to help them combine their unique knowledge and skills and achieve their goals. When collaborating across distances, situation awareness is mediated by technology. There are few guidelines to help system analysts design systems or applications that support the creation and maintenance of situation awareness for teams or groups. We propose a framework to guide design decisions to enhance computer-mediated situation awareness during scientific research collaboration. The foundation for this framework is previous research in situation awareness and virtual reality, combined with our analysis of interviews with and observations of collaborating scientists. The framework suggests that situation awareness is comprised of contextual, task and process, and socio-emotional information. Research in virtual reality systems suggests control, sensory, distraction and realism attributes of technology contribute to a sense of presence [Presence 7 (1998) 225]. We suggest that consideration of these attributes with respect to contextual, task and process, and socio-emotional information provides insights to guide design decisions. We used the framework when designing a scientific collaboratory system. Results from a controlled experimental evaluation of the collaboratory system help illustrate the framework's utility.

The evaluation of scientific collaboratories has lagged behind their development. Do the capabilities afforded by collaboratories outweigh their disadvantages? To evaluate a scientific collaboratory system, we conducted a repeated-measures controlled experiment that compared the outcomes and process of scientific work completed by 20 pairs of participants (upper level undergraduate science students) working face-to-face and remotely. We collected scientific outcomes (graded lab reports) to investigate the quality of scientific work, post-questionnaire data to measure the adoptability of the system, and post-interviews to understand the participants' views of doing science under both conditions. We hypothesized that study participants would be less effective, report more difficulty, and be less favorably inclined to adopt the system when collaborating remotely. Contrary to expectations, the quantitative data showed no statistically significant differences with respect to effectiveness and adoption.The qualitative data helped explain this null result: participants reported advantages and disadvantages working under both conditions and developed work-arounds to cope with the perceived disadvantages of collaborating remotely. While the data analysis produced null results, considered as a whole, the analysis leads us to conclude there is positive potential for the development and adoption of scientific collaboratory systems.

Scientific collaboratories have the potential to be centers "without walls, in which researchers can perform their research without regard to physical location – interacting with colleagues, accessing instrumentation, sharing data and computational resources, and accessing information in digital libraries" [Wulf, W.A. (1993). The collaboratory opportunity. Science, 261, 854-855]. A number of scientific collaboratories have been developed, and a comprehensive discussion of these collaboratories can be found in Finholt, T. (2002). Collaboratories, In B. Kronin (Ed.) Annual Review of Information Science and Technology. Washington, DC: American Society for Information Science and Technology. However, the evaluation of scientific collaboratories has lagged behind their development. So few evaluations of scientific collaboratories exist that fundamental questions regarding their potential have yet to be answered: Can distributed scientific research produce high quality results? Do the capabilities afforded by collaboratories outweigh their disadvantages from scientists' perspectives? How does the scientific process change in the context of a collaboratory? The answers to these questions are not obvious. Previous research in computer-supported cooperative work [e.g., Olson, G.M. & Olson, J.S. (2000). Distance matters. Human-Computer Interaction, 15 (2-3), 139-178] and theory of language [Clark, H. (1996). Using Language. Cambridge, UK: Cambridge University Press] would predict that working remotely would lack the richness of collocation and face-to-face interaction such as multiple and redundant communication channels, implicit cues and spatial co-references, which are difficult to support via computer-mediated communications. This lack of richness is thought to impair performance because it is more difficult to establish the common ground that enables individuals to understand the meaning of each other's utterances. The collaboratory system we evaluated provides distributed, collaborative access to a specialized scientific instrument called a nanoManipulator (nM). The single-user nM provides haptic and 3D visualization interfaces to a local (co-located) atomic force microscope (AFM), providing a natural scientist with the ability to interact directly with physical samples ranging in size from DNA to single cells. The nM and its uses are described in Guthold, M., Falvo, M.R., Matthews, W.G., Paulson, S., Washburn, S., Erie, D.A., Superfine, R., Brooks, Jr., F.P., & Taylor, III, R.M.(2000). Controlled Manipulation of Molecular Samples with the nanoManipulator. IEEE/ASME Transactions on Mechatronics, 5(2), 189-198. The collaboratory version of the nM was designed based on results of an ethnographic study from which we developed an understanding of the scientific research process, current collaborative work practices, the role of an nM as a scientific instrument and scientists' expectations regarding technology to support scientific collaborations [Sonnenwald, D.H., Bergquist, R., Maglaughlin, K.L., Kupstas-Soo, E. & Whitton, M.C. (2001). Designing to support collaborative scientific research across distances: The nanoManipulator example, In E. Churchill, D. Snowdon, &A. Munro, (Eds.), Collaborative Virtual Environments (pp. 202-224). London: Springer Verlag]. One PC is equipped with a Sensable Devices Phantom force-feedback device. This PC and its associated software provide haptic and 3D visualization interfaces to a local or remote atomic force microscope (AFM) and support collaborative manipulation and exploration of scientific data. Scientists can dynamically switch between working together in shared mode and working independently in private mode. In shared mode, remote, that is, non-collocated, collaborators view and analyze the same (scientific) data. Mutual awareness is supported via multiple pointers, each showing the focus of attention and interaction state for one collaborator. Collaborators can perform almost all operations synchronously. Because of the risk of damage to an AFM, control of the microscope tip is explicitly passed between collaborators. In private mode, each collaborator can independently analyze the same or different data from stream files previously generated. When switching back to private from shared mode, collaborators return to the exact data they were previously using. Another PC supports shared application functionality and video conferencing (via Microsoft NetMeeting) and an electronic writing/drawing tablet. This PC allows collaborators to work together synchronously using a variety of domain-specific and off-the-shelf applications, including specialized data analysis, word processing and whiteboard applications. Video conferencing is supported by two cameras. One camera is mounted on a gooseneck stand so it can be pointed at the scientist's hands, sketches or other physical artifacts scientists may use during experiments; the other is positioned to capture a head and shoulders view of the user. Collaborators have software control of which camera view is broadcast from their site. A wireless telephone headset and speakerphone connected to a commercial telephone network provides high quality audio communications for collaborators. The experimental evaluation study was a repeated-measures, or within-subjects, controlled experiment comparing working face-to-face and working remotely with the order of conditions counterbalanced. Twenty pairs of study participants (upper level undergraduate natural science students) conducted two realistic scientific research activities each requiring two to three hours to complete. Ten pairs of study participants worked face-to-face first and, on a different day, worked remotely (in different locations.) Another 10 pairs worked remotely first and, on a different day, face-to-face. When face-to-face, the participants shared a single collaboratory system; when collaborating remotely, each location was equipped with its own complete collaboratory system. The scientific research activities completed by the participants were designed in collaboration with natural scientists. The tasks were actual activities the scientists completed and documented during the course of their investigations. To complete the tasks the participants had to engage in the following activities typical of scientific research: operate the scientific equipment properly; capture and record data in their (electronic) notebooks; perform analysis using scientific data analysis software applications and include the results of that analysis in their notebooks; draw conclusions, create hypotheses and support those hypotheses based on their data and analysis; and prepare a formal report of their work. We collected a variety of quantitative and qualitative evaluation data, including task performance measures to compare the quality of scientific work produced in the two collaboration conditions, and post-interviews to gain, from participants' perspectives, a more in-depth understanding of the scientific process in both conditions. Task performance was measured through graded lab reports. The information participants were asked to provide in the reports mirrored the information found in the scientists' lab notes created when they conducted their original research. Each pair of study participants collaboratively created a lab report under each condition, generating a total of 40 lab reports; 20 created working remotely and 20 created working face-to-face. The lab reports were graded blindly; the graders had no knowledge of the report authors or under which condition the report was created. To further our understanding of participants' perceptions of the system, we conducted semi-structured interviews with each participant after each task. Study participants were asked what they thought about their experience, including the most satisfying and dissatisfying aspects of their experience. In addition, we inquired about work patterns that emerged during the experience, and the impact technology may have had on their interactions with their collaborator. After completing their second task, participants were also asked to compare working face-to-face and working remotely. To better learn each participant's perspective, participants were interviewed individually, for a total of 80 interviews, each lasting from 30 to 60 minutes. Task Performance: Analysis of Graded Lab Reports. The average lab report scores for the first task session were identical (70/100) for both the face-to-face and remote condition. Previous research would predict that scores from a remote first session would be lower because the remote session would lack the richness of collocation and face-to-face interaction, including multiple and redundant communication channels, implicit cues and spatial co-references, that are difficult to support via computer-mediated communications. This lack of richness is often thought to impair performance. Perhaps technical features such as seeing your partner's pointer and functions, optimized shared control of scientific instrumentation and applications, improved video that provides multiple views and high quality audio communications may be "good enough" for scientific tasks focusing on collecting, analyzing and interpreting data. The data further suggest that collaborating first remotely may have a positive effect. Using a multivariate analysis of variance (MANOVA) test, the differences in scores for the face-to-face and remote conditions were not statistically significant. However, when order is taken into account, participants who collaborated remotely first scored significantly higher on the second task than did those who collaborated face-to-face first. There was no statistically significant difference between face-to-face and remote lab scores for participants who collaborated face-to-face first. In general, the literature suggests that participants would learn more about the system, science and each other when collaborating face-to-face and that this knowledge helps increase their current and future performance. Our performance data suggest collaborating first remotely does not negatively impact current performance, and may positively impact future performance for scientific tasks such as data collection, analysis and interpretation. We looked to our interview data for explanations of this result. Participants' Perceptions of the Scientific Process: Post-Interview Analysis. As expected, participants reported disadvantages to collaborating remotely. However, participants also reported that some of these disadvantages are not significant in scientific work contexts and that coping strategies, or work-arounds, can reduce the impact of other disadvantages. Furthermore, participants reported that remote collaboration provided several relative advantages compared with face-to-face collaboration (see Table 1). Similar to previous studies [e.g., Olson, G.M. & Olson, J.S. (2000). Distance matters. Human-Computer Interaction, 15 (2-3), 139-178], study participants reported remote collaboration was less personal than face-to-face collaboration. When comparing working face-to-face and remotely, participants reported collaborating face-to-face was more personal and it was easier to express themselves. However, participants also reported that lacking this type of interaction when working remotely did not seem to have a negative impact on their work. The impersonal nature of remote collaboration increased their productivity and facilitated collaborative intellectual contributions. As participants explained: If we were. . . working side by side, we might tell more stories or something like that. . . . [However] if you're trying to get something done, sometimes the stories and stuff can get in your way. I think that being in separate rooms helps a little bit because it's more impersonal. . . [You] just throw stuff back and forth more easily. Participants also reported that when working remotely they received fewer implicit cues about what their partners were doing and thinking. The study participants explained that without these cues, it may be difficult to follow social interaction norms and assist your collaborators: [when collaborating face to face] it was a lot easier to ask questions of each other. . . since you have a feeling [about] when to interrupt them . . . if you're in the same room . . . you'll wait [to ask a question] until the other person is not doing as much or not doing something very specific. It is hard to get the context of any question that's asked because you're not paying attention to what the other person is doing because they're in a little [video-conferencing] screen. To compensate for this lack of cues, several participants reported they needed to talk more frequently and descriptively when collaborating remotely. Participants reported Even though we were in separate rooms, it kind of seemed like there was more interaction compared to being face-to-face, which seems kind of strange. . . . It just seemed more interaction was expected. . . . Maybe needed. We had a really good interaction [when collaborating remotely]. . . . You're conscious that you're not together and you can't see [some things, and] so you think more about [interacting. For example, you think] 'I need to let this person know that I'm about to do this' or 'this is what I'm seeing and I'm trying to let you know so, and you're like doing the same to me.' Thus to compensate for the absence of implicit cues in the remote condition participants provided explicit cues for their partner. When working remotely, it appears that individuals recognize they do not have a common shared physical reality and subsequently may not have a shared cognitive reality. However, humans are intrinsically motivated to develop a shared reality [Schutz, A., & Luckmann, T. (1983). The Structures of the Life-World, Vol. I. Evanston, IL: Northwestern University Press]. Subsequently, study participants developed a strategy, providing explicit cues to their partners, to develop a shared reality. These explicit cues, or joint actions, typically contribute to faster and more accurate formation of common ground and mutual understanding [Clark, H. (1996). Using Language. Cambridge, UK: Cambridge University Press]. In addition to receiving fewer cues from a partner, participants also reported that some physical tasks are more difficult when collaborating remotely. These tasks include drawing, e.g., creating sketches of scientific structures and manipulating mathematical equations, and sharing control of applications within NetMeeting. Some of these problems may be remedied with advances in technology, such as shared applications that support multiple pointers and concurrent floor control. Participants explained [when collaborating face to face] you could draw more easily, communicate diagrams more easily, and you could look at the other person and see their level of understanding more easily. The thing that frustrated me the most [collaborating remotely] was the shared applications [NetMeeting] . . . you could see the other person doing things but you couldn't do anything [simultaneously]. Although technology made some tasks more difficult, study participants also reported that the collaboratory system provides some advantages over collaborating face-to-face. These advantages include the ability to easily explore the scientific instrument and data and their own ideas both independently and collaboratively, having identical views of the data visualization, and working simultaneously with the data visualization. I liked that we were separate. I think it gave a whole new twist on the interactions, and if one of us got snagged up with something the other could independently work and get it done rather than both of us being bogged down by having to work on it simultaneously. Sometimes when you're working side by side with somebody, you have to deal with 'Well, you're looking at [the data] from a different angle than I am, and so you're seeing a different perspective there.' Now [working remotely] we could both of us be straight on, having the exact same perspective from where we're sitting. It made it easier. [My partner] could be changing the light focusing somewhere, while I could be zooming or moving [the plane] around. And that was really helpful because you're thinking, 'OK, as soon as I'm done moving the light I want to go ahead and shift [the plane] . . . [to be able to] say to [my partner], 'Why don't you [shift the plane] while I'm shining the light,' was really cool. It was really helpful. The participants in this study experienced disadvantages attributed to remote collaboration that have been previously reported in the literature. However, the study participants also reported that some disadvantages had minimal impact on their scientific work, and they used coping strategies to compensate for disadvantages. In addition, they perceived remote collaboration to provide some advantages relative to face-to-face collaboration. These findings corroborate our findings regarding task performance. The results illustrate the potential of collaboratories, allowing individuals who do not have specialized, state-of-the-art scientific instruments locally to access scientific instruments remotely and conduct scientific experiments in collaboration with other students, faculty and staff at institutions that have the instruments. When collaborating first remotely, study participants were able to use the technology and conduct science as well as or better than if they were face-to-face. Working face-to-face before working remotely did not produce the anticipated positive impact on the scientific process or outcomes. These conclusions are supported by the similarity in lab report grades for the first task session and statistically significant higher lab report grades for pairs who first worked remotely, as well as interview data that illustrate that the technology provides unique advantages and, although the technology has disadvantages, many individuals can develop coping strategies to reduce the impact of these disadvantages. The evaluation data combine to illustrate the potential of the collaboratory system for adoption by scientists and how this technology mediates collaborative scientific work processes without negatively impacting scientific data collection and analysis task outcomes. However, the tasks used in the study do not encompass the entire life cycle of the scientific process. For example, problem formulation, research design and research dissemination were not included in the tasks. Furthermore, the tasks in the first and second sessions differed. Although designed to be similar in complexity, additional investigation may uncover aspects of the tasks that are inherently impacted by an interaction condition. Future work includes a longitudinal field study to investigate whether the results reported here hold for professional scientific contexts. In the field study, the technology will be provided to scientists who have expressed an interest in conducting scientific investigations using the system. We plan to investigate the similarities and differences between scientists' and study participants' perceptions and use of the technology to further our understanding of the impact of collaboratories on the scientific process and outcomes. Our thanks to the study participants; to students who helped run the experiment sessions and assisted in data analysis; to the team who built the nM, including Frederick P. Brooks, Jr., Martin Guthold, Aron Helser, Tom Hudson, Richard Superfine and Russell M. Taylor II. The development of the nM and this work have been funded by the NIH National Center for Research Resources, NCRR 5-P41-RR02170. The nanoManipulator project is part of the Computer Graphics for Molecular Studies and Microscopy Research Resource at the University of North Carolina at Chapel Hill.

This study investigates the use of criteria to assess relevant, partially relevant, and not-relevant documents. Study participants identified passages within 20 document representations that they used to make relevance judgments; judged each document representation as a whole to be relevant, partially relevant, or not relevant to their information need; and explained their decisions in an interview. Analysis revealed 29 criteria, discussed positively and negatively, that were used by the participants when selecting passages that contributed or detracted from a document's relevance. These criteria can be grouped into six categories: abstract (e.g., citability, informativeness), author (e.g., novelty, discipline, affiliation, perceived status), content (e.g., accuracy/validity, background, novelty, contrast, depth/scope, domain, citations, links, relevant to other interests, rarity, subject matter, thought catalyst), full text (e.g., audience, novelty, type, possible content, utility), journal/publisher (e.g., novelty, main focus, perceived quality), and personal (e.g., competition, time requirements). Results further indicate that multiple criteria are used when making relevant, partially relevant, and not-relevant judgments, and that most criteria can have either a positive or negative contribution to the relevance of a document. The criteria most frequently mentioned by study participants were content, followed by criteria characterizing the full text document. These findings may have implications for relevance feedback in information retrieval systems, suggesting that systems accept and utilize multiple positive and negative relevance criteria from users. Systems designers may want to focus on supporting content criteria followed by full text criteria as these may provide the greatest cost benefit.

Preface Essays The Kurds and Kurdistan: A General Background by Lokman I. Meho The Kurds in Lebanon: An Overview by Lokman I. Meho Bibliography General Works Anthropology Archaeology Decorative, Fine, and Performing Arts Description and Travel Economy and Development Education Health Conditions Jewish Kurds Journalism, Mass Communication, and Freedom of the Press Kurdish Diaspora Kurds in Syria, Lebanon, and Former Soviet Union Language Studies/Works Literature, Folklore, and Oral Traditions Music, Dancer and Songs National, Cultural, and Ethnic Identity National Identity and the Language Question Population and Urban Studies Religion in Kurdistan Sociology Women Miscellaneous Name, Title and Subject Index

The cost effective development of collaboration technology requires evaluation methods that consider group practices and can be used early in a system's life-cycle. To address this challenge, we developed a survey to evaluate collaboration technology based on innovation diffusion theory (E. Rogers, 1995). The theory proposes five attributes of innovations that influence technology adoption: relative advantage, compatibility, complexity, trialability and observability. Selecting items from existing surveys related to these attributes, we developed a prototype multi-scale survey to help evaluate whether using a system face-to-face or distributively influences study participants' attitudes towards system adoption. We have begun refining the survey instrument and report on this process, the proposed survey questions, and the reliability and validity of the survey instrument.

Collaboration, and increasingly multidisciplinary collaboration across distances, is a fundamental and strategic component of the scientific research process. The importance of collaboration in the scientific research process has long been recognized by the National Institutes of Health (NIH) in the USA. Its current policy is to fund the development or purchase of specialized scientific instruments in (not-for- profit) research labs across the USA and to help fund scientists to travel to those labs to collaborate and conduct scientific experiments using these specialized scientific instruments.

We compare a user-defined passage feedback (pf) system to a document feedback (df) system. Df employed the adaptive linear model for retrieval, while pf used weighted query expansion based on positive and negative feedback. Twenty-four searchers performed the same six tasks in varying search and system-order per TREC-8 guidelines. We hypothesized that pf, which featured interactive query expansion, would outperform df, which relied on automatic query expansion. Initial analysis appeared to reject this hypothesis, as df showed slightly higher overall performance than pf. However, analysis by system-order groups indicates only the first pf use had lower performance. These data suggest that pf was more difficult to learn than df, though the second pf use yielded competitive performance. If performance of pf is indeed affected by learning, an improved pf system with usability enhancements may prove to be an effective mechanism for interactive information retrieval.

CSCW’00, December 2-6, 2000, Philadelphia, PA. ACM 1-58113-222-0/00/0012. Enabling Distributed Collaborative Science Tom Hudson1, Diane Sonnenwald2, Kelly Maglaughlin2, Mary Whitton1, and Ronald Bergquist2 1Department of Computer Science 2School of Information and Library Science University of North Carolina, Chapel Hill University of North Carolina, Chapel Hill Campus Box 3175, Sitterson Hall Campus Box 3360, Manning Hall Chapel Hill, NC 27599-3175 Chapel Hill, NC 27599-3360 {hudson, whitton}@cs.unc.edu {dhs, maglk, bergr}@ils.unc.edu

We tested two relevance feedback models, an adaptive linear model and a probabilistic model, using massive feedback query expansion in TREC-5 (Sumner & Shaw, 1997), experimented with a three-valued scale of relevance and reduced feedback query expansion in TREC-6 (Sumner, Yang, Akers & Shaw, 1998), and examined the effectiveness of relevance feedback using a subcollection and the effect of system features in an interactive retrieval system called IRIS (Information Retrieval Interactive System) in TREC-7 (Yang, Maglaughlin, Mehol & Sumner, 1999). In TREC-8, we continued our exploration of relevance feedback approaches. Based on the result of our TREC-7 interactive experiment, which suggested relevance feedback using user-selected passages to be an effective alternative to conventional document feedback, our TREC-8 interactive experiment compared a passage feedback system and a document feedback system that were identical in all aspects except for the feedback mechanism. For the TREC-8 ad-hoc task, we merged results of pseudo-relevance feedback to subcollections as in TREC-7. Our results were consistent with that of TREC-7. The results of passage feedback, whose system log showed high level of searcher intervention, was superior to the document feedback results. As in TREC-7, our ad-hoc results showed high precision in top few documents, but performed poorly overall compared to results using the collection as a whole.

In our TREC-5 ad-hoc experiment, we tested two relevance feedback models, an adaptive linear model and a probabilistic model, using massive feedback query expansion (Sumner & Shaw, 1997). For our TREC-6 interactive experiment, we developed an interactive retrieval system called IRIS (Information Retrieval Interactive System), which implemented modified versions of the feedback models with a three-valued scale of relevance and reduced feedback query expansion (Sumner, Yang, Akers & Shaw, 1998). The goal of the IRIS design was to provide users with ample opportunities to interact with the system throughout the search process. For example, users could supplement the initial query by choosing from a list of statistically significant, two-word collocations, or add and delete query terms as well as change their weights at each search iteration. Unfortunately, it was difficult to tell how much effect each IRIS feature had on the retrieval outcome due to such factors as strong searcher effect and major differences between the experimental and control systems. In our TREC-7 interactive experiment, we attempted to isolate the effect of a given system feature by making the experimental and control systems identical, save for the feature we were studying. In one interactive experiment, the difference between the experimental and control systems was the display and modification capability of term weights. In another experiment, the difference was relevance feedback by passage versus document. For the TREC-7 ad-hoc task, we wanted to examine the effectiveness of relevance feedback using a subcollection in order to lay the groundwork for future participation in the Very Large Corpus experiment. Though the pre-test results showed the retrieval effectiveness of a subcollection approach to be competitive with a whole collection approach, we were not able to execute the subcollection retrieval in the actual ad-hoc experiment due to hardware problems. Instead, our ad-hoc experiment consisted of a simple initial retrieval run and a pseudo-relevance feedback run using the top 5 documents as relevant and the 100 document as non-relevant. Though the precision was high in the top few documents, the ad-hoc results were below average by TREC measures as expected. In the interactive experiment, the passage feedback results were better than the document feedback results, and the results of the simple interface system that did not display query term weights were better than that of the more complex interface system that displayed query term weights and allowed users to change these weights. Overall interactive results were about average among participants.