This report represents the culmination of the NTOIP which looked at outcome indicators in four categories (quality, access, acceptability and cost). The study was conducted in four phases, each presented as a separate section here. (1) Strategy and approach – Several approaches identified from other studies were adapted for use in the NTOIP. These included processes to define outcome indicators and build consensus. In addition, a framework was developed to help guide telehealth outcome development. (2) Systematic review of the literature – The goal of the literature review was to determine what outcome indicators have been used previously and what definitions for those indicators exist. The results showed that a myriad of telehealth outcome indicators have been used. However, these indicators have typically been those for which data could be collected easily (often process indicators), rather than indicators that represent a meaningful outcome for the user. Furthermore, given that very few definitions or descriptions exist, the same measures have rarely been used consistently between studies. (3) National experts workshop – Telehealth experts from Canada were invited to participate in a workshop and were provided with information about the study’s strategy, approach and the literature review. The workshop therefore built on the literature review. The results emphasized the difficulty of identifying a limited number of generic outcome indicators. From the workshop, a list of Candidate Outcome Indicators (COIs) was identified for further consideration. (4) Refinement and consensus – The final phase included a review of the COIs by the research team to establish proposed outcome indicators (POIs), followed by a Webbased consensus process to determine a list of Tentative Outcome Indicators (TOIs), and finally approved outcome indicators (AOIs) for quality, access, acceptability and cost. The final step in Phase 4 was Web-based selection of the top three AOIs from each category which resulted in a small number of unique and recommended AOIs.

Through a process of didactic presentation, breakout group discussion and plenary debate, workshop participants identified a variety of items that could be used to evaluate telehealth initiatives. As a result of the process, a total of 12 COIs were identified for quality, six for access, 15 for acceptability and 11 for cost. Each of these was subject to Phase 4 of the NTOIP, where the research team reviewed and refined each COI and an open consensus building process took place.

A Canadian project (the National Initiative for Telehealth Guidelines) was established to develop telehealth guidelines that would be used by health professionals, by telehealth providers as benchmarks for standards of service and by accrediting agencies for accreditation criteria. An environmental scan was conducted, which focused on organizational, human resource, clinical and technological issues. A literature review, a stakeholder survey (245 mail-outs, 84 complete responses) and 48 key informant interviews were conducted. A framework of guidelines was developed and published as a preliminary step towards pan-Canadian policies. Interim recommendations were that organizations and jurisdictions might consider formal agreements to specify: (1) organizational interoperability; (2) technical interoperability; (3) personnel requirements; (4) quality and continuity-of-care responsibilities; (5) telehealth services; (6) remuneration; and (7) quality assurance processes. An additional recommendation was that flexible mechanisms were needed to ensure that accreditation criteria will be realistic and achievable in the context of rapid changes in technology, service integration and delivery, as well as in the context of operating telehealth services in remote or underserved areas.

There are limited data on the effectiveness of mannequin-based simulations in pediatrics. This study developed a training program using a high-fidelity child mannequin to simulate critical cases in an emergency department, and examined the learning gains derived from this simulation. Eighteen pediatric residents, as pairs, participated in a high-fidelity simulation pretest, training session and a posttest. The training session, developed based on participants' pretest performance, included videotape review, feedback, and hands-on practice, and focused on the improvement of management skills for shock and tachydyspnea. The pre- and posttest performances were scored for task-specific technical skills and behaviors. The learning gains between the pre- and posttests were significant (p < 0.001) for task-specific technical skills (from 64 +/- 15% to 93 +/- 4%) and behaviors (from 65 +/- 18% to 85 +/- 12%). This study suggests that high-fidelity simulation can enhance learning about how to manage critical cases in the pediatric emergency department.

We asked the views of potential users of a proposed Canadian broadband Internet Protocol (IP) network for health, the Alberta SuperNet. The three user groups were drawn from the public, provider and private sectors. In all, 35 health-sector participants were selected (17 government, nine health-care organizations, five providers/practitioners and four private sector). The questionnaire was Web-based, semistructured and self-administered. It consisted of four major areas: value, readiness, effect on usual care and limitations. A total of 28 (80%) individuals responded to the questionnaire: 21 (81%) were from the public sector (three provincial, nine regional and nine organizational), three (60%) were from the provider sector and four (100%) were from the private sector. Overall, the items related to health services and health human resources were considered to be the most valuable to rural communities. Respondents identified the expansion of telehealth services as the most important, except those from the private sector, who ranked this a close second. The health system's move to the use of electronic health records was ranked second in importance by all respondents. The private-sector respondents viewed all user groups to be generally less ready (mean score 2.5 on a seven-point scale from 1 = not ready to 7 = ready), while the public-sector respondents were the most optimistic (mean score 4.0). Specific socioeconomic impact data were limited. The top-ranked disadvantage of the 10 suggested was that 'Changes in health-service delivery practices and/or processes will be required'.

This paper examines telehealth readiness from an organizational perspective and explores the essence of telehealth readiness among four domains, namely, patients, practitioners, the public, and organizations in rural Canadian communities. Because readiness is a necessary requirement for the successful implementation of an innovation, it is important to identify and ensure core factors of readiness before costly investments are made. The findings presented here derive from a qualitative phenomenological research approach involving semistructured telephone interviews with four key informants (respondents). The data identified four categories of readiness in an organizational setting: core readiness, engagement, structural readiness, and nonreadiness. Understanding organizational readiness within rural and remote communities is an important step for the successful implementation of telehealth services into existing systems of health care.

Economic evaluation of telemedicine applications is required to provide decision makers in health care with appropriate information on costs and benefits of this information and communications technology. The level of economic evaluation should evolve as telemedicine applications mature. At the basic level, economic evaluation may include basic cost analysis and primarily observational data on nonmonetary benefits. The focus will change as telemedicine programs develop. At this intermediate level, practice patterns and workforce issues are addressed as they affect utilization and costs of telemedicine services. Longer-term economic evaluation, thus far not achieved in telemedicine assessment, should focus on assessment of health outcomes and economic impact. Alberta, Canada has made progress assessing telemedicine applications in psychiatry, radiology, rheumatology, and rehabilitation. Data availability and analytic resources continue to present challenges to economic assessment of telemedicine.

Challenges to the development of appropriate yet adaptable policy and tools for security of the individual patient electronic health record (EHR) are proving to be significant. Compounding this is the unique capability of e-health to transgress all existing geo-political and other barriers. Initiatives to develop and advance policy, standards, and tools in relation to EHR access control and authorisation management must address this capability. Currently policy development initiatives take place largely in an isolated manner. This jeopardises the potential of e-health because decisions made in one jurisdiction might hamper, even prevent, an e-health opportunity in another. This paper places access and authorisation issues in an overall policy context through describing current Canadian initiatives. The National Initiative for Telehealth (NIFTE) Guidelines project is developing a framework of national guidelines for telehealth. The Policy and Peer Permission (PPP) project is developing a unique tool that provides persistent protection of data. The new corporate body ‘Infoway’ is developing a pan-Canadian electronic health record solution. Finally, the Glocal e-Health Policy initiative is developing a tool with which to identify and describe the inter-relationships of e-health issues amongst policy levels, themes, and actors.

PURPOSE:To evaluate the use of remote reading of digital retinal photographs in the diagnosis of severe (referral-warranted) retinopathy of prematurity (ROP) during longitudinal screening for ROP. STUDY DESIGN:Prospective, longitudinal cohort study. SUBJECTS:Forty-four consecutive premature infants at risk for ROP. METHODS:All infants were examined longitudinally, over a series of examinations, by indirect ophthalmoscopy (gold standard) and digital photography using the RetCam-120 Digital Retinal Camera (Massie Research Laboratories Inc., Dublin, CA) equipped with an ROP lens. Images were stored and read remotely by a masked reader. Referral-warranted ROP was defined as ROP in zone 1, the presence of plus disease or the presence of any stage 3 ROP. We determined whether and when referral-warranted ROP was diagnosed for each eye, of each infant, on each examination, during the course of each of the infant's screening. RESULTS:Severe (referral-warranted) ROP was diagnosed in 23 eyes by indirect ophthalmoscopy during their series of examinations. Digital photography had a sensitivity of 100% and a specificity of 96% in detecting referral-warranted ROP. The positive predictive value of digital photography was 92%, and the negative predictive value was 100%. In 87% of eyes, referral-warranted ROP was diagnosed by digital photography before or at the same time as indirect ophthalmoscopy. CONCLUSIONS:Longitudinal remote reading of digital photographs using the RetCam-120 system has excellent specificity and sensitivity in detecting referral-warranted ROP. This pilot study has shown that remote reading of digital photographs has promise for telemedicine strategies in ROP screening.

A qualitative approach was used to explore the readiness of a rural community for the implementation of telehealth services. There were four domains of interest: patient, practitioner, public and organization. Sixteen semistructured telephone interviews (three to five in each domain) were carried out with key informants and recorded on audio-tape. Two community awareness sessions were held, which were followed by five audio-taped focus groups (with five to eight people in each) in the practitioner, patient and public domains. In addition, two in-depth interviews were conducted with community physicians. Analysis of the data suggested that there were four types of community readiness: core, engagement, structural and non-readiness. The level of readiness varied across domains. There were six main themes: core readiness; structural readiness; projection of benefits; assessment of risk; awareness and education; and intra-group and inter-group dynamics. The results of the study can be used to investigate the readiness of rural and remote communities for telehealth, which should improve the chance of successful implementation.

Purpose Developing quality examinations that measure physicians' clinical performance in simulations is difficult. The goal of this study was to develop a quality simulation examination using a high-fidelity child manikin in evaluating paediatric residents' competence about managing critical cases in a simulated emergency room. Quality was determined by evidence of the reliability, validity and feasibility of the examination. In addition, the participants' responses regarding its realism, effectiveness and value are presented.Method Scenario scripts and rating instruments were carefully developed in this study. Experts were used to validate the case scenarios and provide evidence of construct validity. Eighteen paediatric residents, 'working' as pairs, participated in a manikin-based simulation pre-test, a training session and a post-test. Three independent raters rated the participants' performance on task-specific technical skills, medications used and behaviours displayed. At the end of the simulation, the participants completed an evaluation questionnaire.Results The manikin-based simulation examination was found to be a realistic, valid and reliable tool. Validity (i.e. face, content and construct) of the test instrument was evident. The level of inter-rater concordance of participants' clinical performance was good to excellent. The item analysis showed good to excellent internal consistency on all the performance scores except the post-test technical score.Conclusions With a carefully designed rating instrument and simulation operation, the manikin-based simulation examination was shown to be reliable and valid. However, a further refinement of the test instrument will be required for higher stake examinations.

Telehealth "readiness" can be defined as the degree to which users, health care organizations, and the health system itself are prepared to participate and succeed in its application. This project developed a readiness model for rural/remote locations in Canada. Specifically defined groups or communities with shared characteristics within a rural geographical community (i.e. practitioners, patients, the public, and health care organizations) participated in key informant interviews, awareness sessions, focus groups, and face-to-face interviews. The data were examined and organized keeping in mind Weiss' Program's Theory of Change. This approach allowed concrete and abstract factors to be considered. The model that emerged suggests that there are four types of readiness for each of the defined communities: core, engagement, structural, and non-readiness. The "communities" share some readiness factors and risks, but also exhibit unique elements. This finding is critical to acknowledge when the goal is to implement a useful, effective, and sustainable telehealth system within remote settings. Study results hold a key to understanding why technology systems have failed in the past, in spite of dedicating considerable human and financial resources towards their implementation. Notations of these findings will be helpful in future telehealth implementations within rural and isolated areas.

OBJECTIVE In response to increased referrals to Florida's Child Protection Teams and concern regarding statewide availability of medical expertise in the area of child abuse and neglect, Children's Medical Services of the Florida Department of Health established a telemedicine project to facilitate immediate expert medical evaluations of alleged child abuse or neglect. This article describes a baseline examination of the project, including the technique of concept mapping, to examine how larger systematic factors influence the adaptation of telemedicine technology in child abuse examination settings. METHODS This study included interviews of key staff plus the incorporation of concept mapping, which takes qualitative data (individual statements and opinions) and quantifies them (sorts and ranks them by order of group importance). RESULTS Findings from interviews revealed that the frequency of use of telehealth services varies across the state as a result of several factors, including space limitations and staff training. Patients, however, seem to be comfortable with the use of the new technology. The concept mapping exercise displayed a progression of issues that are perceived to have an impact on the use of this technology. CONCLUSIONS Technology use is affected by unforeseen variables, such as physical space limitations and examination room availability. Family concerns about patient privacy issues were rare and were resolved quickly by the health care practitioner. Although using this equipment is not difficult, the search for user-friendliness should be continued. Staff engagement early in the process likely will result in a greater likelihood of use of the technology.telehealth, telemedicine, child protection, child abuse and neglect, concept mapping.

The goal of this project was to develop and test a multi-application telehealth workstation with an interface seamless to the end-users. This project required collaboration among the private sector, a clinical regional referral setting, a clinical tertiary receiving center, and a university academic unit. The project applied the usability testing methodology to design and test the workstation: (1) the developmental phase focused on planning, prototype workstation development, and end-user trials in the private sector research and development environment, and (2) the operational testing phase moved the workstation into the clinical setting. The latter included training of end-users and addressing policy and ethical issues. In addition, the partners documented the goals achieved and lessons learned. The project resulted in the refinement of the workstation for clinical applications, The unique, diverse, and at times complementary goals and lessons learned by each partner are noted. Collaboration around a shared goal was a key element in achieving the refined workstation. Collaborators reported that the lessons learned will aid them as they pursue telehealth activities.

The University of Calgary's Faculty of Medicine and the Calgary Regional Health Authority understand that telehealth is an evolving field requiring both academic enquiry and operational readiness. Both parties are committed to quality educational programmes--the Faculty through its commitment to excellence and the Authority with its charge to maintain and enhance such programmes. There are shared applications, multi-learner user groups, shared strategies to overcome distances and shared infrastructure--technologies, communication pathways and resources. Having embarked on a joint telelearning venture, we have learned a number of lessons. Central to progress has been an appreciation and respect for unique mandates, a spirit of trust and flexibility, an agreement on a set of principles, ongoing communication between and participation from the users and, at times, redirection. Questions being answered include the following. How well is this collaborative model working? How functional is it at this time of health reform and restructuring? Can one meet complementary telelearning goals within a faculty-health authority relationship? These all have implications for future success.

The College of Physicians and Surgeons of Alberta, in collaboration with the Universities of Calgary and Alberta, has developed a program to routinely assess the performance of physicians, intended primarily for quality improvement in medical practice. The Physician Achievement Review (PAR) provides a multidimensional view of performance through structured feedback to physicians. The program will also provide a new mechanism for identifying physicians for whom more detailed assessment of practice performance or medical competence may be needed. Questionnaires were created to assess an array of performance attributes, and then appropriate assessors were designated--the physician himself or herself (self-evaluation), patients, medical peers, consultants and referring physicians, and non-physician coworkers. A pilot study with 308 physician volunteers was used to evaluate the psychometric and statistical properties of the questionnaires and to develop operating policies. The pilot surveys showed good statistical validity and technical reliability of the PAR questionnaires. For only 28 (9.1%) of the physicians were the PAR results more than one standard deviation from the peer group means for 3 or more of the 5 major domains of assessment (self, patients, peers, consultants and coworkers). In post-survey feedback, two-thirds of the physicians indicated that they were considering or had implemented changes to their medical practice on the basis of their PAR data. The estimated operating cost of the PAR program is approximately $200 per physician. In February 1999, on the basis of the operating experience and the results of the pilot survey, the College of Physicians and Surgeons of Alberta implemented this innovative program, in which all Alberta physicians will be required to participate every 5 years.

The ratio of physicians to population in Canada peaked in the mid-1990s and is now falling. The decrease in the number of family physicians has had a disproportionate effect on rural and remote communities, and surveys have indicated that the availability of physicians and services is likely to deteriorate in rural and remote communities. Telemedicine is increasingly evident in every Canadian province and territory, and it could assist in more effective use of workforce resources by linking appropriate experts at central sites to patients and practitioners at remote sites. Positive effects on recruitment and retention of health providers and morale of the local workforce can be expected. In spite of national interest, evidence for the effect of telemedicine on staff distribution, roles and recruitment, use of health resources, health management and system integration is very limited. Telephone interviews were therefore conducted to collect information from 12 telehealth projects in Canada, one from each province or territory. The responses confirmed observations in the literature that telemedicine has positive outcomes for the workforce.