HTF 402 National First Nations Telehealth Research Project

Project Evaluation

1. Evaluation Questions

The evaluation addressed three issues central to the project objectives:

• telehealth impacts on patient and community access to needed, quality care
• role of telehealth in health services delivery, including cost-effectiveness and
• linkages of telehealth with existing health resources.

Since each community among the five is unique, the evaluation attempted to gather information relevant to each as well as to telehealth in First Nations' communities in general.

Evaluation questions relating to each of the above issues were developed though the literature review, the needs assessments and consultations with project officers, with support from existing general frameworks for evaluation questions, including Treasury Board guidelines² and other major approaches to evaluation of health and social programs³. The main evaluation questions are shown in Table 2 below, while site specific questions are addressed in later chapters about each community.

² Treasury Board of Canada, Program Evaluation Branch, Office of the Comptroller General (1991). Program Evaluation Methods: Measurement and Attribution of Program Results. Ottawa: Minister of Supply and Serices; Treasury Board of Canada, Program Evaluation Branch, Office of the Comptroller General (1989). Working standards for the evaluation of programs in federal departments and agencies. Ottawa: Minister of Supply and Services.

³ Stufflebeam, D. (1987). The CIPP model for program evaluation. in G. Madaus, M. Scriven, D. Stufflebeam (Eds.), Evaluation Models: Viewpoints on Educational and Human Services Evaluation. Boston: Kluwer-Nijhoff.

Table 2: Evaluation questions

• Issue: Access to needed, quality care
  Specific Evaluation Questions: To what extent do the telehealth applications respond to the community's needs, as defined by the needs assessment? To what extent do patients and families find each telehealth application acceptable? To what extent has telehealth improved access to needed, quality care? To what extent are services provided through telehealth consistent with established means of improving patient health outcomes?
  
  
• Issue: Health services delivery
  Specific Evaluation Questions: To what extent has telehealth use been organized successfully? To what extent have the professional skills and competencies required for telehealth been identified and successfully addressed through training? To what extent are telehealth applications used by eligible patients in the community? To what extent does telehealth improve competencies and confidence of local health personnel?
  
  
• Issue: Health services delivery (continued)
  Specific Evaluation Questions: How does telehealth affect staff workload, task allocation and professional practices? To what extent does telehealth result in cost increases, decreases or shifts for health service delivery within the communities? What is the level of technical success of the platforms, applications and suppliers in the implementing communities?
  
  
• Issue: Linkages among health resources
  Specific Evaluation Questions: To what extent is telehealth appropriated, integrated and sustained as a part of the community's self-governed health care system? To what extent have the telehealth applications become linked and integrated to provincial initiatives? To what extent does telehealth improve access of secondary, tertiary and education providers to local health service providers? To what extent does telehealth improve health service providers' awareness and knowledge of local conditions and resources?

  

2. Evaluation Methods

The overall approach to the evaluation used multiple methods to assess changes over time from the perspectives of patients, personnel, communities and other stakeholders. Consistent with the overall project philosophy, communities were involved in developing the evaluation methodologies. After developing an initial evaluation plan (Appendix 8), the evaluation team visited each of the communities at the beginning of the implementation period (spring 2000) in order to develop and adapt the proposed evaluation instruments and procedures to each community needs and functioning. As a result, the data collection methods and instruments varied somewhat from community to community.

Monitoring telehealth system usage

Monitoring the usage of the telehealth applications provided information on the nature, level, quality and implications of usage. The main data collection tools were patient encounter forms completed by nursing station personnel and by remote health providers (Appendices 2 and 3).

Nursing station patient encounters. During the study period, nursing station staff recorded basic information about each encounter with patients using the telehealth applications, on a checklist-type form immediately after each telehealth usage. The forms included:

• date, time and length of usage, as a proportion of total encounter
• the type of health problem prompting the consultation
• who was present, at the local and remote sites
• what was done during the telehealth encounter
• the results of the visits in terms of subsequent actions
• any technical problems during the usage
• the implications of using telehealth in terms of costs incurred or avoided.

The section of the forms recording what was done during the visit provided indicators of quality of care. For the telehealth applications that addressed conditions for which guidelines for appropriate primary clinical practice have been established (diabetes, mental health, and cardiology), the forms included checkpoints for the recommended components of appropriate care.

The forms were also used to record usage of telehealth for patient education, continuing professional education, and community development.

In Fort Chipewyan, a different system was used, based on a utilization record developed by the vendor and modified to accommodate the evaluation.

Each patient was assigned an identifier code, used on all visits to the nursing station during the study period. The telehealth coordinators in each community were responsible for assigning codes to patients and for maintaining a master list of patients' names and codes. No nominative information was sent outside the nursing station.

The telehealth coordinator in each site faxed the completed encounter forms every two weeks to the evaluation team. These were received at a secure fax site at McGill University.

Over the evaluation period, information was received about 927 telehealth sessions involving 176 patients. The number of sessions per community varied from 40 to 755, and the number of patients seen from 17 to 59.

Remote centre patient encounters. Practitioners (nurses or physicians) in the remote centres completed brief patient encounter form after each telehealth encounter for patients in the study group in each site. This encounter form included:

• date, time and length of usage
• what was done during the telehealth encounter
• the results of the visits in terms of subsequent actions
• any technical problems during the usage
• the implications of using telehealth in terms of costs incurred or avoided.

These forms were also regularly faxed to the evaluation team, using the same patient identifier as in the community form. Remote encounter forms were not completed for the Southend project.

Patient satisfaction

Patients using telehealth in each community were asked to complete a brief satisfaction questionnaire about their reactions to and comfort with the telehealth system. This questionnaire was based on an instrument developed by Saskatchewan's Northern Telehealth Network (see Appendix 4 for the different versions of this questionnaire used in each community). A total of 110 questionnaires were received.

Qualitative interviews

Qualitative interviews were conducted at the end of the project during in depth data-gathering visits to each community (in February and March 2001). They were conducted with stakeholders in each community and in the participating health systems, using semi-structured interview guides. The stakeholders were asked to respond as key informants giving their views on the evaluation questions from their perspectives within the project and the communities. They were identified in collaboration with project leaders and nursing station staff. The interview guides may be found in Appendix 5.

The interviews lasted from one half hour to three hours in length. Some were conducted as group consultations, and some were conducted by telephone or videoconference. If participants gave permission, they were tape-recorded.

The table below summarizes the qualitative interviews conducted about each community:

Table 3: Summary of interviews conducted about each community

• Community: La Romaine
  Number of Interviews: 9 interviews with 9 individuals
  Types of respondents: Telehealth coordinator, nursing staff, director, remote providers, remote telehealth coordinator, provincial telehealth representatives, vendor
  
  
• Community: Berens River
  Number of Interviews: 9 interviews with 14 individuals
  Types of respondents: Telehealth coordinator/project officer, telehealth supervisor, nursing staff, community workers, remote providers and educators, remote telehealth coordinator
  
  
• Community: Southend
  Number of Interviews: 13 interviews with 20 individuals
  Types of respondents: Telehealth coordinator, nursing station board members and management, nursing and mental health staff, community workers visiting and remote providers, remote telehealth coordinator, provincial telehealth representative, project officers
  
  
• Community: Fort Chipewyan
  Number of Interviews: 12 interviews with 22 individuals
  Types of respondents: Telehealth coordinator, Band and nursing station managers, nursing staff, nursing station board member, elders, patients, tertiary providers and management , provincial telehealth representatives, Health Canada representative

3. Ethical Issues

The conduct of the evaluation respected the principles and assumptions of the Project Accountability framework, as well as the Ethical Principles for Research with First Nations Communities set forth in Chapter 6 of the Tri-Council Policy Statement on Ethical Conduct for Research Involving Humans. To protect community privacy, the procedures were designed so that the evaluation team had no access to any patient identification information.

In its initial visits to the communities, the evaluation team discussed ethical issues with community representatives, and where deemed appropriate, obtained formal authorization from the Band for the study according to the agreed-upon ethical principles.

In those communities that required it, each patient using telehealth completed a consent form for participation in the project and/or the evaluation, at the time of their first visit. The telehealth system was explained, stating that the care provided would be equivalent to usual care. The evaluation procedures and the procedures for ensuring confidentiality were explained, as was the voluntary nature of participation without prejudice to care or services. The signed consent forms were retained in nursing station files.

Key informants were not asked to sign a consent form, as participation in this type of assessment was considered as part of their normal roles or professional responsibilities. However, they were formally asked for consent to participate and told that their responses would be kept confidential, and that no respondent would be identified in the evaluation reports.

A complete operational research protocol was prepared for the entire project, for submission to Research Ethics Review Committee in research and medical institutions, which required it. This may be found in Attachment 6.

4. Analyses

All data from the patient encounter forms and satisfaction questionnaires were entered into databases and analysed using SPSS. Qualitative data from the key informant interviews was transcribed and analysed using matrix analysis techniques. For the economic analysis, the cost-consequence analysis matrix developed by McIntosh & Cairns was used (1999).