HTF 402 National First Nations Telehealth Research Project

Attachment 8: Evaluation Plan con't

Evaluation Methods

The overall approach to the evaluation uses multiple methods to assess changes over time from the perspectives of patients, personnel, communities and other stakeholders. It is based on the following principles:

• Consistent with the overall project philosophy, there will be community involvement in the evaluation at every phase, from the planning and development of tools and procedures, through the analysis of the data and interpretation of the results. Concretely, the evaluation team will visit each of the communities at the beginning of the project (in spring 2000) in order to adapt the proposed evaluation instruments and procedures and to plan the data collection. The evaluation will also try to build capacities for communities to conduct ongoing evaluation of their telehealth applications beyond the end of the formal research phase, 1) working directly with interested communities to build evaluation skills and resources, 2) adapting evaluation tools as closely as possible to existing procedures and resources, and 3) developing evaluation tools and instruments which are easy to use and inexpensive yet valid.

• The evaluation will try to minimize the amount of data collection activity required from the project sites.

• The conduct of the evaluation will respect the principles and assumptions of the Project Accountability framework. The Ethical Principles for Research with First Nations Communities set forth in Chapter 6 of the Tri-Council Policy Statement on Ethical Conduct for Research Involving Humans will also be respected. Moreover, to protect community privacy, the procedures have been designed so that the evaluation team will not have access to any patient identification information.

• The evaluation will be as rigorous as possible, and conclusions and recommendations will not go beyond the evidence found, particularly in the areas of patient outcomes and cost-effectiveness.

Monitoring telehealth system usage

Monitoring the use of the telehealth applications will provide information on the nature, level, quality and cost of usage. The main data collection tool will be patient encounter forms completed by health personnel and by remote health providers.

Nursing station patient encounters. During the study period, nursing station staff will be asked to record basic information about each encounter with patients who have the health conditions that would make them eligible for using a telehealth application. In order to assess the impact of the telehealth applications in the patient population, a record will be kept of patients seen at the nursing station who would be eligible for using telehealth, but who do not. )

These data will be recorded on a checklist-type form immediately after each encounter with an eligible patient. The forms will include:

• date, time and length of usage, as a proportion of total encounter
• health problem prompting the consultation
• who was present, at the local and remote sites
• what was done during the telehealth encounter
• the results of the visits in terms of subsequent actions
• any technical problems during the usage
• the implications of using telehealth in terms of costs incurred or avoided.

Draft versions of the patient encounter forms have been developed for each community, taking into account the applications to be implemented (Appendix 2). (The form has also been adapted for use in the one community using store-and forward technology.)

It is important to note that the section of the forms recording what was done during the visit provide indicators of quality of care. For those telehealth applications which address conditions for which guidelines for appropriate primary clinical practice have been established (diabetes, mental health, and cardiology), the forms include check points for the recommended components of appropriate care. These data will allow the evaluation to assess the extent to which care provided through telehealth meets accepted standards of care, as a proxy measure of the extent to which patient health outcomes are equivalent in telehealth and usual care.

Each patient will be assigned a code, to be used on all visits to the nursing station during the study period. The telehealth coordinator will be responsible for assigning codes to patients and for maintaining a master list of patients' names and codes. No nominative information will be sent outside the health centre. However, it will be essential that the same code be used for patients who visit the health centre more than once in the study period for health problems related to the telehealth applications, so that the analyses can examine the extent to which telehealth affects subsequent steps in patient care, including transfers.

The telehealth coordinator in each site will be asked to fax the completed encounter forms every two weeks to the evaluation team. These will be received at a secure fax site at McGill University.

Remote centre patient encounters. Practitioners (nurses or physicians) in the remote centres will be asked to complete a very brief patient encounter form after each telehealth encounter for patients in the study group in each site. This encounter form will include (see Appendix 3):

• date, time and length of usage
• what was done during the telehealth encounter
• the results of the visits in terms of subsequent actions
• any technical problems during the usage
• the implications of using telehealth in terms of costs incurred or avoided
• satisfaction with the session.

The telehealth coordinator will fax the required form to the provider before the session, with the patient's code already filled in, and the provider will return the completed form by fax after each session. These forms will be faxed to the evaluation team at the same time as the health centre forms, every two weeks. The two forms will be matched by the nursing station's patient code.

Patient satisfaction

All patients using telehealth in each community will be asked to complete a brief satisfaction questionnaire or interview to assess their reactions to and comfort with the telehealth system. The questionnaire (Appendix 4) has been developed from existing studies of telehealth. This questionnaire will be completely anonymous, and to the extent possible, completed in confidentiality from the nursing station staff (for example, self-administered and returned in sealed envelopes).

Qualitative interviews

Qualitative interviews will be conducted at two points in the study period, in the middle and near the end of the project. The first set of interviews will be conducted by telephone, and the second during in depth data-gathering visits to each community (likely in January of 2001). They will be conducted with stakeholders in each community, in the participating health systems, and at the national level. The stakeholders will be asked to respond as key informants giving their views on the evaluation questions from their perspectives within the project and the communities. They will be identified in collaboration with project leaders, health centre staff and other relevant sources such as the Peer review Committee.

Approximately ten semi-structured interviews will be conducted in each site at each time point, and will be one half hour to one hour in length. They may also be conducted as group consultations if this is more appropriate in the community. With participants' permission, the interviews will be tape-recorded.

The qualitative interviews will be conducted with four types of informants:

• Nursing station personnel (managers, nurses, CHRs, social services providers, translators): addressing local organization of telehealth, patients' reaction and outcomes, population coverage and accessibility, impacts on health centre work including workload, organization, skilling vs. deskilling, and relationships with tertiary providers;

• Remote centre and education telehealth partners (physicians, nurses and technical teams in the health centres linked to the participating communities; staff in educational institutions linked to the communities): addressing impacts on remote centre service organization and workload, patient reactions and outcomes, impacts on accessibility and quality of care and on relationships with community providers

• Community leaders (elders, council members and health committee members): in continuity with needs assessment data, addressing community level impacts on health care quality and accessibility, role of telehealth in service provision and management, overall positive and negative social and economic impacts and implications.

• Federal and provincial agency representatives: where relevant, addressing the integration of telehealth-based systems and services into provincial health networks and related legal, professional and administrative issues.

Relevant documentation on the telehealth project which may relate to the key informants' experiences (meeting minutes, tapes of community meetings, etc.) may also be reviewed and summarized. Appendix 5 contains a draft list of questions for the semi-structured interviews.

Financial data

Communities will be asked to provide the financial data required for us to conduct the cost-effectiveness part of the evaluation. This information will include:

• Direct costs for telehealth system:
  • equipment and supplies
  • maintenance contracts
  • renovations and systems upgrades
  • staff training and support.
• Per transfer costs for all patient transfers during the study period - For each transfer :
  • patient identifier code (same code as used on patient encounter forms)
  • dates of transfer out and in
  • transfer location
  • transport method
  • transportation and accommodation costs borne by the community
  • staff time costs for accompanying personnel.
• Telecommunication costs
  • costs for long distance to remote health centers using a) telephone and fax; and b) telehealth system, by month, for the study period
• Average hourly wage and benefits of health centre staff during the study period, by type and/or level.

Summary of Proposed Evaluation Methods

• Method
  1. Monitoring telehealth system usage
     • Data gathering activities - Logs of patient encounters using telehealth
     • Done by whom - Health centre staff; Remote health staff
     • When - Continuously throughout study period
  2. Patient satisfaction assessment
     • Data gathering activities - Questionnaires or interviews
     • Done by whom - Patients
     • When - Continuously throughout study period
  3. Qualitative interviews
     • Data gathering activities - Key informant interviews (10+ per site, at two points)Review of documents
     • Done by whom - Evaluation team in collaboration with health centre staff
     • When - By telephone in mid project;In person, near end of project
  4. Cost assessment
     • Data gathering activities - Compilation of cost data
     • Done by whom - Health centre management
     • When - At end of project

Ethical Issues

Informed consent to participate in the evaluation

Each patient who visits the health centre and uses telehealth will be asked for informed consent to participate in the evaluation, at the time of their first visit. Their consent will be obtained orally, by the health centre staff member who conducts the session, prior to the session. The telehealth system will be explained, stating that the care provided will be equivalent or better to usual care. The evaluation procedures will be explained, stating that 1) the staff members will record information about the visit, and 2) the patient will be asked to rate their satisfaction with the session (optional). It will be explained that the data will be confidential and that no individual will be identified by name, that participation is voluntary, that declining to participate will in no way affect care or services, and that the patient can change his mind about participating at any time. The consenting patient will sign a witnessed consent form or give witnessed oral consent. The signed consent forms will be retained in the health centre files for five years, in a location apart from the evaluation data. Appendix 6 shows a draft patient consent forms for adults and guardians of children who are patients.

As mentioned earlier, patient data transmitted to the evaluation team will be completely denominalized. Matching of patients over multiple encounters and with transfer information will de done through a patient code generated and managed within the community. The evaluation team will be aware only of the patient's community.

Key informants will not be asked to sign a consent form, as participation in this type of assessment may be considered as part of their social roles or professional responsibilities. However, they will be formally asked for consent to participate, and they will be told that they are free to decline participation without consequence. They will also be told that their responses will be kept confidential, and that no respondent will be identifiable in the evaluation reports. Once the audio-tapes of the interviews have been transcribed, the tapes will be destroyed and no identifying information will appear in the transcripts.

Because this evaluation will occur in the work settings of nursing station staff, particular care will be taken to preserve the confidentiality of information related to work organization, work performance and staff relations, in order to protect the interests of all participants in the project.

Analyses

All data from the patient encounter forms and satisfaction questionnaires will be entered into databases and analyzed using SPSS. Qualitative data from the key informant interviews will be transcribed verbatim and analyzed with the assistance of NUDIST.

In-depth, embedded case studies of each site

A multiple case study methodology will be used to integrate information from all of the available data sources . Each community involved in this project will constitute a case, for which the horizontal and site specific evaluation questions will be examined in a holistic, comprehensive fashion. A case report will be produced for each site, to be reviewed and validated before it is released.

Horizontal analysis

In addition, a comparative analysis of cross-case findings would be produced, addressing the horizontal evaluation questions. These would include quantitative analyses of telehealth usage, penetration, problems and impacts, as well as an overall qualitative appreciation of the success of the telehealth program and its potential for improving health delivery in First Nations communities.

Economic analysis

Although the potential overall cost savings for telehealth is of great interest in First Nations communities, it is our assessment that a strong cost reduction business case will be unlikely at the end of the project. Based on our review of the literature, given the expected usage levels and the expected proportion of transfers avoided in relation to the direct and indirect costs of telehealth (including equipment, maintenance, telecommunications, coordination, renovations; amortized over five years) we expect that the costs of telehealth will exceed the cost savings in the first several years of the project.

Moreover, in terms of benefit to patients, cost-effectiveness analysis is not appropriate because: 1) the sample sizes will be too small to detect any important health effects (e.g. diabetes complication avoided); 2) the number of practitioners is both small and confounded with the applications, so it will be difficult to separate issues of professional competence and telehealth impacts (for example, for a teledermatology application, it would be ideal to assess patient outcomes with several dermatology specialists, some with and some without telehealth, in order to say that any effects observed were due to the system and not the physicians' competence.) The evaluation will instead assess the extent to which the services provided using telehealth can be shown to be consistent with accepted guidelines for clinical practice, for applications where these exist. These data will not lend themselves to cost-effectiveness analyses because they cannot be directly translated into patient health outcomes or costs.

For the economic analysis, we will use the cost-consequence analysis matrix developed by McIntosh & Cairns. While more qualitative, it will allow us to include the many intangible consequences of telehealth (e.g. nurse empowerment; patient satisfaction). The approach will entail categorizing health and non-health consequences of telehealth, as measured in the evaluation data collection, as: beneficial, little difference, negative, or with insufficient consequence evidence. The costs of each of these consequences would be categorized as producing savings, little difference in cost, greater costs, or with insufficient cost evidence. The overall pattern of positive versus negative consequence for greater or lesser costs would be the basis for the business case.

Evaluation Schedule

The main milestones in the evaluation timetables are:

April 2000 - Visits to communities to adapt and plan the evaluation
April 30 2000 - Patient encounter data collection begins
September 1- 15 2000 - First set of key informant interviews
January 31 2001 - Patient encounter data collection ends
January 2001 - Visits to communities: second set of key informant interviews
February 1 to 15 - Case studies analyses and reports
February 15 to March 15 2001 - Review and validation of case studies by communities
February 1 to 28 2001 - Horizontal data analyses
March 1 to 31 2001 - Report preparation
March 31 2001 - Final report submitted