HTF 402 National First Nations Telehealth Research Project
Draft Evaluation Plan
(Revised Version) March 6, 2000
Submitted by: Communications Infotelmed
-- working document only --
Please note that this plan has not yet been approved by participating
communities.
 (88.3
K)
Attachment 8: Evaluation Plan
Introduction
First Nations' Telehealth Research Project
Telehealth -- the delivery of health information, resources and
services through remote technology -- is becoming increasingly
accessible as part of comprehensive health care systems. For underserved
and remote communities, telehealth offers the promise of reducing
the constraints imposed by distance and poor infrastructure while
improving health and well-being. As one of several telehealth demonstration
and evaluation projects funded through the Federal Health Transition
Fund, the First Nations' Telehealth Project will deploy and evaluate
applications of telehealth in five remote First Nations communities.
The overall objectives of the First Nations' Telehealth Research
Project are: 1) to improve patient and community access to high
quality health care, including timeliness of access to medical
advice, services and health information; 2) to improve the delivery
of cost-effective health services in the communities; and 3) to
improve linkages of remote health care centres to secondary, tertiary
and educational facilities in each province.
The communities involved in this project, each of which will implement
three different telehealth applications, are diverse. Four have
already assumed governance of a major portion of their health systems,
and the fifth is in the preliminary stages of negotiation for the
transfer. The health resources available within each community
differ, as do their access to secondary and tertiary care.
The table below summarizes the telehealth applications which will
be implemented in each community. These applications were identified
through a detailed needs assessment involving extensive community
consultations about health priorities and issues relating to telehealth.
The needs assessment process involved: qualitative interviews with
key informants (health centre personnel, community leaders and
opinion leaders, and secondary/tertiary centre partners); review
of health status information gleaned from local or regional health
statistics or records; review of background information on communities'
social and demographic characteristics.
Telehealth Applications Selected for Five Communitie
- Community: Anahim
Lake
Population: 659 band
members 400 non band members 5,000 tourist season
Referral: Williams
Lake
Telehealth Applications (First three priorities):
- Trauma triage and early intervention
- Cardiology
- Mental health services
- Community: Berens
River
Population: 1759
band members
Referral: Winnipeg
Telehealth Applications (First three priorities):
- Diabetes prevention and clinical
care
- Mental health services: patient
assessment and counselling; specialist consult, group therapy
- Continuing medical education programs
for health care staff, administrators and paraprofessionals
- Community: La Romaine
Population: 100 off-reserve
Referral: Sept-Iles
Quebec City
Telehealth Applications (First three priorities):
- Diabetes medical consultations:
remote home care, retinal examination, skin examination
- Cardiovascular health: tele
ECG, telemonitoring (heart rate, saturometer, blood pressure)
- Ear-Nose-Throat remote specialist
consultations
- Community: Fort Chipewyan
Population: 2900
total 1864 in area
Referral: Fort McMurray
Telehealth Applications (First three priorities):
- Rehabilitation services: Occupational,
speech and physiotherapy
- Distance education for health
staff
- Telefamily/ televisitation
- Community: Southend
Population: 918 in
community, one of several in Peter Ballantyne Cree Nation
Referral: La Ronge
Prince Albert Saskatoon
Telehealth Applications (First three priorities):
- Specialist consults
- Diabetic education and management:
monitoring, outreach and education
- Mental health: remote consultation
for adults and children
Literature review of telehealth applications
A review of the research literature on those telehealth applications
to be implemented by the five communities has shown that relevant
information is available from empirical studies of these applications.
However, there is greater literature coverage in the areas of mental
health, diabetes management, dermatology and emergency medicine,
with relatively little focus on rehabilitation, ENT and continuing
medical education. Moreover, the quality of the studies varies
greatly: few have appropriate control groups, long-term follow-up,
or assessment of patient health outcomes. In addition, the studies
tend to be conducted in rural but not necessarily remote locations.
Below, we summarize the main general conclusions across all applications.
A detailed summary of the lessons learned from the articles on
relevant telehealth applications may be found in Appendix 1.
- To date, the research literature in telemedicine has tended
to concentrate on the accuracy and reliability of information
and diagnoses provided through telehealth applications. In general,
these studies show adequate levels of agreement between remote
and in-person consultations. In this sense, it can be said that
there is at this point, relatively strong evidence that quality
of care provided through telehealth is comparable to that provided
through usual channels. However, other aspects of quality of
care, including continuity and comprehensiveness, have not been
addressed.
- It is unclear as yet whether telehealth represents an improvement
in the quality and accessibility of care, versus a shifting of
the same care to less costly methods of delivery.
- The organizational implications of implementing telehealth
are not well documented, although some studies mention changes
in work loads and work organization for health personnel. Scheduling
telehealth consultations seems to consume more time than expected.
The discussion sections in some studies emphasize that the success
of telehealth depends mostly on human factors: planning, cooperation,
and an accepting organizational culture.
- While many studies address the cost-effectiveness of telemedicine,
they tend to limit their assessment of costs to the time involved
from the specialty physicians. These studies usually show that
telehealth is cost-effective. More comprehensive studies including
equipment, telecommunications, and organizational costs suggest
that cost-effectiveness is not guaranteed, and at the very least
must be assessed over a relatively long amortization period.
- Levels of use of telehealth systems are often less than expected.
No studies have systematically examined the proportion of eligible
patients in a given practice population who use telehealth. Usage
level is a major factor in cost-effectiveness when the cost of
equipment and communication infrastructure is considered. (It
is also central to the evaluation of this project, as a minimum
of 30 different patients per community receiving services over
the telehealth system during the life of the project (January
to September 2000) will be required for valid quantitative analyses,
including the cost-effectiveness assessment.)
- Cost reductions have been examined in terms of decreased travel
(patient transfers to specialized centres) and decreased wait
time. Use of telehealth may diminish patient transfers, but not
in all cases. While avoidance of transfers or of in-person visits
depends greatly on the applications, very roughly speaking we
could expect that about 40-50% of transfers might be avoided
by using telehealth. The extent to which telehealth defers rather
than replaces in-person consultations has not been addressed.
- Reactions among local health providers (in most studies, general
practitioners) are generally quite positive, and there is some
evidence that telehealth provides learning opportunities for
them, thus indirectly improving the quality of care received.
- Reactions among local health providers (in most studies, general
practitioners) are generally quite positive, and there is some
evidence that telehealth provides learning opportunities for
them, thus indirectly improving the quality of care received.
- Financial and administrative practices may be major barriers
to implementation, especially for remunerating remote personnel.
Legal barriers have been overcome in some settings but have limited
the deployment of telehealth in others.
- Technical problems are not absent in telehealth use, and are
more frequent in the early stages of implementation. Adequate
training and learning time are required, and skill levels must
be maintained. Difficulties with audio and images are more frequent
than connection problems. Lighting and camera operation have
the greatest impact on image quality.
- Overall, the research literature in telehealth suggests that
it has the potential to improve health services delivery while
maintaining patient health outcomes, but that its cost-effectiveness
and implementability have yet to be clearly demonstrated.s
Project Evaluation
Evaluation Questions
The evaluation has been designed to address, as comprehensively
as possible, three issues central to the project objectives:
- telehealth impacts on patient and community access to needed,
quality care
- role of telehealth in health services delivery, including cost-effectiveness
and
- linkages of telehealth with existing health resources.
Since each community among the five is unique in terms of its
history, culture, needs, preferences, resources, social organization,
political organization, resource people and infrastructures, the
evaluation must be appropriate and responsive to each community
while at the same time providing information relevant to telehealth
implementation across all five sites as well as for other First
Nations communities.
The evaluation questions relating to each of the above issues
were developed though the literature review, the needs assessments
and consultations with project officers, with support from existing
general frameworks for evaluation questions, including Treasury
Board guidelines(Treasury Board of Canada, Program Evaluation Branch,
Office of the Comptroller General (1991). Program Evaluation Methods:
Measurement and Attribution of Program Results. Ottawa: Minister
of Supply and Services; Treasury Board of Canada, Program Evaluation
Branch, Office of the Comptroller General (1989). Working standards
for the evaluation of programs in federal departments and agencies.
Ottawa: Minister of Supply and Services.) and other major approaches
to evaluation of health and social programs(Stufflebeam, D. (1987).
The CIPP model for program evaluation. in G. Madaus, M. Scriven,
D. Stufflebeam (Eds.), Evaluation Models: Viewpoints on Educational
and Human Services Evaluation. Boston: Kluwer-Nijhoff.). There
are two sets:
- questions common to all the participating communities (the
horizontal questions)
- questions specific to each community (site-specific questions).
Indicators have been developed for each question. Examples are
provided in the tables below.
Horizontal evaluation questions (all sites)
- Issue
- Access to needed, quality care
- Specific evaluation questions
- To what extent do the telehealth applications respond
to the community's needs, as defined by the needs
assessment?
- Indicators - Stakeholders
views of telehealth application responsiveness
to needs identified in the needs assessment
- To what extent do patients and families find each
telehealth application acceptable?
- Indicators - Proportion of
eligible patients who accept and refuse to use
telehealth; refusal reasonsPatient reports of
acceptability and satisfaction
- To what extent has telehealth improved access to
needed, quality care?
- Indicators - Proportion of
eligible patients receiving relevant specialist
care, with and without telehealth;Delays to access
relevant specialist care, with and without telehealth;Views
of health personnel on quality and accessibility
of links to specialists and on communication
flow
- To what extent are services provided through telehealth
consistent with established means of improving patient
health outcomes?
- Indicators - Proportion of
telehealth interventions which meet accepted
standards of care (clinical guidelines); Proportion
which improve likelihood of meeting accepted
standards of care;
- Health services delivery
- Specific evaluation questions
- To what extent has telehealth use been organized
successfully?
- Indicators - Health staff
reports of organizational problemsProportion
of patient encounters involving scheduling difficulties
- To what extent have the professional skills and
competencies required for telehealth been identified
and successfully addressed through training?
- Indicators - Health staff
reports of skill and competency requirements
for using telehealth/ Health staff assessments
of training adequacyNumber and type of difficulties
encountered which could been prevented through
training
- To what extent are telehealth applications used
by eligible patients in the community?
- Indicators - Proportion of
patients with targeted health conditions as primary
or secondary diagnosis using telehealth/ Proportion
of health centre visits of patients with targeted
health conditions as a primary complaint using
telehealth
- To what extent does telehealth improve competencies
and confidence of local health personnel?
- Indicators - Health personnel
and tertiary provider views of own and each others'
competency and confidence in application areas
- How and how much does telehealth affect staff workload,
task allocation and professional practices?
- Indicators - Health staff
workload : active cases, patients per week, time
spent on non-transferred patients; Task allocation
and professional practices: shifts in types of
services given, integration of telehealth responsibilities
into ongoing responsibilities
- To what extent does telehealth result in cost increases,
decreases or shifts for health service delivery within
the communities?
- Indicators - Direct costs
: health personnel time, other staff time, telecommunications,
supplies, training etc./ Costs avoided : costs
of patient transfers avoidedOpportunity costs
: shift of staff time to curative from preventive
care
- What is the level of technical success of the platforms,
applications and suppliers in the implementing communities?
- Indicators - Ease of meeting
infrastructure requirements (space, telecommunications)Proportion
of down time/ Proportion of patient encounters
affected by technical problems/ Number of incidents
requiring technical supportTraining time required/
Staff views of ease of use and technical quality
- Linkages among health resources
- Specific evaluation questions
- To what extent is telehealth appropriated, integrated
and sustained as a part of the community's self-governed
health care system, or integrated into the negotiations
of transfer agreements?
- Indicators - Stakeholders'
views of telehealth sustainability : post project,
short term, long termSustainability plans and
budgetsIntentions of community leaders to continue
support
- To what extent have the telehealth applications
become linked and integrated to provincial initiatives?
- Indicators - Provincial health
systems' views of strength and compatibility
of links
- To what extent does telehealth improve access of
secondary, tertiary and education providers to local
health service providers?
- Indicators - Secondary, tertiary
end education provider views of access to local
health centre staff
- To what extent does telehealth improve health service
providers' awareness and knowledge of local conditions
and resources?
- Indicators - Tertiary providers
views of own awareness and knowledge of local
health conditions and resources
Site specific evaluation questions
Based on our discussions with project officers and teams, specific
evaluation questions have emerged as priorities for each site.
(These questions must be validated by the participating communities.)
While some of these questions will also be addressed by the cross-site
evaluation, they will be given special emphasis in the sites where
they are a priority.
- Community
- Fort Chipewayan
- Evaluation Questions
- To what extent does telehealth increase accessibility
of rehabilitation services?
- To what extent and with whom does telehealth position
the community as a source of expertise in First Nations
rehabilitation care?
- Indicators
- Proportion of eligible patients receiving relevant
rehab care in community; Delays to access rehab care;Stakeholder
views of impacts on referral and request patterns
to Fort Chipewyan
- Anahim Lake
- Evaluation Questions
- To what extent does telehealth increase access
to tertiary, secondary and education resources?
- To what extent does telehealth create perceived
inequities in access to specialized services?
- What is the net effect of telehealth on workload
of nursing staff and translators?
- How have telehealth applications in mental health
contributed to social development issues management
in the community?
- Indicators
- Proportion of eligible patients receiving relevant
specialist care; Delays to access relevant specialist
care; Views of health personnel on quality and accessibility
of links to specialists and on communication flow;
Patient perceptions of access equityHealth staff
and translator workload : active cases, patients
per week, time spent on non-transferred patientsStakeholder
views of relations to other social development initiatives
- La Romaine
- Evaluation Questions
- To what extent does telehealth contribute to an
increasing emphasis of the community health centre
on community health and prevention?
- To what extent does telehealth contribute to an
increasing emphasis of the community health centre
on community health and prevention?
- How well do home telehealth applications work in
this community?
- Indicators
- Staff time spent in curative versus preventive
servicesHealth personnel and tertiary provider views
of own and each others' competency and confidence
in application areasHealth provider and CHR views
of home telehealth success. Rates of home telehealth
usage and complications
- Berens River
- Evaluation Questions
- To what extent does telehealth improve case management
for patients with chronic disease?
- To what extent does telehealth produce more timely
diagnoses through more immediate access to specialists?
- What is the net effect of telehealth on workload
of nursing staff and translators?
- To what extent does telehealth produce cost savings
for escorted health-related travel?
- To what extent does telehealth help Berens Rivers
become a referral hub for other First Nations communities?
- To what extent does CME through telehealth improve
local capacities?
- Indicators
- Proportion of patients for whom outcomes are judged
in expert opinion to have been improved by telehealth
use Proportion of eligible patients receiving relevant
specialist care; Delays to access relevant specialist
care;Views of health personnel on quality and accessibility
of links to specialists Costs avoided : costs of
patient transfers avoidedStakeholder views of impacts
on referral patterns to Berens RiverStakeholder views
of competencies developed
- Southend
- Evaluation Questions
- To what extent does telehealth increase access
to specialists?
- To what extent does telehealth improve the specialist-health
centre information flow?
- To what extent does telehealth shift curative care
into the community and shift community resources
away from prevention programs?
- What is the net effect of telehealth on workload
of health staff and translators?
- Indicators
- Proportion of eligible patients receiving relevant
specialist care; Delays in access to relevant specialist
care;Views of health personnel on quality and accessibility
of links to specialists and on communication flowStaff
time spent in curative versus preventive servicesHealth
staff workload : active cases, patients per week,
time spent on non-transferred patients
|
