National Native Alcohol and Drug Abuse Program (NNADAP) -
General Review 1998 - Final Report
Results of the NNADAP Review
Research and Development
In the original plan for NNADAP, the research and development component of
the program was intended to provide the means for determining levels of the
various addictions within First Nations and Inuit communities, assisting in
the development of effective prevention and intervention strategies, and in
implementing an information system that would provide for an ongoing evaluation
and research capacity. Part of the presumed rationale for this would have arisen
while researching necessary data and information upon which to develop the
Cabinet Document and the subsequent Treasury Board submission for NNADAP. It
is interesting to see the degree to which assumptions were made about the level
of care and the degree to which extrapolations were used to determine need
and level of services required within the proposed programs.
It is interesting to note, as well, that in conducting the necessary literature
search in Aboriginal addictions prior to developing the review framework, the
former NNADAP research and development program has left a visible legacy of
studies that can be used to study aboriginal addictions.
While there may have been operational considerations to decentralizing funds
formerly allocated to Research and Development to regional operations, in retrospect,
it would seem to have been more strategic to refocus the Research and Design
component from essentially a proposal driven process to more of a directed
research program which could develop and carry out some of the basic research
needs faced by addictions programs.
There was an important opportunity within the First Nations longitudinal
health study which could have resulted in community based collection of utilization
of various substances. In the current version of the study, there are nosystematic
incidence related questions. This is a curious circumstance given the detailed
focus of the study on other more intrusive areas such as sexual abuse of children.
The conduct of pertinent surveys could assist in anticipating areas emerging
as addictions needs or simply as an educational tool to use in educating First
Nations community members concerning trends affecting them as individuals,
their children and their community. This would include issues arising out of
recent reports issued by the Auditor General and potential problem areas of
prescription drug abuse and the rising concerns regarding forms of gambling
in particular bingo.
If one applied the benchmark envisioned in the cabinet document that the program
in its mature state should devote 5% towards research and development, this
would amount to $2.75 million on an annual basis out of a total estimated $55
million dollar program. While it would be obviously difficult for Health Canada
to simply reconstitute this fund out of thin air, there would be an advantage
for both Health Canada and First Nations and Inuit organizations to oversee
a partial reconstitution of this budget and to combine this amount with funding
anticipated by Health Canada to establish an Aboriginal Health Institute(s)
and centers of excellence recently announced as part of the government response
to the Royal Commission.
Information Systems
The NNADAP program would benefit from outcome information, if the data could
be obtained systematically during each stage of assessment, treatment and follow-up.
This information would be invaluable for ensuring that the various persons
involved throughout the treatment process carry out their respective responsibilities.
Systematic and routine evaluation of program components would ensure improved
and continued effectiveness.
There is an important opportunity to build systematic follow-up and client
outcome information into an overall health information system targeted at the
community level. In the Ontario region of Health Canada, a comprehensive community
based health information system has been developed, which has a component for
alcohol and drug abuse. This registry could be modified to schedule follow-up
with very minor modifications and could also be utilized to track client progress.
Of considerable significance to this study, is the potential of this system
for networking with other providers within the health system such as community
health nurses and mental health providers.
Since Medical Services Branch has obtained new resources to implement the
H.I.S. system across the country, it would seem logical that such a system
could accommodate the modification described above. It would be essential to
have this system cover the 49 treatment centers within the same networking
plan. Such a system would be a critical means of tracking follow up to treatment
programming, as well.
In addition, although there will be sufficient challenges in implementing
the scenario described above, there should be contact made with the Department
of Indian Affairs with respect to systems in place for Child and Family services
programs to determine whether there are opportunities to review existing or
future information systems to explore case-working and coordination opportunities.
Program Evaluation
Program evaluation is a process used to determine whether or not the objectives
of a program have been achieved. It should be an integral part of the program
planning process rather than an afterthought. There are two types of program
evaluation: formative (or process) and summative (or impact) evaluation. Formative
evaluation examines activities associated with the ongoing operations of a
program. Summative evaluation examines activities associated with the more
long-term effects or impact of a program. Both types of evaluation are important,
because conducting one type ofevaluation while excluding the other type could
result in incorrect conclusions being made about the performance of the program.
In order to perform effective program evaluation, the program elements must
be very well defined. There must also be measurable objectives developed, which
describe the change that one would like to achieve in a particular program,
as well as the magnitude of the change and the time frame within which the
desired change is to be achieved. In addition, baseline data should be collected
to provide a snapshot of the situation that existed prior to the implementation
of the program. Each objective must be associated with realistic outcome measures
or performance indicators, and a data collection process must be in place to
provide reliable data for ongoing monitoring of program performance, as well
as for subsequent comparisons with the baseline data.
A portion of this General Review of the National Native Alcohol and Drug Abuse
Program focused on the issue of program evaluation and it identified many gaps
and inherent weaknesses this General Review of the National Native Alcohol
and Drug Abuse Program focused on the issue of program evaluation and it identified
many gaps and inherent weaknesses
Findings
- There is no standardized approach to the collection of reliable
data for ongoing monitoring of program performance at the community level.
- Survey respondents and participants during the on-site community
visits identified the need to develop a systematic approach to the evaluation
of the effectiveness of the NNADAP program.
- Much of the evaluation of NNADAP that has occurred has focused
on process issues (e.g., bed occupancy rates) rather than actual patient
outcome.
- NNADAP Treatment Centers tended to have more structured evaluation
processes than did the community-based elements of the NNADAP Program.
However, treatment center evaluations were more focused on program design
rather than actual patient outcome measures.
- At the regional level, program evaluation does not appear tobe
a priority. In some regions like Saskatchewan, however, a formal process
of evaluation has been established to assist in the ongoing monitoring
of program activities so those problems can be addressed as they arise.
Again the focus tends to be more on process rather than outcome.
- At the national level, only the Treatment
Activity Reporting System (TARS) and expenditure data are
available for analysis. No direct focus has been placed on the issue
of effective program evaluation, particularly as this pertains to the
assessment of the overall impact of the NNADAP Program.
In order to address future requirements for effective program evaluations,
Health Canada should support development of a training package on program evaluation
which could be used to train NNADAP staff and treatment centre and at community
level to perform effective program evaluation. This package should address
both process evaluation and impact evaluation. There must be sufficient resources
to assure that staff have the necessary training.
Health Canada in collaboration with stakeholder representatives should develop
a core list of performance indicators to be used by regions to conduct a process
and impact evaluation of the NNADAP on an annual basis. This list could be
adjusted to satisfy local requirements. However, a core list of indicators
would facilitate a regional comparisons of program performance and would allow
Health Canada and First Nations and Inuit organizations to have a national
perspective of the impact of the NNADAP.
There is an obvious need for reliable computerized data collection processes
that will provide data on an ongoing basis for case management and for program
evaluations. Health Canada with the stakeholder group should establish a working
group to examine data requirements and potential sources of data for effective
case management and ongoing evaluation of the NNADAP. For example, this group
could examine the possibility of using abuse profile subsystem of the National
Health Information System as one of the potential sources of data for NNADAP.
It is imperative that any computerized system identified to collect reliable
data should also provide data for NNADAP staff for case management purposes.
In fact the primary focus of such as system should be to support the day to
day activities of the NNADAP staff. The availability of aggregate reports for
programplanning and evaluation should only be a secondary benefit of the system.
If the system is not capable of supporting the work of NNADAP staff in case
management activities, there will be little support for the system at the community
level and the availability of data for program evaluation will be significantly
diminished.
|
