Home : Reports and Publications : Audit & Evaluation : Evaluation of the ABSN
Evaluation Methodology
Data Analysis
Following data collection, qualitative data was entered into Microsoft Word. In some cases, this involved transcribing the entire content of audio taped focus groups, whereas in others the notes made by the evaluator during the session were entered. This qualitative data was analyzed using traditional content analysis techniques (Krippendorff, 1980). Comments were separated by region and were grouped according to main themes that emerged from the data and through reference to the DCM. Emergent codes were discussed among the evaluation team and adjusted, or clarified, as necessary.
Findings from the qualitative analysis were then summarized in this report according to the topics identified in the Data Collection Matrix and Logic Model.
Evaluation Limitations
Several limitations are associated with the research methods employed in this study. As a result, the findings presented in this report should be understood with the limitations noted below in mind.
Document Variability
The only documents available for the document review were those obtained directly from the regions or publicly available online. Therefore, the document review could only include those documents the regions felt were important for the evaluation and in some regions, it was difficult to obtain relevant or current documents. Because the regions provided the evaluators with such a wide range of disparate documents, comparisons also became difficult. Although this was to be expected, given the variability in program structure and implementation across the four regions, it is nonetheless a limitation of the data collected for the review. Furthermore, the lack of clearly identified outputs based on shared definitions made assessing the concrete impact of the initiative considerably more difficult.
Selection of Focus Group Participants
The ABSN programs themselves selected the participants for both the ABSN Business Users focus groups the ABSN Partners/ Business Intermediary focus groups. It is likely that this process introduced a selection bias that influenced the responses obtained. Participants were certainly aware of ABSN and able to access its services or comment on its community impact. As a result, it was not possible for the evaluators to accurately assess the level of awareness and accessibility in the Aboriginal community as a whole. In addition, ABSN programs would be more likely to select participants for the Business User focus groups whom they knew and who may have been more satisfied with the program than those who may have accessed services only once. The ABSN Partners/ Business Intermediaries focus groups were frequently attached to other scheduled activities; thus participation was likely influenced by group members’ ability to attend these other functions.
The practicalities of a focus group may have also affected responses to a certain extent. Prior to the evaluator’s visit, ABSN regional staff members arranged the groups. Participants may not have felt comfortable providing complete, accurate responses in front of their peers and or in front of an ABSN staff member (as one sometimes attended these activities for information purposes). As such, the accuracy of focus group data was limited by the evaluators’ ability to judge the accuracy and completeness of responses.
Personal Recollection
As interviews and focus groups are retrospective in nature, they are subject to the quality of personal recollection and level of bias demonstrated by participants. Therefore, the accuracy of data obtained from focus groups and interviews was limited by the ability of the evaluators to determine the accuracy and completeness of the responses provided.
Limited Number of Cases
As only a small number of interviews were conducted, and as participation in focus groups was fairly limited as well, the data that could be included in this report had to be carefully scrutinized in order to protect individuals’ privacy. As a result, some useful information may not have been included.
In many cases, however, the limitations of individual methods are offset by the use of multiple sources of data, multiple methods of data collection, and the use of triangulation methods to validate findings. The Evaluation Steering Committee, made up of a variety of stakeholders (but excluding partners and users), did review a draft version of the data collected in this study and provided feedback and correction. In addition, this report was submitted to the Committee in draft form for final review. These feedback loops have added both validity and credibility to the findings reported here.
The following four chapters provide the evaluation study findings for each region, presented from east to west. Each chapter is organized according to the topics in the logic model and DCM for which information was obtained.