Home : Reports and Publications : Audit & Evaluation : Evaluation of WD's Sustainable Development Strategy 2000-2004
3.0 Methodology
To assess the WD SDS 2000, the evaluation focused on four key areas (i.e., Relevance, Success, Effectiveness and Lessons Learned/Future Directions). A series of evaluation questions was developed to address whether WD's SDS 2000 was relevant in terms of what it was trying to achieve, if WD was successful in imp lementing SDS 2000, and whether their approach to implementation was effective. In order to answer these evaluation questions, a series of indicators were developed for each, and data collection was focused on addressing these indicators. Key evaluation questions and their respective data sources are highlighted below in the Data Collection Matrix.
Data Collection Matrix| Evaluation Area | Evaluation Question | Sources of Data |
|---|---|---|
Relevance |
|
|
Success |
|
|
Effectiveness |
|
|
Lessons Learned/Future Directions |
|
|
Data collection consisted of two major processes; a review of documents related to WD'S SDS 2000 and its development, and telephone interviews with relevant WD staff.
Document review
Documents for review were chosen in consultation with the WD SDS Evaluation Project Team. Relevant information was extracted and summarized. Information was then linked to the key evaluation questions in the Data Collection Matrix. In particular, documents were examined to identify how activities and outcomes were linked with SDS objectives, and how these objectives were linked with CESD expectations and the WD mandate.
The following documents were reviewed as part of the present evaluation: Western Economic Diversification Canada:
Commissioner of the Environment and Sustainable Development:
Auditor General of Canada:
Treasury Board Secretariat and Government of Canada:
Other:
|
Telephone interviews
Telephone interviews were conducted between July 16 th and August 13 th, 2003 with:
|
Interview guides were designed in consultation with the WD SDS Evaluation Project Team. The SDS Evaluation Project Team also identified key staff to be interviewed. Respondents included Green Team members and SDIT members as both groups are responsible for delivering parts of the Strategy. In addition, five managers and project officers with WD who do not work directly on the sustainable development file were interviewed in an attempt to gain an "external" perspective on the Strategy.
Data analysis
Qualitative data resulting from open-ended interview questions was entered into QSR NUD*IST Vivo (NVivo) and analyzed using traditional content analysis techniques (Krippendorff, 1980). That is, comments were grouped according to main themes that emerged from the data and through consultation with the Data Collection Matrix.
Documents and interview data were analyzed by different researchers and then triangulated. Emergent themes and key findings were discussed among members of the research team and clarified as necessary.
Limitation of the evaluation
There was no opportunity to systematically examine the impacts of the Strategy external to WD (e.g., network partners and funded projects). A list of funded projects that contain an element of sustainable development was reviewed along with summaries of key projects that were chosen by the regions. These summaries contained information on the projects' purpose and objectives, background, applicant profile, relation to sustainability, and project costs/financing. However, the impacts of these projects could not be examined based on these sources, and as WD has not yet identified expected impacts (and associated measures) due to the long-term nature of the projects for these sustainable development activities, one cannot ascertain whether the Strategy is really producing any significant results. For future evaluations, information on external impacts needs to be obtained from sources such as the GX database, reports on completed projects, and interviews with network partners and Western SMEs.