|
|
     |
||
 |
     |
 |
|
3.1 The Role of the Key Informant InterviewsThe interviews that we conducted with 30 key informants provide an important perspective on many of the questions in this formative evaluation. Key informants are often important sources of many types of evaluation information, but their perspectives are the most useful in a formative evaluation. The semi-structured nature of the key informant interview protocols allows us to focus on the particular area of expertise of each key informant. As a result, the interviews provide useful information on a wide range of evaluation questions. The key informant interviews have been conducted with individuals from a variety of backgrounds. These include:
Many of the key informants were also members of the Advisory Panel of Experts established by the OLT. This perspective meant that they were frequently able to provide detailed and well-considered responses to many of the questions. Other respondents were project partners with the OLT or representatives from organizations who had received funding through the Contribution Program. As a result, the interviews cannot be taken as an indicator of the degree of recognition of the work of the OLT. Participants were selected because they knew of the activities of the OLT and would be able to discuss issues of implementation. In a summative evaluation, a different strategy for selecting key informants would be likely. Most informants indicated that they had been involved with the OLT during most of its existence. Some respondents reported shorter periods of involvement frequently associated with applications to the Contribution Program. A series of visits by OLT staff with individuals working in the area of learning technologies in the early days of the program were mentioned as the initial contacts with the OLT. 3.2 The Role and Profile of the OLTAn assessment of the role of the OLT is provided by the responses of key informants about the key issues in the area of learning technologies. Profile issues reflect responses to questions about awareness, visibility, communications and the OLT website. Respondents regarded the key issues question as very broad, and it elicited a range of responses. Many respondents provided a series of issues with access to learning technologies being mentioned by a large proportion of the key informants. However, access had a variety of interpretations, with concerns being expressed about remote and rural learners and about persons with disabilities. The majority view was that although learning technologies open a wide range of new opportunities for learners, the short-term impacts might be to increase the degree of disparity in access to learning opportunities. Among the other issues that were raised, the following provide an overall sense of the views of the key informants about the core issues in the area of learning technologies:
Almost all respondents rated the OLT highly in terms of developing awareness of learning technology issues, of establishing a high and respected profile in this field and in terms of communicating to a wide range of individuals and organizations likely to be affected by new learning technologies. In answering this question about visibility and communications, many respondents pointed out that the OLT had been established relatively recently and that its potential range of influence was very large. The individuals and organizations most likely to be communicated with readily and effectively were those already working in the field. As a result, academics, organizations with related objectives and firms for whom learning technologies are of prime importance are most likely to have interacted with the OLT. The key informants, generally, are located in organizations with which the OLT has communicated effectively, or this reason, they are not the best sources of information on gaps. The OLT itself feels that its communications efforts could be improved substantially and has committed new resources to this end. Gaps clearly exist in the coverage of the OLT. Given the size and resource base of the OLT, this was described as inevitable. The most frequently identified gaps in coverage affected smaller and "less plugged in" community organizations and small and medium-sized firms, or example, some respondents noted that larger firms, active in the use of learning technologies for worker training with a corporate training and development function, would likely know of the work of the OLT. However, beyond this range of firms, there would likely be little recognition of the OLT. Some of the respondents who referred to the gaps above also noted that the OLT itself recognized these gaps and was dealing with them. Emerging key issues for the OLT are community learning and workplace learning. Given the large absolute numbers of small and medium-sized firms and of communities with needs in this area, the penetration rate of the OLT is likely to increase relatively slowly, but respondents noted that efforts are being directed to these areas. Key informants were also asked how effectively the OLT website communicated with individuals and organizations interested in learning technologies. Overall, the website was rated very highly by key informants. Many reported accessing it regularly, particularly for project summaries, and participating in on-line conferences. There were suggestions that it might benefit from more links to related sites and the provision of a site map. 3.3 Objectives and Their AchievabilityThe key issues reviewed in the previous section of this chapter provide a context for considering the objectives of the OLT. In this evaluation, the three general objectives of the OLT are to:
To achieve these three objectives, OLT has three main activities: (1) the Contribution Program, (2) the website and on-line conferences and (3) its offerings of demonstrations, workshops and forums. How closely do the objectives of the OLT match the key issues identified by key informants in Section B? The underlying theme of the key issue responses deals with the rapid process of change. This makes it difficult for both learners and "educators" to make the best use of what is available. This is essentially an issue of information. The rationale component of the literature review in Chapter II focussed on the role of information as an economic commodity and the problems that are faced when it is provided only through markets. Against this backdrop, OLT objectives can be interpreted as responses to information issues that are unlikely to be provided fully through markets. In promoting the effective use of learning technologies, there is an implicit assumption of less than entirely effective current use. This view is fully supported by the key informant interviews. There was a high degree of consensus that the objectives of the OLT are realistic and achievable. Some respondents specifically emphasized the importance of the learner focus in understanding the objectives of the OLT. This focus means that the OLT must be involved in a wide range of areas related to learning technologies. Some questions were raised about the breadth of the objectives relative to the size of the OLT and its resources. Another respondent noted that the objectives were designed to be achievable, noting that it would be possible to "promote," "support" and "increase" objectives without making a significant overall mark in a large field. One respondent raised the distinction between "goals" and "objectives," noting that objectives are often seen as being subject to measurement of the degree of achievement but that the OLT objectives would be difficult to assess quantitatively. On the research component of the OLT objectives, there were some views that supported "doing, in terms of content development and support" versus "researching." Related viewpoints questioned OLT support for university-based academic research that has other funding sources (NSERC, SSHRC) available to it. This issue was raised even by some informants who fully supported the research focus. These individuals raised "level playing field" questions about the research supported through the Contribution Program. Specifically, they pointed out the experience of university researchers in preparing grant applications and in research methodology. Even with the simplified OLT application procedures, concerns were raised about the inherent advantages of some competitors. As a follow-up to the question about the overall reasonableness of the objectives of the OLT, we also asked key informants to comment on the obstacles that the OLT might face in trying to achieve these objectives. Some respondents referred to their responses on the key issues. The rapid pace of technological change, software literacy, the hardware as opposed to the learner focus of many other players in this field and the broadness of the field were all mentioned as obstacles. Many key informants also noted that resource constraints were also a major obstacle for the OLT. Many specific obstacles were mentioned by key informants. Some of these obstacles include the absence of a national mandate or co-ordinated focus on learning technologies. Some respondents made related points in terms of the inherent political constraints given the relationship between lifelong learning and the capacity to learn provided to young learners. Another respondent focussed on the management challenge faced by the OLT. The issue for the OLT, according to this perspective, is to define its key competencies and responsibilities. This defines core activities to be carried out internally with other activities being contracted to specialized providers. This view suggests that even with growing resources, the OLT should not attempt to be all things to all members of the learning technology community. 3.4 Design and DeliveryKey informants were asked to provide their assessments of the most positive and the most negative features of the OLT's specific design and delivery issues. The most frequently identified positive feature of the OLT was the effectiveness of the staff. Other positive mentions were recorded for the website, the Contribution Program, the OLT's publications and research, and the effectiveness of the communications strategies of the OLT. Although several respondents said that they had no negative features to report, there were a number of specific features of the program that were described negatively. A representative selection of the negatives includes the following:
The design of the OLT provides for a federal government presence in the area of learning technologies. Many respondents referred to the related role of Industry Canada in operating the Community Access Program (CAP). Some respondents saw a complementarity between CAP and the activities of the OLT. According to this view, the hardware provided by the CAP can be used in communities, as part of Community Learning Networks, to pursue the learning objectives of adult learners. Among the key informants, there was strong support for a federal government role in the area of learning technologies. Many of the respondents also underlined the importance of the OLT's working effectively and closely with the provinces. They believe that the federal government should be involved in the clearinghouse, linking or information co-ordination role that they see the OLT playing. The OLT can reduce the extent to which people across Canada might end up working in isolation from each other. Some of these respondents referred to the desirability of a national strategy on learning, and several noted that virtually all countries, including other federal states, have a national focus in this area. It is clear that many other departments and organizations operate in the same general areas as the OLT. Provincial education departments, for example, have varying degrees of focus on using new learning technologies. However, their focus is not on the adult learners targeted by the OLT. Industry Canada operates the SchoolNet Program and the Community Access Program, but these programs are more hardware or technology-oriented than the OLT. These examples and many others that could be provided show that wasteful duplication is possible. However, most respondents argued that the initiatives of the OLT were either complementary to related activities or filled important gaps in the coverage of other groups. Several respondents suggested that "more co-ordination" would be desirable but recognized the practical difficulties in doing so. One respondent pointed to some overlap in the funding of research in this area by academics. Key informants were asked a series of questions related to the operation of the Contribution Program. This provided responses from a series of perspectives, since key informants had been involved with the program in a variety of ways. Several respondents had been assessors so that they had a detailed understanding of the review process, others had been successful applicants, and some had also been unsuccessful applicants. Other key informants, while familiar with other elements of the OLT, were not directly familiar with the operation of the Contribution Program. Edited comments on the funding decisions of the OLT and the review process include the following:
The preceding comments are intended to provide the flavour and wide-ranging nature of the responses. There clearly is not a consistent picture — keeping applications confidential may conflict, for example, with the other suggestion of linking related researchers. The process is regarded as "fair" but inherently favouring organized groups. The same criticism could likely be made of any funding agency that wishes to maximize effectiveness by having as much information as possible about the project plan and the likely results. Delays, if mentioned, were generally not viewed as being excessive. Many respondents did share the view that more targeting away from "pure" research would be advisable, although this view was not universally held. A key element of program design relates to the catalyst role of the OLT, particularly in terms of establishing partnerships with other organizations in the area of learning technologies. In some of the key informant interviews, questions were raised about precisely how "partnerships" should be defined. One respondent felt that partnerships consisted of formal, legal agreements to carry out some activity. Most other respondents, however, had less rigid definitions, including informal arrangements involving working together, sharing information, sharing resources, or establishing linkages. Most respondents were familiar with the networking role of the OLT and felt that it was highly effective. Others did provide information about more formal partnership arrangements. Some noted that it was likely premature to offer opinions on how effective these partnership arrangements have been. Overall, most respondents seemed to agree that the OLT has been very active in this area and that this is an appropriate role for it to play. 3.5 Success IndicatorsKey informants were asked a series of questions about program success. Many noted that it was early in the life of the program to provide clear answers to questions of this kind. In probing this area with respondents, the following issues were pursued:
This is clearly a lengthy and ambitious list of objectives or success indicators. Many respondents referred to attribution issues as a key problem at this point and more importantly in any future outcome evaluation. Many organizations are active in the area of learning technologies so that attributing subsequent changes to the activities of the OLT alone is difficult, or example, in the case of improving the ability of Canadians to adapt to changing labour markets, it seems clear the new learning technologies will be important. The OLT is working in this area but so are many other public and private sector organizations. It seems unlikely that it will ever be possible to estimate an overall impact here in terms of a changed degree of adaptability that can be attributed to the OLT. 3.5.1 Meeting Goals and Objectives Most respondents indicated that the OLT has made reasonable progress toward its objectives. There is a general perception that the OLT is headed in the right direction and has done well for a new organization. Another respondent noted that the OLT has made significant progress in terms of its three overall objectives. It was noted that a serious and consistent effort has been made in the direction of its objective. Other comments focussed on specific OLT activities with the general view that all of these were successfully promoting the objectives of the OLT.
3.5.2 Major Achievements Many specific items were referred to as major achievements. The dominant item, however, related to the work that the OLT has done in building networks of contacts in the field of learning technologies. Some respondents referred to this in terms of the OLT acting as a catalyst in bringing different organizations and individuals together through its conferences and related networking activities. Other mentions in terms of major achievements included the emphasis on a learner as opposed to a technology focus and its recent work in the area of community learning networks. 3.5.3 Increasing Awareness and Understanding of Learning Technologies Most respondents recognized the difficulty of answering the question dealing with the contribution OLT has made in increasing awareness and understanding in making learning technologies more effective. Many specific instances of OLT activities that increased awareness and understanding were referred to, but respondents were not sure how to extrapolate this to a higher level. More time and more information were referred to as necessary for providing a real answer to the question. When asked about specific OLT-supported projects that might have made learning technologies more effective, most respondents had no direct knowledge of the degree of learner satisfaction with the technology. Many were familiar with the OLT projects and in most cases argued that without the OLT funding, these projects would not have been carried out at all or would have been carried out on a smaller and less effective scale. 3.5.4 Increased Use, Awareness of, or Research into Learning Technologies Many respondents again pointed to the difficult nature of this question, particularly in terms of attributing changes to the OLT. The area of learning technologies is one in which there has been a rapid growth in activity from many sources, so that respondents were unsure about how to attribute changes to the OLT. Many specific things can be related to work funded through the Contribution Program, making it possible to trace out some direct impacts. Given the scale of the OLT, the response of one key informant was that the OLT contribution has been " small but worthy." This question relates to the rationale for the OLT program. That rationale is related to the fact that learning technologies are being developed so rapidly and that there are important issues regarding the quality and usefulness of these technologies. The OLT can be seen by adult learners as the federal government's window on these issues. As such, it will be more difficult for the OLT to affect national use and awareness, but research and promotion of learner-focussed approaches continue to be relevant objectives. 3.5.5 Gaps In Access and the OLT Focus on Access Issues This chapter previously identified access for a variety of groups, such as remote learners, as one of the key issues in the area of learning technologies. There are gaps in existing learning opportunities, and there were concerns that these gaps are likely to widen as new learning technologies are introduced. Most respondents recognized that many OLT initiatives are aimed at issues of access, with particular emphasis on rural areas and on the economically disadvantaged. A common view from the key informants on the OLT role here is that the issue has been correctly identified by the OLT as an important one, and work is underway. Many respondents also noted that the issue is much larger than can be dealt with effectively by the OLT alone. Some respondents referred to the recent OLT initiatives in the area of community learning networks as an important response to the issue of access. One respondent noted specifically that the Community Access Program is not sufficient and that approaches that focus on learning needs as well as hardware are important. 3.5.6 Labour Market Adaptability of Canadians Improving the labour market adaptability of Canadians is a core objective of all of HRDC's initiatives, including the OLT. Increased adaptability enhances Canadian competitiveness and minimizes the negative impacts of global change on the Canadian workforce. All key respondents recognized the crucial role of lifelong learning in promoting labour market adaptability. The efforts of the OLT focus on activities that can be described as R&D; in the area of learning technologies for adult learners. Many of these activities also have the potential to increase the effectiveness of workplace learning efforts. Key informants provided a wide variety of perspectives about the role of the OLT and about new learning technologies as they relate to the labour market. These include the following:
The key problem with assessing the success indicators of many programs is the issue of attribution. This is a particularly challenging issue in the case of labour market adaptability. The actual amount of adaptation in the labour market reflects both demand and supply side factors. The initiatives of the OLT can affect the supply side along with many other factors, most importantly the primary and secondary school systems. The OLT can have a direct influence on training provided to members of the labour force. Several examples of this were provided during the key informant interviews from respondents who had participated in the Contribution Program. In a future summative evaluation, it will be possible to trace specific links such as these; but the OLT's research, networking and related activities do not appear to generate impacts that will be readily attributable. 3.6 Program AlternativesThe key respondents who were interviewed have a wide range of expertise in the development and use of learning technologies. As a result, they are well-placed to provide input on ways in which the OLT program could be altered to allow it to achieve its objectives more effectively. The diverse perspectives of the respondents also mean that their specific priorities will be reflected in their suggestions. The single most common response to this question about program alternatives from the key informants was that more resources would allow the OLT to operate more effectively. Additional staff would allow the OLT to be more responsive and to maintain greater contact with its own projects and with work outside the OLT. A related comment was that with more resources, the OLT could provide increased help to applicants to the Contribution Program whose projects had potential but required further work. Other comments about different approaches to program delivery for the OLT included the following:
Most respondents did not focus on evaluation and accountability issues in providing opinions on how the OLT could alter its current activities. Among respondents who did mention this issue, there was a suggestion that the OLT should be active in developing indicators that would be relevant for a subsequent outcome evaluation. It was suggested that, in the area of learning technologies, there is too much emphasis on inputs (courses delivered, software installed) as opposed to outputs. Measures of change are not easy to develop, but some respondents thought that this was an area in which the OLT should be involved. The objective is to develop studies in which the impacts of learning technologies on individuals and organizations are measured. 3.7 Implications for the Process Evaluation of the OLTThis chapter of our report summarizes a wide range of perspectives on the operation of the OLT program. Most key informants are positive about the program concept and about most of its operational features. Their perspectives on program outcomes or objectives achievement are somewhat more limited because the OLT is relatively new and because of the difficulty of attributing outcomes to the activities of the OLT. Their outcome perspectives are either a reflection of what they expect from what they know about overall program activities or an extrapolation of what they know about the experiences of specific projects or activities. Our assessment is that the majority of the key informants rate the OLT very highly in all of the areas in which we questioned them. The key issues in the area of learning technology identified by the respondents fit well with the objectives and activities of the OLT. The goals and objectives of the OLT were viewed as reasonable and achievable, although many respondents noted that resource constraints were a major obstacle to objectives achievement. Respondents were positive about OLT staff and most program activities. When asked for negative aspects, they mentioned issues such as the Contribution Program's being too complicated for community groups, not enough work with the private sector and too much "pure research." In the context of these interviews, many process issues were raised by the key informants. Related to this, many respondents focussed on the resource constraints of the program. It was clear that many of the respondents felt that this would limit the extent to which the program could fully achieve the ambitious objectives that have been established. Among the key informants, there was strong support for a federal government role in the area of learning technologies. It is clear that many other departments and organizations operate in the same general areas as the OLT. However, the consensus view of respondents was that the initiatives of the OLT were either complementary to related activities or filled important gaps in the coverage of other groups or levels of government. With regard to the Contribution Program, there were many differing points of view. Although very positive about the program as a group, the key informants had a number of ideas for altering the way it operates. These included doing more to link proponents with related projects, adapting the process to make application easier for client groups, targeting more effectively, and limiting funding to applied work as opposed to pure research by academics. The process was regarded as fair, but several respondents were concerned that it favoured more organized groups. In commenting on program success, most key informants were very positive. The list of objectives or success indicators is an ambitious one, and many respondents referred to attribution issues as a key problem at this point and more importantly in a subsequent outcomes evaluation. Many organizations are active in the area of learning technologies, so that attributing subsequent changes to the activities of the OLT alone is difficult, or example, in the case of improving the ability of Canadians to adapt to changing labour markets, it seems clear the new learning technologies will be important. The OLT is working in this area but so are many other public and private sector organizations. It may never be possible to estimate an overall impact here in terms of a changed degree of labour market adaptability that can be attributed to the OLT. The same point can be made for other success indicators of the OLT. Key informants had a variety of opinions on measures that could improve the effectiveness of the OLT. In addition to more resources, respondents mentioned finding better ways to disseminate information, more co-ordination with Industry Canada, more influence within HRDC, more work with small and medium-sized firms, a lower research hurdle for community group applications to the Contribution Program, more private sector orientation, more work in the area of needs assessment, an improved communication strategy and more continuing focus on evaluation and accountability issues.
| |||||||
![[Previous Page]](/web/20060127014812im_/http://www11.hrdc-drhc.gc.ca/edd-img/back.gif){kind=small}
![[Table of Contents]](/web/20060127014812im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif)
![[Next Page]](/web/20060127014812im_/http://www11.hrdc-drhc.gc.ca/edd-img/next.gif){kind=small}