|
|
     |
||
 |
     |
 |
|
By way of introduction to this chapter, and to the remainder of this report, it will be helpful to provide a definition of learning technologies, or the purposes of this report, this definition is as follows:
Learning technologies are information and communication technologies used to support learning. New learning technologies include the Internet, computer networks, CD-ROM, video and computer conferencing, interactive television, computer-assisted instruction, multimedia, animation, virtual reality and simulations. 2.1 Review of Project Files2.1.1 The Contents of the Files The files contain the original proposal and varying amounts of supporting material. The application procedure has become more formalized since the first round. A detailed Application Form — SDE 0032 (05-96) — is now used. Later files also contain the proposal review forms and, in some cases, specific comments by the reviewers which led to modifications in the application. Communications on the acceptance of the project, instructions, minutes of meetings, media releases and other correspondence including e-mail messages are also in the files. "Problem" projects have more correspondence than ones with fewer difficulties. All files contain the original Contribution Agreement which constitutes the contract. The following clauses are included:
The Contribution Agreement also includes the following schedules as part of the complete contractual arrangement:
Some of the files included a Telephone Monitoring Report which has, to date, been done for selected projects. The purpose does not seem to be to prepare a formal report, but simply to identify potential problems or "stars" — a more recent addition is a question about the possible publicity value of the project. A formal monitoring system, including a policy on the frequency of calls and site visits, is being put into place. The Telephone Monitoring form includes the following:
Items requiring follow-up are red flagged. A box is checked if the interviewer thinks a site visit may be advisable, and there is space to explain the reason for this recommendation (whether there appears to be trouble, or if the project might be an interesting one to observe for promotional purposes). As mentioned above, the more recent files contain an In-Depth Assessment of Proposals, from at least two reviewers. This assessment includes:
Recommendations
The main reasons a project would NOT be recommended are:
The files also contain financial information, including claims for progress payment, the compulsory Monthly Cashflow Forecast, journal vouchers for expenditures and a Summary Sheet of claims received and payments sent. Claims, which are filed quarterly, are accompanied by brief progress reports, which are also bound in the file. The Monthly Cashflow Forecast (from April to March) is to be submitted within 30 days of the agreement being signed. The categories included are:
2.1.2 The Initial 68 Approved Projects For the purposes of this evaluation, project files were reviewed for the 68 projects which had received the OLT funding as of April 1998. Of these projects, 27 started in 1996. The remainder started in 1997. As of August 1998, only nine of these projects had received approval of their final reports to the OLT. Appendix A to this report provides brief descriptive profiles of these projects as prepared by the OLT. Selected characteristics of these projects are discussed below. Sponsor types. It is useful to examine the types of organizations and institutions which received project funding in this initial period of the program's operation. The sponsors of the 68 initial projects can be classified as follows:
Post-secondary educational institutions clearly dominated the first two waves of funded projects. Sponsor locations. The breakdown by province of the locations of the sponsors of the initial 68 funded projects is as follows:
Identified target groups. While most of the funded projects identified one or more learnware developers, administrators or users among their target groups, some projects also made explicit reference to other target groups. The breakdown of these target group references is as follows:
2.1.3 Rejected Applications ARC Applied Research Consultants conducted a review of files of rejected applications from the three application periods, selected more or less at random. An attempt was made to pull files from applicant organizations located across the country. In total, 38 files were reviewed: 13 from 1996, 10 from 1997, and 15 from the most recent proposal call. The 13 files chosen for review for the 1996 proposal call could be broken down as follows:
To be eligible for the OLT funding, a project had to meet one or more of the following objectives of the contribution program:
As expected, a number of applications from the initial call for proposals (July 1996) appeared to have been rejected because the application did not meet these basic eligibility requirements. A number appeared to be programs that were already being designed and to which learning technologies were added or enhanced to increase the possibility of funding from the OLT. While most files indicated more than one deficiency in the application, the main reasons for the rejections were as follows:
Notes in the file indicated that three of the 13 randomly selected applicants had called to follow up regarding the reasons for the proposal being rejected. One had also called before the end of the review period and was given information on shortcomings in the proposal. This tended to confirm the OLT view that a relatively small number of applicants follow up on reasons for rejection. Due to limitations of staff resources, the OLT provides a generic letter of rejection which states the number of applications received during the proposal call, and the number accepted. This apparently provides enough information for most applicants. Those who phone or write for additional information are provided with a summary of the reviewers' comments. The 10 files chosen for review for the 1997 proposal call could be broken down as follows:
Fewer of these applications appear to have been rejected for not meeting basic eligibility requirements, which suggests that OLT objectives may have become better known or understood by applicants. Lack of innovation in the use of learning technologies, lack of addition to the knowledge base and limitations in the scope of impact were the most frequently cited reasons for rejection in this group of proposals. The 15 proposals chosen for review from the 1998 call were as follows:
While most files indicated more than one deficiency in the application, the main reasons for the rejections were as follows:
In order to provide more detail on the views of rejected applicants for OLT Contribution Program funding, telephone interviews were conducted with a random sample of 10 such individuals. The interviews followed a modified version of the key informant interview guide (described below). The results of these interviews did not differ significantly from those of other persons contacted for this evaluation and are reflected in the discussion of these broader findings. 2.2 Literature ReviewThis literature review is confined to areas where it provides direct input into the evaluation questions. The key issues that are pursued are the following:
There are two thrusts for the literature review. The first provides a brief overview of the status of learning technologies in the Canadian and international context. The second is more narrowly focused on the economics of learning technologies, the rationale for support and the potential impacts on competitiveness and on worker productivity. 2.2.1 The Role of Learning Technologies The primary focus of the OLT is on new learning technologies as defined earlier in the text. The literature indicates that there is a variety of such technologies that have evolved historically but that the pace of change has accelerated in recent years. New technologies usually bring both opportunities and challenges, and the OLT has as a major objective the promotion of the effective use of learning technologies. Government decision-makers in the area of training policy and educational policy for adult learners need current and effective input on the potential roles of new learning technologies. In its recent report entitled Adult Learning and Technology in OECD Countries (1996), the Organization for Economic Co-operation and Development (OECD) argues that these decision-makers must "understand and anticipate" the changes that are taking place and will take place. In this context, the OLT can be viewed as the "window" of the federal government in this area that many commentators argue will have important impacts on Canada's labour force and its overall competitiveness. A number of perspectives emerge from the OECD reviews of learning technologies. The 1996 report cited above argues that "the most difficult and expensive issues are human and not technical," that "the most appropriate technology is not always the 'leading edge' one" and that "the technologies are unlikely to reduce social or educational inequality without intervention in the market" (p. 16). In developing the case for new learning technologies to be applied more intensively to meet the needs of adult learners, the OECD, in its report Adult Learning in a New Technological Era (1996), makes the case that "the use of computers, video and telecommunications for adult learning is becoming essential because of the ways such technology is used in the world outside education" (p. 141). This perspective makes the dividing line between the learning technology and what is being learned less clear than may be the case for earlier learning technologies. Again in support of the "window" rationale for the activities of the OLT, this OECD report asserts that "national decision-makers need a set of indicators about the need for change, the readiness for change, the process of change and the outcomes of change" (p. 145). Two other recent OECD publications reflect many of the same themes. Learning Beyond Schooling (1995) and Lifelong Learning or All (1996) deal with infrastructure and distance issues as they affect the cost of and access to learning technologies. These publications also deal with the "window" issue, a theme that recurs throughout this section of our report. A key requirement of governments in the area of new learning technologies is information both to develop new policies and to manage existing ones. In Lifelong Learning, it is argued that "information on needs and 'what works' derives less from theory than from sustained enquiry among, and interaction with, users, employers and trade unions" (p. 217). Roberts, Rossiter and Duncan (1996) report on evaluation results from a publication entitled Lifelong Learning on the Information Highway developed with financial support from the OLT. This report focuses on many of the "lessons learned" from using interactive learning technologies with adult learners in a variety of locations. These lessons, relating to both learning and technology issues, include the following:
An economic perspective on the impacts of new learning technologies is provided by Stager (1995). This overview report focuses on technology-based learning and training and was prepared for the Working Group on Learning and Training of the Information Highway Advisory Council. Stager deals with a variety of issues in this report, including changing demand and supply patterns for skilled workers, the economic benefits of lifelong education and training, and the fixed and variable cost relationships associated with new learning technologies. In this survey paper, Stager points out that traditional and technology-based learning and training differ importantly in their cost relationships. With traditional learning technologies, fixed costs are smaller and costs vary with output through replication at a variety of sites with increasing staff resources, Fixed costs for technology-based systems are generally higher, associated with both hardware and software costs. However, if there is a large market for such technologies, variable costs and total unit costs can be much lower than for traditional technologies. These unit cost relationships and their implications for implementing new learning technologies are also discussed in Bates (1995). Stager refers to the large literature on network economics and network externalities as being relevant for these learning issues but points out that not much research has been done in this area. From the point of view of the OLT, the key conclusions of the Stager Information Highway study are captured in the following quotation: "Throughout the research background of this report, there runs a common theme: data are not available, research has not been done, processes and linkages are not well understood." Specifically, what is being referred to here is the system of adult education and training as it is affected by the introduction of new learning technologies. The Stager view is reflected in a variety of publications from the Information Highway Advisory Council. This literature appears to provide a rationale for a "window" and catalyst type of organization like the OLT to contribute to advances in knowledge and understanding in this area and to provide this and related information to Canadian policy-makers. This component of the literature review has touched only lightly on many of the core issues associated with learning technologies and the role of the OLT. The OECD studies that are cited contain extensive bibliographies with more supporting references. In the Canadian context, the work of the OLT appears consistent with the recommendations and directions of the Information Highway Advisory Council. As well, there appears to be a consensus in the international context of the OECD that policies on these important issues of access, co-operation, research and standards require public input. The rationale for the OLT and its activities and objectives appear consistent with this literature. 2.2.2 Economic Rationale Issues This section of the report provides an economic perspective on the development of the Office of Learning Technologies. There are a number of possible approaches to the assessment of the economic rationale for the development of this initiative. In this report, the focus is on the rationale for public sector promotion of these efforts and on the way market forces might operate in the absence of government support. The standard economic efficiency rationale for government programs of this kind relates to the existence of an externality that leads private markets to misallocate resources relative to the social optimum. An externality, sometimes referred to as a spillover, is a component of overall benefits or costs to society that does not correspond to the benefits or costs of decision-makers, or example, if society benefits from a new learning technology to an extent greater than the developer who incurred the costs of doing the research and bringing it to market, then the social gain exceeds the private gain, and the gap is the externality. This section of the report investigates the efficiency rationale of the OLT primarily in terms of research and information. The information issue is relatively straightforward. Information as a commodity has important problems of appropriability that generally imply that there may be scope for government intervention to promote efficiency in its provision. The appropriability problem means that individuals or firms who generate information related to learning technologies or their effectiveness will be unable to capture all of the benefits of that information. (Appropriable is the term that economists use when referring to the extent to which the producer or creator of something of value is able to capture all the economic returns from its use.) The economics literature recognizes that information is a public good in that the benefits of its production are available to non-payers and payers alike, so that markets will under-invest in its production. These externality-related issues provide the rationale for a wide range of government activities extending well beyond the OLT. This paper considers those rationale issues but attempts to go beyond them as well in terms of some of the specific activities and anticipated impacts of the OLT. The related issues pursued in this paper are:
In the case of the activities of the OLT, markets may fail to provide the best outcome, relative to the ideal, in a number of areas. This paper focuses on the economic rationale for these activities, particularly with regard to the following:
The following paragraphs deal with each of these rationale issues. Information as a commodity is likely to be under-provided relative to the theoretical optimum as Arrow (1962) and others have argued. The literature on innovations, including the work of Nordhaus (1969), also suggests potential problems of markets failing to achieve the social optimum. In the case of both information issues and innovation, the literature on market failure suggests that, in such cases, there may be a potential for public sector intervention to promote improvements in economic well-being. Research activities are intended to produce knowledge or information. In the case of industrial R&D; labs, that knowledge can produce new products or processes. However, innovations can extend beyond the production level to include the process of learning and how it is organized or operated. The economics literature now recognizes that information, or knowledge, has important properties that are likely to lead to its under-provision through private markets. This characteristic of knowledge is likely to lead to under-provision in a wide range of areas because knowledge, generated through costly investments, is not completely appropriable. In a market economy, firms will fund activities to generate new knowledge only when the return from doing so is competitive with other investments. That is, they will invest in such activities only up to the point at which the marginal private benefits equal the marginal private costs. Some research activities might be profitable if all users who benefit could be made to pay, but they will be unattractive if these users cannot be made to cover their share of the development costs. It is also the case that society has an interest in the widespread diffusion of new knowledge. Unlike the case of private goods, the use of knowledge by one firm does not reduce the amount available for other users, From a social perspective, the marginal cost of using knowledge that is already produced is zero. The efficient use of any commodity with a marginal cost of zero is one in which the price to the marginal user equals the marginal cost of zero. This creates an obvious dilemma for private production because development costs can never be recouped if the price is zero. That is, the optimal price from the point of view of producing this kind of public good is positive, whereas the optimum price once produced is zero. Comparable issues of the rationale for government support are also faced in the area of R&D.; The federal government and some provinces provide a variety of support programs, including both direct grants and tax credits for industrial R&D.; In spite of intellectual property laws, the benefits of R&D; appear to spill over to non-performers. These spillovers represent the appropriability problem and are widely recognized as the main source of the economic rationale for government intervention. The evidence of Bassi (1994) is directly related to the rationale for the activities of the OLT that are related to labour market training. The focus of the Bassi study is on education programs within firms. This is instructive because these programs typically provide the most general forms of training, the most portable, and the ones that are most likely to be under-provided through the market. Bassi determines that within her sample of 72 firms, the following was true:
The primary barrier to training that was identified by Bassi related to the high fixed costs of developing the content and methods of instruction for training programs. Bassi argues that the public goods properties of the front-end components of training programs means that they will be under-provided without public support, presumably because they are easily imitated and difficult to protect through copyright. Responding firms identified employer networks as the best vehicle to share and spread fixed costs in the development of these front-end components of training. Learning technologies in particular appear to face the barrier created by training costs. More recent information on measuring returns to workplace training is provided by Mavrinac and Jones (1995). The area of measuring returns remains somewhat of a puzzle, since we have evidence that many firms are devoting more resources to training and development without requiring quantitative assessments. The workplace training focus of the OLT is a reflection of the important structural changes in the world economy that continue to influence developments in the Canadian labour market. Harris (1993) and other economists have noted that globalization reflects continuing world-wide increases in the ratio of trade to GDP. Firms are investing in production facilities around the world and will source different products or components from the most cost effective locations. The result, in Canada, has been a reduction in opportunities and a reduction in relative wages for workers in the lower skill categories. Documenting the labour market effects of globalization directly requires data that are often difficult to acquire, but some U.S. data are relevant in this regard. Murphy and Welch (1992) show large recent earnings gains for college graduates relative to high school graduates, but the more detailed analysis of Katz and Murphy (1992) may be more relevant in terms of training implications. These authors show that within educational levels, earnings inequality also has increased significantly since 1970. At least one factor to explain this increased inequality of earnings within education categories consists of different firm and industry training approaches to globalization. As described by Harris (1993), the reaction of some firms to increased competitive pressures is to contract for more inputs from lower wage sources and lay off their least skilled workers. In other cases, however, the reaction has been for firms to adopt new technologies and to train their workers to use them. In other words, firms responding to global competition have reduced their demand for unskilled workers relative to skilled workers with obvious implications for relative wages. Bartel and Lichtenberg (1987) document the importance of having trained workers for the successful transfer of new technologies to firms. More recently, Bartel (1995) provides firm-level evidence linking training, wage growth and worker performance on the job. In cases like this, OLT activities that promote such technology-based training initiatives may be the source of both better and more jobs in Canada than would exist without training. Governments are elected to provide programs desired by voters, and there is a case to be made that developing mechanisms to provide greater workplace learning opportunities for Canadians is such a priority. This could be desired on market failure grounds alone, but equity or income distribution issues may also be important. Gunderson (1974) discusses the training externality in the context of merit goods which can include an income distribution concern. In the context of changes that are occurring in low skill labour markets, training efforts, depending on how they are targeted, may have an income distribution rationale. One important question that relates to much of this literature deals with evaluating the impacts of the OLT. This section of our report focuses on rationale issues, but it would have been most useful to contrast stated rationales and activities with the actual activities and outcomes of the OLT. Data are not yet available to do this in this formative evaluation. In related work in the United States, Mavrinac and Jones (1995) review much of the literature on financial and non-financial returns to workplace innovations. These returns are measured at the level of the firm, and the overall conclusion of the research is that workplace innovations do have important returns that can be measured. In the case of the OLT, however, there is a serious attribution problem that will be a key issue in designing the outcome evaluation. The ultimate impacts of the OLT on workplace learning, in which we are interested, will be observed at the level of firms and their workers. However, it will be difficult to disentangle firm initiatives, which are independent of the input from the OLT, from changes that can be traced back to the OLT itself. ReferencesArrow, Kenneth J. (1962). "Economic Welfare and the Allocation of Resources for Invention." In Richard R. Nelson, (ed.), The Rate and Direction of Inventive Activity. Princeton: Princeton University Press. Bartel, Ann P. (1995). "Training, Wage Growth and Job Performance; Evidence from a Company Database." Journal of Labor Economics, v. 13, no. 3, pp. 401-425. Bartel, Ann P., and . Lichtenberg, (1987). "The Comparative Advantage of Educated Workers in Implementing New Technologies." -Review of Economics and Statistics, v. 69, no. 1, pp. 1-17. Bassi, L.J. (1994). "Workplace Education for Hourly Workers." Journal of Policy Analysis and Management, v. 13, no. 1, pp. 55-74. Bates, A.W. (1995). Technology, Open Learning and Distance Education, London: Routledge Studies in Distance Education. Becker, G. (1964). Human Capital. New York: Columbia University Press. Betcherman, G. (1992). "Are Canadian Firms Under-Investing in Training?" Canadian Business Economics, v. 1, no. 1, pp. 25-33. Grossman, Gene (1992). "Promoting New Industrial Activity: A Survey of Recent Arguments and Evidence." OECD Economic Studies, Paris. Gunderson, Morley (1974). "The Case for Government-Supported Training Programs." Industrial Relations/Relations Industrielles, v. 29, no. 4. Harris, R. (1993). " Globalization, Trade and Income." Canadian Journal of Economics, v. 26, no. 4, pp.755-776. Katz, L., and K. Murphy (1992). "Changes in Relative Wages, 1963-1987: Supply and Demand actors." Quarterly Journal of Economics, v. 106, pp. 35-78. Mavrinac, S.C., and N. R. Jones (1995). The Financial and Non- Financial Returns to Innovative Workplace Practices. Harvard Graduate School of Business Administration, processed. Murphy, K., and . Welch (1992). "The Structure of Wages." Quarterly Journal of Economics, v. 107, no. 2, pp. 285-326. Nordhaus, W. (1969). Invention, Growth and Welfare, Cambridge: M.I.T. Press. Organization for Economic Co-operation and Development (1995). Learning Beyond Schooling. Centre for Educational Research and Innovation. Organization for Economic Co-operation and Development (1996). Lifelong Learning for All. Education Committee at the Ministerial Level. Organization for Economic Co-operation and Development (1996). Adult Learning and Technology in OECD Countries. Centre for Educational Research and Innovation. Organization for Economic Co-operation and Development (1996). Adult Learning in a New Technological Era. Centre for Educational Research and Innovation. Parsons, Donald M. (1990). "The inn's Decision to Train." Research in Labor Economics, v. 11, pp.53-75. Porter, M. (1991). Canada at the Crossroads. Ottawa: Supply and Services Canada. Roberts, J. M., J. R. Rossiter, and S. Duncan (1996). Lessons from the Lifelong Learning on the Information Highway Series: Renewing the Focus on Learning. Presentation to Educational Technology 2000, A Global Vision for Open and Distance Learning, Singapore, http: //www.kcc.ca/project/conference/cnfal006.html. Stager, David (1995). Economics of Technology-Based Learning and Training on the Information Highway. Report to the Working Group on Learning and Training of the Information Highway Advisory Council. 2.3 Workshop and Website EvaluationsAs part of this evaluation, we reviewed the available information on the demonstration workshops held by the OLT and on the OLT's website. As noted elsewhere in this report, key informants were generally very positive about both the Conference Program of the OLT and the website. 2.3.1 Workshops The OLT sponsors workshops on-site and in partnership with organizations at other sites across the country. These workshops are a vehicle for bringing practitioners, policy makers, administrators, developers and others interested in learning technologies together to share information, and to facilitate networking and opportunities for partnerships. We reviewed the data from the evaluation forms used by the OLT at four of its eight workshops, following are the topics for which evaluations were available:
The respondent evaluations shown in Exhibit 2.1 have all been converted to or based on a scale ranging from 1 to 5 in which 5 is excellent and 1 is poor.
These results from the participants are generally indicative of positive reactions to the workshops. The first two have lower average scores, while the scores for Conference 8 are quite high. A larger number of sets of evaluations would have been useful but it appears that the OLT has learned from its early experience in presenting workshops and that its efforts are now very well-received. 2.3.2 The OLT Website The website is one of the key initiatives of OLT, providing a mechanism for information sharing and discussion. In addition to providing information about OLT and its programs, it serves as an extensive source of more general information on learning technologies, offering searchable databases, bibliographies and electronic linkages to related sites across the country. It also offers a venue for participation in on-line electronic conferences related to learning technologies. Its "Learners' Corner" provides links to on-line and distance education institutions, courses and other resources for learners interested in using learning technologies, Finally, the "Practitioners' Corner" provides teachers, trainers, instructors and educators with links to resources to help them learn more about how to incorporate learning technologies into their work. Although there are no website assessment data comparable to those for the workshops, we reviewed a set of users' comments on the website. This sample of comments on the website is limited to users who chose to make use of the comments portion of the site. Some of those who commented noted that they had found the site through recommendations from industry groups or educational institutions. The comments from the users are overwhelmingly positive. The OLT also tracks the usage of its website. These data show recent significant increases in usage. In January 1998, average daily hits totalled 149 while in June, the number of hits, per day increased to 564. Visitors to the site from Canada constitute 64.6% of all hits, while visitors from the United States constitute 32.7% of all hits, with the remaining 2.7% from the rest of the world. Analysis of the characteristics of visitors to the website's guest book reveals the following. In terms of organizational affiliation, 25% of respondents were university or college-based, 24% were business people, 17% were from government, 13% represented associations or community groups, 10% were visiting as individuals, and 9% were based in elementary or secondary schools. In terms of the OLT's target audiences, 31% were practitioners, 28% were administrators, 16% were developers/technicians, 15% were researchers, and 11% were learners.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![[Previous Page]](/web/20060127014814im_/http://www11.hrdc-drhc.gc.ca/edd-img/back.gif){kind=small}
![[Table of Contents]](/web/20060127014814im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif)
![[Next Page]](/web/20060127014814im_/http://www11.hrdc-drhc.gc.ca/edd-img/next.gif){kind=small}