|
|
     |
||
 |
     |
 |
|
The second major theme for the evaluation relates to design, delivery, planning and implementation. 4.1 Do Provincial Programs and Services (PPSs)reflect the intent of the Canada-Saskatchewan Labour Market Development Canada (CS-LMDA)?4.1.1 Overlap and duplicationAs already noted, key informants said both the federal and provincial governments offer services to Aboriginal people, youth, and persons with disabilities.40 In some cases, programs receive funding from both orders of government. For example, both federal and provincial governments fund the Employability Assistance for Persons with Disabilities program (EAPD), but it is delivered through the province. In other areas, federal and provincial programs may appear to overlap, but as stressed by some respondents, these are instances where both orders of government coordinate complementary programming. Provincially, income determines eligibility for training support. Therefore, youth, Aboriginal people, and persons with disabilities are able to receive support for training like other Saskatchewan residents,41 and may also be eligible for targeted support from the federal government. Federal programs are more likely to target specific groups. For example, the Youth Employment Strategy is a multi-departmental initiative that provides national training opportunities for youth. The Aboriginal Business Canada program funded by Industry Canada and the Aboriginal Human Resources Development Strategy of HRDC are examples of federal training programs for Aboriginal persons. Federal and provincial managers stressed that close consultation between federal and provincial staff circumvents situations where the same client receives similar support from two programs. Redundancy in programming actually increases access. Key informants told us that close coordination avoids waste. 4.1.2 Community involvementFederal key informants reported that many partnerships existed between government, training providers, employers, and the community prior to the CS-LMDA. According to those we interviewed (federal and provincial key informants), the Agreement has enhanced the development of new partnerships. Government, employers, Saskatchewan Institute of Applied Sciences and Technology (SIAST), regional colleges, Aboriginal groups, and community groups are involved in regional needs assessment and planning processes, as well as in program delivery. For example, the Northern Labour Market Committee, created in 1983, identifies and assesses labour market and economic development issues, and initiates action, enabling Northerners to benefit from regional activities. With three co-chairs, this committee includes representatives from First Nations communities, Métis Nation of Saskatchewan, industry/training institutes, and the provincial government Office of Northern Affairs. Some key informants (federal and provincial) said the transition to the CS-LMDA has been challenging, because partnerships established by the federal government had to be re-established and redefined (n=12). The province has recently developed a framework to define the competitive process for selecting Community-based organization (CBOs) that will receive longer-term contracts. The survey of employers shows a reported rise in partnerships among employers and the Saskatchewan government, Aboriginal groups, and community groups (n=146). However, the table below also shows that most employers using services under the CS-LMDA typically do not get involved in partnerships. This is quite common, as many businesses see this form of action as irrelevant to their immediate needs.
4.1.3 Service in FrenchThe demand for service in French in Saskatchewan reflects the proportion of the Francophone population for Saskatchewan (2 percent). The client survey found only 11 people out of 1,250 (less than 1 percent) who said they received or wanted to receive services in French from Canada-Saskatchewan Career and Employment Services (CSCES) or Human Resources Development Canada (HRDC). Of these, nine respondents said they were satisfied with the level of service. When asked whether service had improved, stayed the same, or become worse after January 1999, four said it had improved, and three said it was the same. Only one thought it was worse, and two respondents could not compare (1 chose not to respond). 4.1.4 Promotion of Canada's contributionAccording to provincial key informants, CSCES centres and promotional materials clearly identify the role of Canada. Early missteps and failure to mention the federal contribution to the CS-LMDA has created negative feeling among some HRDC managers. Federal key informants were particularly concerned about this lack of recognition, but acknowledged that the situation had improved. Efforts to achieve public awareness of Canada's contribution to PPSs include:
Key informants disagreed about whether these efforts have been sufficient. Some federal key informants believe they have been insufficient, while most provincial representatives claim that the province has met its requirements. It is worth noting that using client surveys to test whether the federal profile remains high is not reliable. Most clients of CSCES make only a cursory distinction between the federal and provincial government, and typically think generically of government. 4.2 Information systems and monitoring client progressAt this point, information systems cannot monitor client progress to the extent outlined in the CS-LMDA. The province is currently in the second year of a four-year transitional plan to link Post-Secondary Education and Skills Training (PSEST) computer systems and provide information from all provincial programs. The plan also calls for connecting the provincial network with public post-secondary training institutions (SIAST, regional colleges). While development continues, staff will require training on a new system, and legacy information must be integrated. Our review of the database shows incomplete information in several cases, and some duplication. Comparing the Skills Training Benefit (STB) and One Client Service Model (OCSM) database shows that information is not completely shared between the two systems. Until this system is developed and discrepancies are resolved, it is difficult to exclusively rely on OCSM to provide information on how clients use provincial programs and services. Key informants also reported that data sharing agreements between the federal government and the province limit the ability of the province to provide information on client progress. To obtain information on the EI status of clients, PSEST staff must use the "screen scraper" process43. Provincial and federal staff report that this system is awkward, and believe strongly that the incomplete data sharing between HRDC and CSCES is a barrier to full service. The federal system (OLIS) currently provides "point-in-time" information on EI eligibility, and is not always up-to-date.44 Ideally, the province would like historical information on clients that dates back at least one year. The current process is said to be cumbersome by key informants who have used it. Provincial key informants reported that their staff frequently contacts HRDC by telephone to verify clients' EI status. In some offices, the communication between HRDC and CSCES is effective. Ex-HRDC personnel in these offices often have "pipelines" to HRDC information by virtue of their personal connections. However, in other centres, this close relationship does not exist, and clients noted that HRDC staff failed to direct them to CSCES.45 Key informants (federal and provincial) highlighted other concerns about the current information systems, and the prospect of measuring client progress:
Key informants (n=25) reported that the provincial information system is still in the development stage.47 The system is currently unable to generate aggregate reports that track client outcomes, but the intention is to eventually be capable of doing so. Some key informants observed that tracking client outcomes (i.e., post-program follow-up) is a human resource issue as much as it is a systems issue, and should be viewed in a broader context. If service providers (regional colleges and SIAST in particular) play a greater role in delivering programs and services in the future, computer databases will need to be fully integrated and developed. Even if the province and the federal government proceed in this direction, several key informants suggested that including information from SIAST and regional colleges would be a daunting task. Although this issue is not central to the formative evaluation, it is critical for the implementation of the LMDA. This information lapse could also compromise the summative evaluation, if data on clients cannot be easily retrieved. The summative evaluation will begin in April 2002. By then, system integration should be sufficiently advanced that all clients, including EI eligible clients, can be tracked in terms of PPSs received. Furthermore, federal information from the Status Vector file should be available to identify outcomes (Part 1 benefits received and savings to the EI Account). 4.3 Regional variation in serviceThe difficulty in linking to HRDC data notwithstanding, it is difficult to fully capture current activity in programs from the provincial databases, for three reasons:
With these three cautions in mind, the sections below provide an account of activity to date based on analysis of project databases, the client survey, and interviews with key informants. 4.4 Variation in programs — client survey and databasesSaskatchewan's dispersed population and variations in industry across the province affect how clients use programs. According to the STB database (n=4,019 clients) (Table 6), most EI clients using the STB for income support participate in a Skills Training program of more than 12 weeks (37 percent), or Apprenticeship (30 percent) or a Skills Training program (25 percent) of less than 12 weeks. Approximately 3 percent of EI clients receiving the STB participate in either a Bridging or Self-Employment program.
The OCSM database (see Table 7, next page) supports a markedly different view of program participation than the STB. This is partly due to incomplete data recording. It also suggests that many who participate in Apprenticeship, Bridging, Community Works, and Work Placement do not receive the STB as a training support. Those in Apprenticeship will often collect EI benefits (Part 1) to support themselves. Work Placement and Community Works participants will typically support themselves through subsidized wages, while Bridging students may receive the Provincial Training Allowance (PTA), STB, or EI benefits, or receive no income support.
Observation: The development of the information systems represents a work in progress. The variation in training participation reported by the STB and OCSM reveals the extent to which different information sources provide quite divergent pictures of program participation. Both federal and provincial managers are keenly aware of the urgency in completing the development of the OSCM system and improved coordination of EI and PSEST data. 4.5 Variation in services — client survey and databasesNot all EI clients require training to re-enter the workforce, as many already have the skills required by industry. Others may combine a service (such as searching for work on the Internet or talking with a career counsellor) with short or long-term training. For the purpose of both the formative and summative evaluation, any EI client who uses either training or a service offered through a CSCES office is considered a "participant," as he/she is actively trying to get back into the labour market. Two sources provide information on participation in provincial services: the OCSM database and the client survey. The OCSM database only records a program or service when a client interacts with a counsellor and that counsellor updates the records. The participant survey reported below reveals that many interactions with the system probably go unrecorded, particularly when a client drops in to CSCES and does not see a counsellor, or when he/she uses the SaskNetWork, SaskJobs, or HRDC job bank from a remote site (e.g., home computer). This under-recording of activity by OCSM is unavoidable, as staff cannot record all activity unless clients report all their activity to counsellors on a regular basis, and client service is interrupted for the counsellor to enter data.49 The under-reporting makes it difficult to estimate the true value of services such as counselling, brief advice on the phone, or client use of the web sites. According to the OCSM database (n=11,962 services recorded), the most commonly used service is counselling (86 percent). Other services recorded include orientation (19 percent), referrals (13 percent), job search (6 percent), and work preparation (4 percent). Differences across regions may reflect the type of clients in each area, as well as differences in CSCES operations and the practices of counsellors when recording interactions. For example, based on the data, it would appear that counselling is mandatory for all EI clients in Region 3 (n=3,678 services recorded) (96 percent of clients have counselling listed), whereas it is much lower in Regions 1 (n=588 services recorded) and 4 (n=3,735 services recorded). Whether this is due to coding practices should be investigated further during the summative evaluation. Clearly, the reliability of data entry and the standards/practices used in different CSCES offices may have a profound impact on the evaluation.
The client survey (sample of 1,250) also records use of CSCES services, capturing information in a different way than the OCSM database. Based on the survey, two-thirds (67 percent) report talking to a counsellor about training, while 62 percent said they read information on training and education programs. Approximately half used a computer at the centre to search web sites for a job, while the same proportion talked to a career/employment counsellor to plan a strategy for returning to work. One-quarter of participants (24 percent) phoned the centre to enquire about job openings, while the same number used multimedia products or the computer to write a resume.
Observation: The regional variation is expected. The lower use of services in some centres may reflect remoteness and their recent creation. An important issue for the summative evaluation is whether CSCES centres are able to serve EI clients in all parts of the province. 4.6 Variation in program and services — key informantsMost key informants said that although all field offices deliver all programs and services, the actual PPSs used vary by region (n=15).50 For example, clients in the North tend to be SA recipients rather than EI clients, and tend to have multiple barriers to employment compared to clients elsewhere. Although some key informants expressed uncertainty about whether clients using PPSs represent eligible EI clients, many suspected that a large fraction of EI clients may not be using PPSs for the following reasons:
Many provincial key informants noted that the province has no way of knowing the identity of eligible EI clients in a region, and therefore cannot invite them to use PPSs (n=13). Some also doubted whether all HRDC staff direct clients to the provincial offices. However, as we note below, the client survey indicates that an HRDC/EI counsellor referred 30 percent of the 804 respondents. HRDC is reportedly considering an insert with EI applications to advertise PPSs. Key informants said a variety of promotional methods have been undertaken to inform clients and employers about PPSs, including distribution of written materials, TV and radio advertising, posting information on the Internet, grand openings of CSCES offices, presentations to stakeholders, and joint orientation sessions conducted by provincial and federal staff (n=28). They rated direct person-to-person contact and word-of-mouth as the most successful promotional techniques. Many key informants emphasized the need to reach rural and remote areas. Observation: Promotion by HRDC and PSEST is an important step in encouraging EI clients to use programs and services. Although broadcast advertising, newspapers, and other media have a role, personal contact and word-of-mouth appear to remain the most effect methods for promoting the programs and services. 4.7 Demographics of EI clientsAn important issue for the formative evaluation is to understand the demographic composition of EI clientele (n=12,753)51:
4.8 Roles of partnersAlthough a few key informants (from federal and provincial governments) said the roles and responsibilities of the federal and provincial governments are clear to them (as senior managers), more said these are not completely clear, either to management or to frontline staff (n=11). They singled out federal/provincial responsibilities relating to Aboriginal people, youth, and people with disabilities as areas where ambiguity may exist (n=15). A majority of key informants said federal and provincial roles and responsibilities are also unclear to clients and the public (n=13), and they encouraged better marketing of this relationship. Disagreement exists over whether this confusion is a problem, as some felt that current promotional efforts are adequate. A key question is whether managers direct Part 1 clients to CSCES centres (Part 2 programs or the PPSs). Based on the client survey, 30 percent of those who have visited a CSCES office (n=804) were referred by HRDC/EI counsellors. More often, clients learned of CSCES through word-of-mouth or from family or friends (33 percent), or an advertisement (8 percent). The remaining 29 percent reported a wide variety of referral processes, or could not recall how they had first been referred. Observation: The directness of the path from Part 1 benefits to Part 2 programs/services is important. The legislation underlying EI affirms the importance of clients accepting benefits to return to work or seeking training. In the formative evaluation, it is hard to measure the strength of this connection. However, through analysis of non-participant data and a staff survey (scheduled for the summative evaluation), additional insight into promotion of the PPSs will become available. 4.9 Annex 3 and 7 arrangementsService delivery arrangements outlined in Annex 3 of the CS-LMDA have been met. CSCES offices are in place across the province, delivering programs and services to clients. Annex 7 of the Agreement outlines provisions for transferring federal staff to the provincial government, governed by the Employee Transfer Agreement and negotiated between the federal government, the government of Saskatchewan, and the Saskatchewan Government and General Employee's Union (SGEU). According to provincial data, the CS-LMDA resulted in the transfer of 97 HRDC staff (full-time equivalents) to the government of Saskatchewan. Both federal and provincial key informants acknowledged that human resource/organizational culture issues exist with amalgamating federal and provincial staff. According to several key informants, some former HRDC staff working for the province wish to return to HRDC (n=17). Other ex-HRDC staff we interviewed apparently believe they have less discretion to make decisions. However, most ex-federal staff who reported this view also reported that provincial practices are changing, as experience is gained in delivering programs and services. 4.10 CollocationIn general, managers and staff from both orders of government support the concept of collocation. No empirical measure of the incremental impact is possible, but staff note that it should lead to better service. Technically, only one CSCES is truly collocated (North Battleford), but in other areas, HRDC offices and CSCES are in close proximity (close collocation). In cases where close collocation has occurred, key informants believe that clients experience better service (n=25). This improvement is mainly due to the perception of increased convenience to clients for accessing several services or agencies in one location (n=21). Other positive effects of collocation reported by key informants include increased efficiency and promotion of partnerships/cooperation. Perceived difficulties arising from collocation are sensitivities around information sharing and use of different computer systems. A few of the 41 key informants said collocation with the regional colleges has been problematic. Originally, the province wanted to deliver PPSs through the regional colleges and SIAST, but HRDC regarded these as third parties and did not permit their inclusion in the CS-LMDA as a government-to-government agreement. This has reportedly caused some bad feelings on the part of regional colleges. Employers offered their views on collocation in their survey. Among those surveyed, 48 percent (n=70) reported awareness of the collocation of HRDC and provincial offices. Among these employers, 43 (61 percent) said they had visited one of these offices. This number seems high, given the limited number of collocated offices. It is possible that some thought that CSCES offices were all collocated given their name (Canada-Saskatchewan), and that they were comparing CSCES to the collocated sites. Although the sample size is small, employers report that collocated offices provide better service, a higher level of service, and greater administrative efficiency than non-collocated offices.
4.11 Summary on implementationThe PPSs include all the major elements required in the CS-LMDA. Services exist throughout the province, although some variation is apparent. Given the newness of these services, this is expected. Service in French is available where numbers warrant (Prince Albert, Moose Jaw, Saskatoon, Regina), but insufficient recorded use of these services has occurred and their adequacy cannot be assessed. Some redundancy exists between the services offered by the province and the pan-Canadian initiatives (youth, Aboriginal, and persons with disabilities), but senior management (federal and provincial) believe that this does not imply overlap and duplication. Close coordination ensures that the same resources are not directed to the same clients. Staff support the redundancy as offering improved access. The CS-LMDA seems to be associated with improved partnerships and consultation. It is difficult to attribute this to the Agreement. The needs assessment and sector partnership processes progressed independently, and predated the Agreement by several years. The nature of the process has evolved under the CS-LMDA, in that CSCES offices (created as a result of the Agreement) now lead the process. Some areas of implementation work remain:
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![[To Top]](/web/20060126222315im_/http://www11.hrdc-drhc.gc.ca/edd-img/top.gif){kind=small}
![[Previous Page]](/web/20060126222315im_/http://www11.hrdc-drhc.gc.ca/edd-img/back.gif){kind=small}
![[Table of Contents]](/web/20060126222315im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif)
![[Next Page]](/web/20060126222315im_/http://www11.hrdc-drhc.gc.ca/edd-img/next.gif){kind=small}