Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

2.1 Introduction

On January 23, 1996, three-year National Framework Agreements were signed with the Assembly of First Nations (AFN), the Métis National Council and the Inuit Tapirisat of Canada. These Framework Agreements set out the principles and reporting relationships to guide negotiations of Regional Bilateral Agreements (RBAs) between HRDC and Aboriginal organizations on the design and delivery of Aboriginal human resources programs. The principles address national concerns such as equitable access, women's issues, adherence to HRDC's mandate, reporting and evaluation requirements.

HRDC's Aboriginal Relations Office (ARO), along with HRDC regional co-ordinators, has responsibility for the management of the RBA initiatives. In order to provide performance information with respect to the implementation and impact of the RBAs and to meet contractual obligations and Treasury Board requirements, the ARO developed a strategic evaluation plan. As part of the evaluation plan, Mid-Term reviews with selected RBAs were scheduled to provide information on the early experiences with implementation of the initiative, particularly at the level of the RBA.

The objectives of these Mid-Term reviews were to:

Provide a description and analysis of:

• The RBA process, the agreements implemented, and the characteristics of operations in diverse communities;
  • The employment and labour market interventions and activities designed and delivered (or to be delivered) under the Agreements;
  • The early outcomes and achievements for client groups and communities who participated in the development of the RBA interventions and in the RBA interventions; and,
  • The processes and procedures that are in place for ongoing monitoring and future reviews and evaluations under the RBA initiative.
• Collect and analyze information on the processes put in place (e.g., service delivery) to achieve their objectives; and,
• Report findings to:
  • RBA-holders to assist in the improvement and fine-tuning of the RBA process and of activities that are undertaken under these agreements; and,
  • HRDC to assist in the improvement of human resources development policy.

Five HRDC regions — Newfoundland, Quebec, Ontario, Saskatchewan and Alberta — agreed to participate in this exercise and conducted Mid-Term reviews with some of their RBA partners to document their initial experience with the initiative. The purpose of this study is to provide a national synthesis of the information coming from the regional Mid-Term reviews. A total of 11 Mid-Term review reports were examined: 2 from Newfoundland, 2 from Quebec, 4 from Ontario, 2 from Saskatchewan and 1 from Alberta. These 11 Mid-Term reviews covered a total of 14 RBAs and 1 HRDC-Aboriginal Government Agreement. Table 2.1 summarizes the RBAs included in the Mid-Term reviews.

The methodologies employed by the organizations/consultants conducting the Mid-Term review generally included the following:

• A Document Review (e.g., RBA documents, funding framework agreements, evaluation frameworks, policies and procedures, previous reviews, etc.);
• A Database Review (where available);
• Key Informant Interviews with representatives from HRDC and RBA-holders, including program staff at HRDC and Aboriginal service delivery organizations;
• Focus Groups with representatives of Aboriginal service delivery organizations;
• A Participant Survey (a survey of those who have participated in RBA interventions where applicable); and,
• Case Studies, involving interviews and focus groups, in selected communities where time for the review was limited and/or the geographic area covered by RBA was large.

The results of the 11 Mid-Term reviews were summarized in a standard grid format according to seven broad categories of issues identified in the RFP: HRDC-RBA-Holders; Managing, Designing and Delivery; Interventions, Participants and Results; Data and Accountability; Costs; Client and Community Satisfaction; and, Recommendations for Improvements. Once the grids were completed, they were reviewed to identify any gaps in the information provided in the review reports. Where information gaps were identified, supplementary interviews were conducted with HRDC regional program staff. The synthesis study was also sent to Regions and RBAs who participated in the Mid-Term review for their comment.

The results of our national synthesis of the regional Mid-Term reviews of the RBAs are presented below.

2.2 HRDC-RBA-Holders — General Context

How many Agreements have been signed?

A total of 54 Regional Bilateral Agreements (RBAs) have been signed since April 1996. In addition, there are 9 Contribution Agreements with the Congress of Aboriginal People within the RBA budget. Table 2.2 illustrates the number of agreements per province/territory and the total dollar value.

What factors facilitated the signing of agreements?
What factors delayed their signing?

Most of the RBAs included for this study were signed in a timely manner. For many RBAs, the signing of these agreements was generally facilitated by several factors - all linked to the experience of both parties (HRDC and RBA-Holders) under the Pathways to Success program. As a result of their Pathways experience:

• A need for more Aboriginal control was identified. The Pathways experience led both HRDC and Aboriginal organizations to seek an alternative program structure that would place the management of funds with the Aboriginal organizations. Both parties agreed that such a structure would permit Aboriginal organizations to identify more accurately the specific human resource development needs of their respective communities and to develop and implement programs that would increase their members' access to the labour force.

• Administrative capacity gained by Aboriginal organizations as a result of participation in Pathways to Success program. In a number of the RBAs reviewed (the HRDC-Federation of Newfoundland Indians RBA, the HRDC-Miawpukek First Nation RBA, HRDC-London Districts Chiefs Councils RBA, and HRDC-Saskatchewan Indian and Training Assessment Group RBA), participation in the Pathways for Success program permitted the RBA-holder (i.e., the Aboriginal organization) to build on and enhance their administrative capacity (management structures, delivery structures, information systems, etc.). Having this capacity enabled the RBA-holder to take over the administrative control of Aboriginal programming, including design and delivery. In these RBAs, the transition to the new structure appeared to have transpired quite smoothly.

  However, according to the Mid-Term review reports, in other RBAs reviewed, lack of administrative capacity within the Aboriginal organization led to problems in implementing their agreements. In one particular case, this was mainly attributed to the fact that the agreement was signed prematurely; i.e., before both parties, and in particular the RBA-holder, were ready in terms of administrative and delivery capacity to assume their responsibilities under the new structure.

• A prior history of working with HRDC on joint projects was a key facilitative factor for the RBA-holder. For a number of the RBA-holders (the Federation of Newfoundland Indians, the Miawpukek First Nation, the London District Chiefs Council, the Saskatchewan Indian Training and Assessment Group, and the Kativik Regional Administration), a previous history of working jointly with HRDC to deliver human resource development programs to their respective communities was key to the signing and subsequent implementation of the agreement.

  In at least one case, the relationship between the two parties at the senior management level was not as good as it could have been and may have contributed to problems in implementation.

Two of the RBAs included in the Mid-Term review experienced a delay in signing. In one case, this was mainly attributable to issues surrounding the distribution of funds. In another case, the delay was due mainly to issues related to the political impact of RBA negotiations on other agreement negotiations and the capacity of the RBA-holder to quickly assume its new administrative responsibilities.

To what extent is the content of the RBAs consistent with National Framework Agreements?

Although there was little in the way of description of the content of the agreements in the Mid-Term reviews studied, the intent of all the RBAs reviewed was to increase the capacity of the RBA-holder to design and administer their own programs. There is evidence that what is happening through the agreements is generally consistent with most of the principles set out in the National Framework Agreements; i.e., in most of the RBAs reviewed, there is evidence of:

• increased Aboriginal control over program administration, including program design and delivery;
• recognition that Aboriginal communities best understand their human resource needs and are best positioned to develop programs to meet those needs;
• non-derogation of existing rights;
• flexibility in program design and delivery to accommodate unique and special circumstances of Aboriginal peoples (e.g., although training was identified as a priority for their community members, the Federation of Newfoundland Indians recognized that most members did not have their high school leaving certificate. Thus, there was an initial focus on group-based Adult Basic Education (ABE) programs under their RBA.);
• a focus on labour market needs of participants, communities and employers;
• inclusiveness, equality of access and incrementality;
• consistency with applicable legislation;
• gender equity;
• inclusion of a representative proportion of youth among program participants;
• some openness in terms of availability of information about the programs (awareness of programs in the communities). While this is ensured through the marketing activities of the Aboriginal delivery agents, in most of the RBAs reviewed, there are indications that this openness is not present in all RBAs; and,
• accountability.

However, in a number of the RBAs reviewed, there is evidence that not all the principles of the NFA are being respected. In particular:

• the Mid-Term review of one RBA found that more effort is required to enhance the accessibility and availability of training programs for Aboriginal people through better out-reach;
• the Mid-Term review of another RBA found that the inclusiveness principle needs addressing. The review found that the ratio of on-reserve participants to off-reserve participants was greater than 6:1 and recommended that participation of off-reserve participants be increased;
• at the time of the Mid-Term review, systematic monitoring and reporting systems were not yet fully operational for most of the RBAs participating in the Mid-Term review. This made it difficult to assess accountability for RBA funds and to identify results achieved. This was particularly problematic for two of the RBAs; and
• although there is some evidence that there is gender equity among program participants in all the RBAs reviewed, there is no evidence that Aboriginals with disabilities or Elders⁸ are being served by programming under any of the RBAs reviewed.

Has the funding arrangement under the RBA process improved the flexibility of program and services options from which Aboriginal people can select to meet the employment needs of clients?

For most of the RBAs reviewed, the funding arrangement under the RBA process has improved the flexibility of program and services options from which Aboriginals can select to meet the employment needs of clients. Generally speaking, the funding arrangements have allowed RBA-holders the flexibility to design interventions based on local need. There is also evidence that some RBA funding arrangements have allowed more flexibility in terms of eligibility. For example:

• under the HRDC-Federation of Newfoundland Indians (FNI) Aboriginal Flexible Funding Arrangement (AFFA), the FNI has the flexibility to allocate living expenses based on individual need rather than eligibility rules that apply regardless of need (this makes individuals responsible for their own success in the programs); and
• under the HRDC-Métis Nation of Ontario (MNO) RBA, RBA funds are being used to assist individuals who are employed (in the past, regional HRDC guidelines did not permit this type of support).

On the other hand, in the opinion of some RBA-holders, the funding arrangements under some RBAs have not led to increased flexibility and this remains an issue to be addressed. For example:

• in one particular case, the funding arrangement did not result in a discernible difference in flexibility of programs and services options for Aboriginal clients. In the opinion of one Aboriginal official interviewed as part of the review, there was more flexibility under the Pathways for Success program. This is due to the phased-in approach to empowerment that was taken. Under this RBA, only those funds dedicated to administration and implementation of the Agreement were transferred to the RBA-holder, while responsibility for the actual contracting of client programs and purchases on behalf of clients remained with HRDC.

In addition, the RBA-holder indicated that the agreement was not flexible enough to meet the needs of degree-seeking students. This support is currently restricted by regional⁹ HRDC guidelines;

• under one RBA, HRDC continued to manage and deliver training programs and services to Aboriginal clients during the first year even though the Regional Approval Committees (RACs) were mandated to decide on training applications submitted by Aboriginal clients for assistance;
• under another RBA, it was noted in the Mid-Term review report that more flexibility was required in terms of responding to the needs of youth in remote communities, particularly in the area of pre-employment training. In the opinion of the RBA-holder, HRDC guidelines are not flexible enough to meet the 'economic and cultural' realities of remote communities — they focussed on the recently unemployed rather than those with little experience in the workforce; and
• under one of the Alberta RBAs, RBA-holder interviewees indicated that more flexibility in how goals and objectives are to be achieved and how funds are to be spent is required. There is a general feeling that the RBA funding arrangement is not responsive enough to the particular needs of the RBA-holder's clients. In the opinion of the RBA-holder, employment and training should be considered from a holistic perspective that includes a balance of 'spirituality, alcohol and drug abuse, fear of success, and employment equity'.

Has the partnership between HRDC and the RBA-holder facilitated the process of program and services design and delivery?

Overall, the partnership between HRDC and the RBA-holder has contributed to the successful facilitation of the process of program and services design and delivery. In most cases, this partnership is a longstanding one and there is a history of successfully working together on joint projects. Under most RBAs, this partnership has become more informal yet supportive. The Human Resources Centres of Canada (HRCCs),¹⁰ though no longer involved in direct service to clients, provide support to the RBA-holders in the areas of program design, development and implementation and financial accountability system development and implementation. In addition, there is ongoing consultation, counselling and communication between the two parties on an 'as needed' basis.

Overall, from the point of view of the RBA-holder, there is less bureaucracy, less time spent on administration and a greater degree of decision-making authority.

In some cases, there are difficulties in the partnership between HRDC and the RBA-holder. For example:

• under one RBA, the nature of the partnership between HRDC and the RBA-holder was not as facilitative as it could have been. According to the Mid-Term review of this RBA, the senior parties on both sides generally agreed that poor communication between them was at the heart of the many problems facing the agreement. In addition, the parties were unable to agree on several aspects of the administration and have not as yet reached mutually satisfactory accommodations on these issues. However, it must be noted that, at the local delivery level, the partnership between HRDC and the RBA-holder remained strong and supportive;
• in Alberta, under one of the RBAs, there are some challenges in the current working relationship with HRDC. The RBA-holder indicated that there is still some mistrust of HRDC relative to the 'paternalistic approaches' experienced under the Pathways to Success program. HRDC has decentralized the authority to the regional level. Although this has given the regional managers significantly more authority, the result has been less co-ordination at the national level; and
• under another RBA, although the partnership process was, for the most part, effective and harmonious, the federal government did not always make the distinction between the Aboriginal partner and those responsible for other agreements between the federal and Aboriginal governments.

Are interventions that have been offered to clients consistent with the agreements, the Employment Insurance (EI) legislation, and the employment needs of their clients?

According to the Mid-Term reviews of the RBAs included for this study, the interventions that have been offered to clients are generally in line with what HRDC would have done regardless of whether there was an RBA in place or not; e.g., purchase of training benefits, wage subsidy programs, job creation programs, and research and innovation support measures. Although the primary objective of all interventions is to improve the employability of Aboriginal clients, the focus of such programming may differ from RBA to RBA. For example, while the Miawpukek First Nation (MFN) focuses on offering their membership a full array of one-on-one training courses rather than the traditional project-based training approaches, the Federation of Newfoundland Indians (FNI) initially focussed on offering Adult Basic Education (ABE) courses to their clients.

Overall, these interventions are serving the employment needs of the RBA-holders' clients. However, in some areas, RBA-holders indicated that the employment needs are not being adequately served by the interventions offered under the RBA. For example:

• some RBA-holders are frustrated by the fact they cannot offer assistance for degree-seeking Aboriginal clients under the RBA. Although this is consistent with HRDC regional (see footnote 8 on page 11.) guidelines, this is an employment need not being adequately served by the interventions offered under the RBAs; and
• one RBA-holder indicated that the technology-based training needs of their clients were not being adequately addressed by interventions offered through the RBA. In addition, the RBA-holder indicated that a greater focus on career training, counselling, apprenticeship and certification was required.

2.3 Managing, Designing and Delivery at Level of RBA-holder

2.3.1 Managing — Planning and Financial Controls

Guidelines, Procedures and Processes

• Are general guidelines for planning and decision-making in place to determine which programs and services are needed and how they are to be designed and delivered?
• Are formal procedures in place to assist in the planning and decision-making process?
• Are there ongoing approval or confirmation processes?
• Are community partners and/or groups involved in these processes?
• How do results for clients feed back into the decision-making process?

In general, the RBAs set out detailed terms and conditions related to reporting requirements and financial arrangements. Planning and decision-making processes to determine which programs and services are needed are generally left up to the local Aboriginal management organizations (e.g., Local Indian Management Boards or LIMBs, Local Métis Management Boards or LMMBs, Local First Nations Commissions or LFNCs) which are the program delivery agents.

The Mid-Term reviews of the RBAs indicate that formal planning procedures to assist in the decision-making process were not readily apparent. Planning processes are generally informal with very little in the way of formalized community needs assessments and training plans. Planning typically occurs at year-end as a way of allocating the next year's training budget. This results in general objectives and guidelines for the upcoming year. This is not generally seen as a shortcoming by the RBA-holder as local Aboriginal management organization members are usually 'plugged in' to their communities and aware of training needs.

Typical planning partners include regional training agencies (e.g., regional colleges), regional labour market development committees, or other human resource development agencies (including HRCCs).

While targets for primary measures (savings to EI and SA) were negotiated for each RBA, strategic operational planning is not present in most of the RBAs reviewed for this study. However, there are some elements of this type of planning occurring under the HRDC-Kativik Regional Administration Agreement (and at a more local level under the HRDC-AFNQL RBA). Under the HRDC-KRA Agreement, formal annual operational plans are developed which include quantitative targets for each program and financial parameters such as: number of projects; the number of participants, the target discontinuation rate and budgeted CRF (Consolidated Revenue Fund) and EI (Employment Insurance) allocations.

Application and review processes vary from community to community but, in general, clients are responsible for identifying their own goals. Generally, the local Aboriginal management organizations' staff work with clients to identify ways to meet those goals. Clients then formally apply to their local Aboriginal management organization to receive training sponsorship. Depending on the amount of sponsorship requested and the area, the local Aboriginal management organization may approve/deny the application directly. Otherwise, staff prepare a recommendation for approval to the Aboriginal agency (typically a Board of Directors made up of representatives of the local Aboriginal management boards) responsible for overseeing program delivery.

In all of the RBAs reviewed, it is unsure that client outcomes are being fed back into decision-making processes.

Financial Controls

• Are financial controls in place?
• What is the nature of these control processes?
• How is adherence to these processes encouraged, verified?
• Are audit procedures in place?

The review of the RBAs, which participated in the Mid-Term Review process, indicates that there are financial controls in place. In general, the RBA-holders are responsible for controlling the expenditure of program funds (e.g., issuing the cheques to training providers, employers, etc.), with HRDC and the RBA-holder taking on the responsibility for financial monitoring. This is usually done through the submission to HRDC of quarterly claims, an annual report, expenditure plans and participant records. In all cases, the RBA requires the RBA-holder to submit audited financial statements to HRDC. RBA-holders are also held accountable to their membership/communities through regular financial reports.

Financial expenditure data is captured either manually (on a ledger system) or on a computerized system (e.g., SITAG's computerized Training and Administrative System or TAS).

In at least one RBA, there is reinforcement of financial control processes by the RBA-holder. The RBA-holder will not reimburse the individual program delivery organizations for activities not entered into the RBA-holder's computerized financial expenditure data system. In addition, the RBA-holder conducts in-person financial monitors at each office of the local program delivery organization. During such monitors, the RBA-holder staff reviews the computerized expenditure data and general ledgers for inconsistency and irregularity.

There is evidence that financial control processes vary from agreement to agreement in terms of adequacy and our review of the Mid-Term reports indicated that this was an issue under at least one RBA. Suggestions of a lack of financial control and evidence of nepotism in the expenditure of RBA funds led to the suspension by HRDC of the RBA. Until the RBA-holder enacts the required changes in its corporate structure, which would assure that adequate financial control processes were in place, HRDC staff is conducting continuous financial monitoring of the expenditures of the individual program delivery organizations. This requires that all receipts be submitted and examined to see if they are eligible for reimbursement under the agreement.

2.3.2 Managing — Design and Delivery

Operational Manager and Delivery Officer Involvement

• Are operational managers and delivery officers involved in the design of interventions and delivery mechanisms?
• Do managers monitor delivery operations?
• Are there guidelines/procedures that facilitate this work?

In principle, RBAs transfer authority for the design of federal labour market programs from HRDC to the Aboriginal RBA-holder. RBA-holders typically manage and administer agreements through their human resources development agencies (e.g., FNI's Mi'kmaq Resources and Development, the Miawpukek First Nation's Miawpukek Human Resources, Kativik Regional Administration's Employment and Training Service, AFNQL's Commission for the Development of Quebec First Nations Human Resources, etc.) which are, in essence, the principal delivery agents for the agreement. A Board of Directors/Committee made up of Band/community representatives typically governs these agencies.

In some RBAs, these agencies are directly responsible for reviewing and approving applications for program funding; contacting schools and instructors to set up individualized training programs; working with co-ordinators of Group-based programs; conducting regular monitoring and follow-up of clients; and, counselling clients to ensure they complete their programs. In some RBAs, the agency has set up a number of offices to facilitate greater accessibility for their membership.

In other RBAs, the human resources development agency provides regional co-ordination and leadership, with delivery at a more local level through geographically/community-based organizations. These local delivery agents can be local Aboriginal management boards, local First Nations Commissions, agents, or Aboriginal regional councils and it is they who decide who gets trained, for what, and by whom. Local delivery is typically managed through service delivery agreements with the central human resources development agency. This configuration allows both maximum local flexibility and strong central control.

Delivery implementing structures are usually outlined in the agreements. HRDC's Aboriginal Relations Office developed the RBA Aboriginal Flexible Funding Arrangement Handbook to help, among other things, Aboriginal organizations design and develop their own labour market programs. Further refinement of these guidelines was not indicated in most of the reviews of the RBAs included for this study. However, under the HRDC-London District Chiefs Council (LDCC) RBA, the RBA-holder's Central Administration Unit, which co-ordinates delivery at the local level, developed a Program and Funding Criteria Policy Manual.

Configuration of Delivery System

• What is the configuration of the delivery system (e.g., are the interventions designed and delivered directly by the RBA-holder? Or does the RBA-holder contract with others — third parties — to do this work or parts of the work? etc.).
• How varied is the configuration among RBA-holders participating in the Review?

As noted earlier, RBA-holders are responsible for managing and administering the Agreement. This is generally done through their human resources development agencies (e.g., FNI's Mi'kmaq Resources and Development or MRD, MNS's Métis Employment and Training of Saskatchewan, Inc. or METSI) which were either developed specifically in response to the RBA or were in existence prior to the Agreement. These agencies may establish agreements with third parties (e.g., training agencies) to provide services.

At this point the delivery system can take on one of two general configurations. However, it is important to note that further variations may exist within these two themes and these are noted where applicable.

1. Centralized: In this configuration, the Aboriginal human resources development agencies work directly with clients (review and approve/deny applications to the program for funding) and trainers/co-ordinators (to set up individualized training programs or group-based programs). These agencies may establish offices in a number of locations to facilitate membership accessibility to the programs. This is the current delivery configuration for:

• The HRDC-Federation of Newfoundland Indians (FNI) RBA: Its delivery agency, Mi'kmaq Resources and Development or MRD), reviews and approves applications to the program for funding; contacts schools and instructors to set up individualized training programs; works with co-ordinators of Group Based programs; monitors and follows up on clients on a regular basis; and, counsels clients to help them complete their programs. MRD is functionally operated by a Committee and has two offices.

• The HRDC-Miawpukek First Nation (MFN) RBA: Its delivery agency, Miawpukek Human Resources (MHR), reviews and approves/denies applications to the program from clients for funding. A Board of Directors composed of six Band members governs MHR. MHR prepares an annual operating plan that outlines the programs to be delivered, the operations budget, and the management plan for training and specific activities for the upcoming year.

  MHR has a sub-agreement with the Central Aboriginal Training Institute to deliver services (in conjunction with HRDC's St. Alban's Outreach Office) to off-reserve band members.

2. Devolved to Communities: In this configuration, the delivery agency oversees a delivery system that has been devolved to the communities; i.e., delivery agents are located in the communities they serve. This is the current delivery configuration for:

• The HRDC-Kativik Regional Administration (KRA) Agreement: Although KRA is the only service provider for all training and employment services, implementation is primarily done through its Employment and Training Service (ETS) group. ETS is highly decentralized in its structure, with most of its co-ordinators residing in the communities. These local agents are important contributors to estimating the training and employer needs for their particular communities. They are also responsible for post-training follow-up.
• The HRDC-Assembly of First Nations of Quebec and Labrador (AFNQL) RBA: AFNQL's Commission for the Development of Quebec First Nations Human Resources (CDQFNHR) is the regional organization responsible for human resources development under the RBA. As such, it ensures that the local delivery agents, the Local First Nation Commissions (LFNCs), are well informed of national human resources development programs. At the local level, the LFNCs are responsible for implementing RBA programming based on needs analyzes of prospective clients. Twenty-two LFNCs serve twenty-nine communities and are accountable to the CDQFNHR regarding the nature of the interventions being implemented.
• The HRDC-Métis Nation of Ontario (MNO) RBA: MNO's Métis Training Initiative (MTI) manages and administers labour market training programs. MTI is headquartered in Toronto but has Community Development Officers (CDOs) located in each of the nine MNO regions. In addition to these CDOs, MNO established Regional Approval Committees (RACs) in each of the nine regions. Applications to the program for funding are submitted to RAC members who then forward them on to MTI in Toronto for approval. The CDOs are responsible for tracking client progress, while MTI is responsible for program expenditures.
• The HRDC-London District Chiefs Council (LDCC) RBA: LDCC's Central Administration Unit co-ordinates program delivery through sub-agreements with ten Local Delivery Mechanisms (LDMs). The Central Administration Unit provides an overview of annual work plans that contain training components. Although all LDMs assess training components, the approval process for applications to the program for funding varies from LDM to LDM: 4 LDMs approve applications directly, 3 require Band Council approval of applications; and, 3 have Oversight Committees which review and approve applications.
• The HRDC-Nishnawbe-Aski Nation (NAN) RBA: NAN covers a large geographic area that has been divided into sub-areas for administration purposes. Delivery of program services is generally managed by separate organizations in each sub-area which either act as a Local Delivery Mechanism or as a manager of local community-based Economic Development Officers. Applications to the program for funding are submitted at the local level but a central Regional Bilateral Agreement Committee reviews and approves these applications.
• The HRDC-Six Nations Onkwehon:we RBA: Grand River Employment and Training (GREAT) is the administrative authority on behalf of the 6 Nations Council. Delivery of program services is done through delivery agency agreements signed with Friendship Centres located in southern Ontario or where there is a high concentration of Six Nation membership. For example, services to Onkwehon:we living in the Brantford and Fort Erie areas are offered Friendship Centres located in these communities.
• The HRDC-Saskatchewan Indian Training and Assessment Group (SITAG) RBA: SITAG provides regional co-ordination and leadership, with delivery at the local level by 22 Local Indian Management Boards (LIMBs). Each of these LIMBs has a management agreement with SITAG. LIMBs are responsible for: identifying local training needs, delivering services to meet those needs; developing partnerships to enhance opportunities for clients; monitoring and controlling administration of the RBA; and, building capacity to improve administration and delivery. Centrally, SITAG ensures that training activities meet the terms of the RBA and that administration is consistent from LIMB to LIMB.
• The HRDC-Métis Nation of Saskatchewan (MNS) RBA: MNS's affiliate, Métis Employment and Training of Saskatchewan, Inc. (METSI), has the main responsibility for implementing and co-ordinating the agreement throughout the province. To administer the RBA, METSI has entered into service delivery arrangements with Local Métis Management Boards (LMMBs), which represent 12 geographic areas in the province. Each LMMB is responsible for delivering program services to its area. LMMBs are responsible for: identifying local training needs; planning services to meet those needs; developing partnerships to enhance opportunities for clients; and, building capacity to improve administration and delivery. Although LMMBs do not have the authority to contract for program services on their own, they work closely with their local HRCC which does.
• In Alberta, there is considerable variation in the configuration of the delivery system but essentially all reflect a devolved community-based delivery mechanism:
  • HRDC-Métis Nation of Alberta (MNA) RBA: Services are offered to approximately 42,000 non-settlement Métis through six Regional Council offices and 14 Employment Service Centres.
  • HRDC-Métis Settlements General Council (MSGC) RBA: The Strategic Training Initiative (STI) delivers program services to eight Métis settlements. The STI has five full-time staff but each settlement has a 'political, technical, and financial contact' to assist in the delivery of program services.
  • HRDC-Treaty 6 RBA: The Human Resources Development Board delivers program services to 16 First Nations through a Contribution Agreement. A Human Resource Co-ordinator is located in each First Nation community and is responsible for submitting a local plan for approval by the Chief and Council.
  • HRDC-Treaty 7 RBA: The RBA is administered through the Community Futures business corporation under the Treaty 7 Economic Development Corporation. A Board of Directors composed of two representatives from each Band identifies training priorities and the General Manager is responsible for ensuring that priorities are carried out among the Treaty 7 membership in Calgary and on-reserve. Programs and services of the on-reserve members are the responsibility of each Band, while those for off-reserve members are the responsibility of the Urban Projects Review Committee.
  • HRDC-Treaty 8 RBA: Contribution Agreements were signed with the five Regional/Tribal Councils and Urban Board in the second year of the RBA to facilitate Council accountability. The five Councils represent 22 First Nations and each is responsible for its own programs and reports to HRDC. The Urban Board focuses on funding projects, pre-employment programs and individual training.

Short-term Results for Clients and Primary Measures

• How does information on short-term results for clients feed back into the decision-making process?

With the exception of the two Newfoundland RBAs and the HRDC-Kativik Regional Administration (KRA) Agreement, it is unsure from the Mid-Term reviews that information on clients is systematically collected and fed back into decision-making processes.¹¹

In Newfoundland, both RBA-holders have recently begun inputting client data into the newly installed Client Adjustment Tracking System (CATS). This system focuses on the case management approach; i.e., a client-driven approach to data collection. The CATS is designed to:

• measure results and outcomes rather than processes;
• ensure follow-up until the client outcomes are achieved;
• include all clients who receive assisted service (employment benefits and support measures), as well as counselled clients; and,
• focus on identifying client needs and accountability.¹²

Under the HRDC-Kativik Regional Administration (KRA) Agreement, participant results are entered into Employment and Training System's computerized program management system. The system is easy to use as each screen corresponds to the hard copy forms used by delivery agents and program managers. Data analysis is automatic and produces reports containing statistics on a number of topics such as placements, EI requests, training requests, post-training follow-up and counselling sessions. However, there is concern over the quality of the information collected, in particular, employment services data. Client data is not always up-to-date as follow-up by local delivery agents is not conducted on a regular basis. At this point in time, there is no information available as to how the information collected on the ETS is being fed back into decision-making processes.

In Saskatchewan, SITAG is currently working on a comprehensive client follow-up process to organize the outcome data in a more reliable fashion.

Sufficient Capacity and Capacity-building

• Do RBA-holders have sufficient capacity to design and to deliver the interventions as planned?
• Do they have capacity-building processes in place?
• Does the Aboriginal Flexible Funding Arrangement (AFFA) provide them with the assistance and funding necessary to achieve their capacity-building goals?

In most of the RBAs under review for this synthesis study, it was reported that the RBA-holders have capacity to design and deliver the interventions as planned. Capacity building has been particularly successful at the grassroots level. Experience gained during the previous Pathways to Success Program and during the first year of RBA implementation has contributed significantly to building capacity within the Aboriginal organizations to design and deliver labour market programming.

However, in some of the RBAs, the lack of sufficient capacity in program design and delivery needs addressing. For example:

• some are finding it difficult to find appropriately trained individuals to assume the positions of local delivery agent and co-ordinator. Currently, most local delivery agents have received little or no training and undertake a variety of tasks. Although time is likely to remedy the situation, with the high turnover rate of delivery staff, sufficient delivery capacity may take longer to attain than originally anticipated;
• where agreements were signed prematurely, the capacity to design and deliver program services was found to be lacking throughout the delivery configuration; i.e., from the RBA-holder to the community-based delivery agents. Lack of training and human resources were cited as the main reasons why. Furthermore, all requests for HRDC staff secondments by the RBA-holder to ease the transition process could not be granted. However, as this report is being written, steps are being taken to address the capacity issue; and
• in some, the lack of funding for administering the RBA was identified as an issue requiring addressing.

Other than day-to-day ongoing support from HRDC, there is little evidence of formal capacity-building processes in place. Although AFFA guidelines allow for secondment of HRDC staff to Aboriginal organizations to share experience and expertise, none of the Mid-Term reviews indicated that this, in fact, occurred.

Additional training in administrative processes (e.g., computer systems, development of work plans, etc.) and capacity building in the area of client support; i.e., employment/ assessment counselling, career development, job creation strategies and targeted wage subsidy initiatives, were identified as gaps in capacity building.

Development of New Community Infrastructures

• Have any new community infrastructures been developed?
• Have new relationships/partnerships been developed that facilitate the delivery of multiple services to clients?

Among the RBAs selected for this review, there is evidence that new community infrastructures and/or new partnerships have developed to facilitate delivery of multiple services to clients. For example:

• In Newfoundland, one of the key benefits of the RBA process under the HRDC-Miawpukek First Nation RBA, has been the ability to make collateral contacts across communities to share resources. For example, the Cupertino between the Central Aboriginal Training Institute (CATI) and the Federation of Newfoundland Indians facilitated the sharing of the cost of upgrading River Monitors to full River Guardian status.
• In Quebec, in addition to the HRDC-Kativik Regional Administration (KRA) Agreement, agreements with other federal departments (Indian Affairs and Northern Development and Health Canada) linked KRA in the development and delivery of employment and training services and programs. Furthermore, two agreements signed with the Quebec provincial government (Société québécoise de développement de la main-d'oeuvre) permitted the completion of Agreement programs and services, particularly those related to human resources and special projects.
• Under the HRDC-Métis Nation of Ontario (MNO) RBA, the establishment of a Service Delivery Network enabled the MNO to manage and deliver Métis training programs and services.
• Under the HRDC-London District Chiefs Council RBA, several of the Local Delivery Mechanisms (LDMs) experienced significant success in working together on joint projects and developing job placements with the private sector.
• Under the HRDC-Nishnawbe-Aski (NAN) RBA:
  • the creation of the Sioux Lookout Area Aboriginal Management Board (SLAAMB) facilitated the administration of RBA employment training and labour market programs to that geographic area;
  • partnerships with other First Nations have reportedly enhanced skills and increased the opportunities for economic growth and employment for local people; and,
  • linkages established with Local Delivery Mechanisms, the Ministry of Social Services, HRDC, Indian Affairs and Northern Development, education agencies, Métis programs, as well as federal and provincial forestry programs, have helped to facilitate program delivery.
• In Alberta:
  • informal relationships have reportedly been formed between the various RBA-holders for the sharing of information;
  • the Métis Nation of Alberta created a new Board to oversee program delivery and project selection committees to deliver programs in the various geographic zones; and,
  • under the Treaty 6 RBA, partnerships are being actively sought out with business, industry, corporations and government to maximize the use of funds and to leverage other opportunities.

2.4 Interventions, Participants and Results

2.4.1 Interventions

• What types of RBA interventions (e.g., skills training, wage subsidies, self-employment assistance, job creation, counselling, job finding clubs) were offered to clients?

The nature of the interventions being offered is quite varied. They may include the following: Targeted Wage Subsidy; Self-Employment Benefit, Job Creation Partnerships, Purchase of Training Benefit, Job Finding Club, On-the-Job Training, Training Projects, General Projects, Challenge, Job Entry (including Heritage), Job Placement Assistance, Job Seeking Assistance, Career Academic Preparation, Pre-employment Preparation, Basic Academic Preparation, and Specific Career Orientation. Although not all the Mid-Term reviews provided the same level of detail on the nature of the training offered and taken, there was evidence provided indicating that some RBAs had more focus than others in terms of the interventions being offered. For example:

• under the HRDC-FNI RBA, the RBA-holder recognized the need for long-term planning and focussed on ABE programs as a first step in their RBA programming;
• under the HRDC-MFN, the RBA-holder focussed on individualized training programs at community or private colleges. There was a wide variety of training offered, ranging from micro-computer specialist to veterinarian nursing assistant to heavy equipment repair, which demonstrated that many clients did not want to depend on local economies for employment. In addition, under a one-year Special Arrangement, a component of Group Based training focussed on enhancing the skills of individuals already in the workforce; e.g., Basket Making and Canoe Making for on-reserve clients working in the tourism industry;
• KRA recognized the need for programs to target youths and their cultural heritage. The Heritage Program aimed at youths no longer in school or at risk provides individuals with skills and knowledge related to the Inuit culture and other traditional activities; and
• in Alberta, RBA interventions appeared to have a particular focus on employment seeking counselling; i.e., independent job search, career decision assessment, job finder club, and resume writing.

2.4.2 Participants

• How many participated in RBA interventions — (reported by client groups)?
• How many completed their interventions?
• How many participated in more than one intervention during the period?
• How many received certificates, accreditation or awards?
• Did clients receive financial assistance from EI Part II, SA, CRF? How much did they receive?
• What was the funding source for expenditures on interventions to date (based on information from administrative files at NHQ)?

The level of detail and the quality of data provided on the program participants varied among the Mid-Term reviews; however, this was more a function of the quality of existing reporting systems at that time rather than the quality of the review itself. Because it is not reasonable to compare these results across RBAs, the results are not presented in this synthesis. Comparison is not possible for a number of reasons:

• results are based on two different data sources: administrative data systems and participant surveys;
• in some cases, the participant survey covered only some intervention types;
• not all surveys addressed all results indicators;
• information is not captured in the same way across all RBAs; and
• administrative data was not yet available for some RBAs.

2.4.3 Results

• Did clients find work (employment or self-employment)?
• If information available, what types of work did clients obtain (full-time, part-time, seasonal, occupation; industrial sectors?)
• Were jobs created?
• How long did it take for clients to find work after completing or leaving their intervention?
• What were wages/earnings of clients who found work?
• How much did clients receive in EI benefits since leaving their RBA intervention?
• How many did not complete their interventions (reported by reason for not completing)?

There was limited results information reported in the Mid-Term reviews. For the same reasons that participant information could not be compared or aggregated, results information is not presented in this synthesis report.

However, from the data that is available in the Mid-Term reviews, the following overall findings have been identified:

• clients are obtaining employment after an intervention. But, the majority of the jobs are short-term; i.e., seasonal or temporary;
• the majority of participants complete their intervention; and
• the majority of those who do find employment are earning at or around the minimum wage.

2.5 Data and Accountability

• Are RBA-holders collecting the data and information required for proper monitoring and evaluation?
• Is this data being uploaded to HRDC?
• Is the data reliable?
• Is the computerized data in the HRI client and participant file dataset correctly reflecting the data uploaded from RBA-holders?
• Are the primary measures being calculated by the HRDC Human Resources Investment Fund (HRIF) results dataset for RBAs reflecting the true experiences of clients?

From the Mid-Term reviews, it is unsure that the majority of RBA-holders included in this synthesis study are systematically collecting data and information required for strategic monitoring of outcomes for clients and evaluation. Either there was no data collection system fully operational at the time of the review, or if there was such a system, little information was available on the nature of the data being collected.¹³ Our review of the Mid-Term reviews identified at least two data collection systems that appear to hold promise for monitoring and evaluation purposes. These are briefly described below:

• in Newfoundland, the Client Adjustment Tracking System (CATS) has just been put in place to track the results of those who participated in the interventions offered under the two RBAs (the HRDC-FNI and HRDC-MFN RBAs). CATS captures all client-specific (by SIN #) data such as income support, earnings, and interventions completed. The first upload of results to NHQ's database was expected April 1, 1998. As most of the training courses did not get started until September of 1997, the expected results will only reflect what happened in the short-term. Some primary measures (such as savings to EI) will be available. However, the FNI's focus on ABE (Adult Basic Education) courses will have an initial impact on the primary measures for their RBA; and
• under the HRDC-Kativik Regional Administration (KRA) Agreement, participant results are entered into Employment and Training System's computerized program management system. The system is easy to use as each screen corresponds to the hardcopy forms used by delivery agents and program managers. Data analysis is automatic and produces reports containing statistics on a number of topics such as placements, EI requests, training requests, post-training follow-up and counselling sessions. However, at the time of the study client data was not always up-to-date as follow-up by local delivery agents was not conducted on a regular basis.

In addition, four other data collection and client tracking systems were mentioned in the Mid-Term review reports: the Connector database case management software (HRDC-Alberta RBA Partners); the Contact IV (HRDC-Métis Nation of Ontario RBA); the Aboriginal Labour Market Support Information System or ALMSIS (HRDC-Aboriginal First Nations of Quebec and Labrador); and, the Training and Administrative System or TAS (HRDC-SITAG RBA). These may hold promise for evaluation and monitoring provided that database training and standardization of techniques occurs.

In general, as the uploading of data to the HRI dataset began just prior to this synthesis, there is no information as yet on the primary measures.¹⁴ Moreover, there is insufficient evidence to assess whether these will reflect the true experiences of clients.

2.6 Costs

• How much of the RBA funds are being spent on clients who are participating in interventions?
• How much is being spent on administration?
• How much is spent on capacity building?

Detailed information of RBA costs varied from review to review. However, some cost data was provided and is summarized in Table 2.3. Note, the data was not always available in consistent format. For example, in some cases, it is provided in dollar amounts and in others it is provided as a percentage of total program costs. In some cases, the cost figures are budgeted amounts rather than expenditure amounts.

Although none of the Mid-Term reviews provided information on how much is being spent on capacity building, the need to do so is being recognized by some RBA-holders. For example, the FNI recognizes that some funds should be directed towards capacity building, particularly in the areas of career development, employment counselling, job creation strategies, and implementation of CATS.

2.7 Client and Community Satisfaction

• Are clients satisfied with the interventions they received (including e.g., generally with the intervention, interactions with program administrators, training courses attended, job skills upgrading, work experiences)?
• Are communities satisfied (if available from information collected in the regional reviews) with their involvement in design and delivery processes?

2.7.1 Client Satisfaction

Information on client satisfaction was provided through participant surveys or through interviews with community-based program delivery agents. From the information provided in the Mid-Term reviews, there is a high degree of satisfaction among program participants of the training/education they have received through RBA support. According to participants, they would not have been able to access further training or education without the RBA funding support. Under some RBAs, clients are particularly satisfied with the ongoing support and counselling provided by community-based delivery staff.

However, program participants expressed dissatisfaction over the following:

• variable quality of instructors;
• levels of funding for living allowance for those receiving Social Assistance;
• lengthy delays in the verification Métis ancestry process which, in turn, delay access to RBA program services;
• lack of clear information on RBA programs and levels of assistance among the communities; and,
• perception of political influence.

2.7.2 Community Satisfaction

Information provided on community satisfaction varied among the Mid-Term reviews. Much of this information was obtained through interviews with HRDC and RBA-holder delivery staff. Overall, there is consensus among RBA-holders that RBA programs have benefited communities, in both economic terms and in social terms. Although the extent to which communities are satisfied varies from RBA to RBA, a number of community satisfaction indicators were reported. They included the following:

• increased self respect through increased control of programs and creation of role models; (e.g., a more positive attitude towards education as a result of completing ABE courses has led to participants feeling that they are serving as role models for their children);
• increased sense of well-being within the family unit;
• community well-being;
• reduced numbers of crisis and police interventions in communities during the training period (this is particularly true for remote communities where high unemployment levels among youth lead to violence, abuse and suicide);
• low turnover rate of local delivery staff; and,
• increased levels of employment and reduced levels of assistance.

However, there was an indication expressed by some RBA-holders that community satisfaction would increase with the following improvements:

• increased focus on long-term programming;
• increased efforts aimed at capacity building; and,
• better marketing and communication strategies to enhance community awareness of RBA program services and the goals of the RBA itself.

2.8 Recommendations for Improvements

2.8.1 Recommendations

• What types of recommendations for improvements were typically made in the regional reviews?

Although the number and nature of recommendations reported varied from report to report, we identified the main ones (those that appeared more than once) and grouped them under common themes. Therefore, the following recommendations for improvement were made:

Process/Implementation

• That HRDC and the RBA-holder hold more formal and regular meetings to ensure information sharing and decision-making between the two partners.
• That the RBA-holder develop strategic/operational plans to clarify its long-term goals in human resources development (e.g., identification of gaps in future employment needs). This would assist the RBA-holder in its negotiations for continued funding once the RBA has terminated.
• That program participants undergo the proper skills assessment and testing prior to selecting their training program. The lack of career counselling/planning services has resulted in a lowered awareness of career choices and an overestimation of aptitude by the student while underestimating the degree of difficulty associated with the program.
• That all procedures/processes in place be reviewed to ensure that they are effective, efficient and clear.
• That a standardised client and program tracking system is established to ensure consistency in reporting across the geographic area covered by an RBA.

Client/Community Needs

• That there be more pre-employment and job-readiness training.
• That assistance is sufficient to meet the needs of women clients, particularly single mothers requiring childcare assistance.
• That the level of funding provided to needy students be revisited, particularly those on Social Assistance.
• There is a need to revisit guidelines pertaining to the funding of university courses, particularly courses offered in the third and fourth years.
• That the RBA-holder find better ways to promote the RBA programs to its membership as many students were unaware of the range of programs being offered.
• That the delivery system in place meets the needs of clients and communities.
• That instructors are qualified to meet the needs of their clients. This requires a review of the training of instructors.
• That there is equal access among all prospective clients to RBA programming.

Capacity Building

• That consideration be given to increasing the human resources required to deliver RBA programs, particularly as the demand for them is increasing significantly. Further, consideration should be given to training and allocating costs to support capacity building.
• That consideration be given to developing a training strategy for case management, employment and assessment counselling.
• That consideration be given to loans of HRDC staff to assist the RBA-holder develop capacity in areas where it is lacking.

2.8.2 Lessons Learned

• What lessons may be learned from the early implementation experiences of HRDC and its RBA partners?

The early implementation experiences of HRDC and its RBA partners resulted in a number of lessons learned. As for recommendations the nature and number of lessons learned differed from region to region, however, we were able to identify those lessons that are applicable to all RBAs. These included:

Program Design/Implementation

• The process of case management is a key component of delivering effective and efficient human resource development.
• The lack of systems support was a significant gap in the first years of RBA implementation. This had a negative impact on the capacity to deliver and maintain an accurate record of participants' progress.
• Existing administrative capacity is an important prerequisite to the success of the empowerment process. It is difficult for any agency to exert effective administrative control until they have developed the staffing structures, systems, procedures and policies needed to establish this control in the first place. Developing agencies need enhanced support, including staff secondments, if they are to be successful in assuming responsibility for program administration.
• A detailed transition plan can help clarify mutual roles and responsibilities in the transfer of a program from one administrative agency to another. Ideally such a plan should include a critical path that outlines exactly what needs to be accomplished, by when, and by whom.

Program Delivery

• Local control over the delivery mechanism is regarded as the most important element of the RBA.
• Delivery of RBA programs is most efficient where client numbers warrant it. Upon expiration of the Pathways to Success Program, the RBA process was the next logical step to devolve process responsibilities to Aboriginal human resource organizations. However, there must be a sufficiently large number of clients with suitable demographic characteristics to make service delivery feasible.
• The appeals process may be open to political influence and should be made more impartial.

Accountability

• It is important that the RBA-holder be accountable not only to HRDC but to its membership for impact of the RBA on client outcomes.

Client/Community Needs

• Provincial government policy on training support (treats it as income and subsequently reduces assistance) acts as a disincentive to participants on Social Assistance accessing the program. This should be made clearer to prospective clients at the outset.
• Having RBA promotional materials in the language of the RBA-holder's membership is an important aspect of RBA program delivery.
• There has been minimal linkage to the private sector. Private sector partnerships are essential to meet long term training and career development needs of individual clients.

Footnotes

5	This is a Contribution Agreement and not a Regional Bilateral Agreement.
6	Although the third Agreement between HRDC and the Katavik Regional Administration was not an RBA, it was included in the RBAs that were subject to a Mid-Term review.
7	All contribution agreements were operating at the time of this report.
8	The term 'Elder' is used in the guiding principles for National Framework Agreements.
9	There appears to be some confusion as to whether this is a regional policy or a corporate policy.
10	Aboriginal people can obtain services from HRCCs if they wish. However, they do not need to identify themselves as Aboriginal people.
11	Primary measures (savings to EI and SA) are intended to be available to RBA-holders on HRDC's Human Resources Investment (HRI) database to help them make decisions based on client results. At the time of this synthesis, the HRI database was just starting up and little information was available to RBA-holders. However, subsequent to the preparation of this synthesis, more data has been uploaded to the HRI database from the regions and, consequently, more information is being made available to RBA-holders.
12	As the first upload of CATS data to NHQ's HRI dataset was expected on April 1, 1998, there was no information available at the time of this synthesis to indicate how such information was being fed back into decision-making processes. However, subsequent to this synthesis, more data has been uploaded to NHQ's HRI dataset and more information is now available to the Newfoundland RBA-holders.
13	However, since the completion of these Mid-Term reviews, progress has been made in the implementation of data collection systems.
14	As noted previously in this report, more data has since been uploaded to the HRIF dataset at NHQ and, consequently, more information is now available to RBA-holders on primary measures.
15	Higher administration costs in 1997-98 due to higher than anticipated costs in establishing the Central Aboriginal Training Institute.
16	Based on average of costs in 2 regions.