Audit of Entitlement to Employment Insurance Benefits

Title Page

Audit of Entitlement to Employment Insurance Benefits

Project No: 6532/02

Internal Audit Services

Director General: J.K. Martin
Audit Director: G. Duclos
Team Leader: B. Cyr
Audit Team:
A. Moumouni
D. Fortin
J. Levesque
S. Auguste
S. Malichanh

APPROVED:

DIRECTOR: Gilles Duclos, January 19, 2004

DIRECTOR GENERAL: James K. Martin, January 18, 2004

November 2003

SP-622-04-04

Paper
ISBN : 0-662-36974-2
Cat. No.: HS3-4/622-04-04E

PDF
ISBN : 0-662-36975-0
Cat. No.: HS3-4/622-04-04E-PDF

HTML
ISBN : 0-662-36976-9
Cat. No.: HS3-4/622-04-04E-HTML

EXECUTIVE SUMMARY

The audit of entitlement to Employment Insurance (EI) benefit activities was planned in the 2002-2003 Resource Utilization Plan of Internal Audit and Risk Management Services (IARMS). Its timing was set to coincide with the audit conducted by the Office of the Auditor General and its scope was adjusted to avoid redundancy.

Employment Insurance is one of the major programs delivered by Human Resources Development Canada (HRDC) and entitlement to benefit is the main operational activity of this program. The amount of EI benefits paid out in 2002/03 was approximately $13 billion.

The scope of the audit, conducted across Canada, was determined by the following objectives:

• Entitlement to EI benefit activities and resources are planned, organized and directed so as to achieve established objectives and goals;
• Decisions on entitlement to EI benefits and resulting payments comply with legislation, regulations, jurisprudence and policies;
• Speed and quality of service, client satisfaction and communications related to entitlement to EI benefits meet their respective objectives and goals¹;
• The EI Fund, benefit programs and data are adequately protected against internal fraud and abuse.

The criteria associated with these objectives are listed in Appendix A.

Standard audit tools and procedures such as document and data analysis, individual and group interviews, direct observation and walk-through were used to determine the existence and adequacy of internal controls.

The effectiveness of the main quality control initiative with regard to entitlement to benefit activities was assessed through the examination of a sample of claims which had already been subjected to that quality control.

This internal audit was conducted in accordance with both the Treasury Board Policy on Internal Audit and the Institute of Internal Auditors' Standards for the Professional Practice of Internal Auditing.

The auditors conclude that:

• In general, the planning, organization and direction of entitlement to benefit activities and resources support the achievement of entitlement to benefit objectives and goals. The base and assumptions related to the resource allocation model should be clarified and reviewed against the results of the cost analysis as soon as they become available.
• Quality management initiatives, especially the National Quality Management Initiative and the Strategic Direction for Improving the Quality of Decisions and Benefit Payments (NQMI/SD), are present in all Regions, comprehensive and well managed. The audit has identified improvement opportunities with regard to the completeness and accuracy of the review of individual files by Insurance Program Advisors. In particular, the impact of regional/local practices should be measured and the use strictly controlled at the national level.
• Goals relating to quality and speed of service and client satisfaction are in place and results are monitored regularly (the IARMS audit did not cover the reliability and use of performance indicators and results which were to be covered in depth by the OAG).
• Protection of EI funds and data against internal fraud and abuse is generally good but could be improved in certain areas.

Overall, it is the opinion of the auditors that the entitlement to benefit activities reviewed are effectively managed and that the National Quality Management Initiative and the Strategic Direction for Improving the Quality of Decisions and Benefit Payments are contributing to the quality of the process.

1. INTRODUCTION

Employment Insurance is one of the major programs delivered by Human Resources Development Canada (HRDC). Entitlement to Employment Insurance (EI) benefits represents the major activity of the Employment Insurance Program. In 2002/03, nearly 3 million applications for EI were received in more than 400 points of service. Some 8,000 employees deliver, manage or support this activity, mainly in the EI Processing Centres but also at the National and Regional Headquarters, in the Human Resources Centres (HRC) and their satellites, and in the Regional Call Centres (RCC). The amount of EI benefits paid out in 2002/03 is approximately $13 billion.

Applications were traditionally submitted in person at points of service or sent by mail. As of April 2002, however, claimants can also complete their application for benefits through the internet using Application for employment benefit on-line (Appli-Web). The entitlement to benefit process can be simple and straightforward in the case of a regular claim following a loss of employment due to lack of work or complex, when, for example, a claim involves voluntary leaving, a labour dispute, multiple employment or other circumstances that require more complex calculations and/or more knowledge and judgment on the part of the processing officer.

There is a national quality management program called the "National Quality Management Initiative" (NQMI). Within that national initiative is the Strategic Direction (SD) for Improving the Quality of Decisions and Benefit Payments, which prescribes an approach and a methodology for all Regions which can also put in place additional quality management activities. The national program, hereinafter referred to the NQMI/SD, is based on the review of a representative sample of EI applications during the course of the year by specialized officers called Insurance Program Advisors (IPA).

The scope of the audit was determined by the following objectives:

• Entitlement to EI benefit activities and resources are planned, organized and directed so as to achieve established objectives and goals;
• Decisions on entitlement to EI benefits and resulting payments comply with legislation, regulations, jurisprudence and policies;
• Speed and quality of service, client satisfaction and communications related to entitlement to EI benefits meet their respective objectives and goals (the IARMS audit did not cover the reliability and use of performance indicators and results as initially planned which were to be covered in depth by the OAG);
• The EI Fund, benefit programs and data are adequately protected against internal fraud and abuse.

The criteria associated with these objectives are listed in Appendix A.

The audit was conducted across Canada and at National Headquarters.

Standard audit tools and procedures such as document and data analysis, individual and group interviews, direct observation and walk-through were used to determine the existence and adequacy of management controls.

To assess the effectiveness of the main quality control initiative with regard to entitlement to benefit activities, a sample of claims which had already been subjected to quality control under the NQMI/SD was drawn from each Region and examined by a group of IPA seconded to IARMS to determine the completeness and accuracy of the initial review.

This internal audit was conducted in accordance with both the Treasury Board Policy on Internal Audit and the Institute of Internal Auditors Standards' for the Professional Practice of Internal Auditing.

2. AUDIT FINDINGS

All significant audit findings are presented in this section in accordance with the audit objective(s) and criteria, which are described in detail in Appendix A, Audit Objectives, Criteria, and Methodology. They include assurance statements on all of the criteria regardless of whether or not the performance expectations have been met.

Audit Objective:

The objective of this audit was to provide assurance that the overall Entitlement to EI Benefits activities are appropriately managed, including assurance that participants comply with the eligible criteria as stated in the Entitlement to EI Benefits Terms of Reference.

2.1 Entitlement to EI benefit activities and resources are planned, organized and directed so as to achieve established objectives and goals

Audit Criterion No. 2.1.1: Planning is conducted and a budget set each year at the national, regional and local levels for entitlement to benefit activities in accordance with corporate standards.

Planning is conducted and a budget is set each year at the national, regional and local levels for entitlement to benefit activities. Each level conducts its own planning in order to meet the objectives and target, working in close collaboration with the other levels. Every Region has developed an allocation model to redistribute the resources obtained from National Headquarters.

The Regions allocate resources according to their priorities but are required to report periodically on discrepancies between actual and budgeted expenses, productivity levels and the achievement of targets related to performance indicators and specific initiatives.

Audit Criterion No. 2.1.2: The allocation of entitlement to benefit resources, including those assigned to quality management, is based on a model and principles that are clear, well defined, known and understood by all the managers involved.

Employment Insurance activities are charged to the EI Fund. Every year, HRDC receives from Treasury Board an authorization to spend on EI activities based on the following:

• A Block (A Base) authorization;
• A claims processing authorization based mainly on the projected volumes of initial and renewal claims for each region;
• Prioritized authorizations for specific projects and initiatives.

The three authorizations are discussed below:

Block funding authorization: Auditors examined a document detailing the distribution of block funding (A Base) in 2002-03 and 2003-04 for several HRDC activities, including Insurance. Although specific as to the amount allocated for the various activities, it was not possible to positively identify all of the activities that belonged to Employment Insurance and the auditors were not able to find a rationale or an explanation supporting the level of allocation for each activity.

Claims processing authorization: The ratio used for the claim processing (variable) authorization at the regional level are established every years by NHQ in consultation with the Regions. The following figures were gathered.

In 2003-2004, funding to the Regions was based on a total projected volume of 2.6 million initial, fishing and renewal claims. Treasury Board based its authorization for the same period on a total projected mix of 3.3 million claims. The difference resulted in a supplementary allocation of about $27.2 million, which was used to fund certain initiatives and create a contingency reserve.

A workload fluctuation contingency planning policy is in place to assist the Regions if the actual number of claims exceeds the budgeted volume by more than 5%.

The productivity parameters used in the 2003-04 budget are based on the handling of 1,022 initial claims, 1,577 fishing claims or 3,066 renewal claims per Full-Time Equivalent (FTE). On average, FTEs are expected to be engaged in productive work 77% of their time.

The average salary authorization for employees involved in entitlement to benefit activities for fiscal year 2002-2003 was $42,000, and the actual average salary paid for that period was $43,667. The average salary allocation for fiscal year 2003-2004 is $43,200.

It was not possible to identify the rationale for all of the above figures. A unit cost project on entitlement to benefit activities is under way. Auditors were informed that NHQ plans to review the Employment Insurance resource allocation model once this study has been completed.

Prioritized authorization: In 2003-04 an amount of $6.2 million was divided among the nine Regional Call Centres (RCC) on the basis of the number of calls anticipated. This allocation is non-recurring and is not intended to cover the full operating costs of the RCC. Regions are expected to finance the difference out of their own operating budgets, using the savings that ongoing technological improvements, operational streamlining in Processing Centres and transfer of workload from the HRC to the RCC are expected to generate.

Amounts for specific initiatives such as quality are established on an annual basis. The funding comes either directly from Treasury Board, in response to specific submissions such as Small Weeks funding, or from Employment Insurance Branch at NHQ which supports certain initiatives, such as EI Program training, out of its own budget.

Resources for quality management are allocated in two separate blocks. One amount based on historical data is allocated annually to fund local initiatives such as technical support for Employment Insurance, training and the coordination of quality-related activities. The second block is allocated specifically to the Strategic Direction (NQMI/SD) for improving the quality of entitlement to benefit decisions in general and payments in particular. It is distributed according to the number of Insurance Program Advisors (IPA) engaged in this initiative.

Audit Criterion No. 2.1.3: Efficiency and economy objectives in terms of resource utilization are set and communicated to all managers concerned.

Over the last few years, Employment Insurance has introduced a number of initiatives to improve efficiency, economy and/or client service. Examples of these initiatives are Appli-Web, Teledec and Direct Deposit. Results are measured and communicated in the Balanced Scoreboard. Further changes to the delivery of EI programs that are now under way will result in new delivery mechanisms and revised efficiency and economy objectives.

Audit Criterion No. 2.1.4: Resources are organized in such a way as to support the effective performance of entitlement determination activities, while still meeting efficiency and economy objectives.

An organizational structure is in place for all activities relating to entitlement to benefits. Each Region organizes and distributes its resources according to its priorities and geographic and demographic characteristics, but follows the same basic model in which services are divided among the HRC, the Processing Centres and the RCC. A trend to centralize processing activities and enhance the use of electronic delivery devices is evident in all Regions.

Audit Criterion No. 2.1.5: The duties and responsibilities of the entitlement to benefit managers, supervisors and employees are clearly defined and communicated.

Work descriptions detailing the duties and responsibilities of managers, supervisors and employees engaged in entitlement to benefit activities are available and communicated.

Audit Criterion No. 2.1.6: Resources allocated to quality management are adequate and used for that purpose.

An analysis of regional and local budgets shows that the resources specifically allocated to quality management are used for that purpose.

The resources allocated to the Strategic Direction for improving the accuracy of EI benefit payments are sufficient to cover the prescribed minimum number of cases to be reviewed under the NQMI/SD but increased coverage is generally funded through regional resources. The quality assurance review conducted by IARMS has shown that the Insurance Policy Advisors who conduct the NQMI/SD review have the required basic skills and knowledge to perform this function.

Audit Criterion No. 2.1.7: Quantitative and qualitative quality management objectives and targets are set and met nationally, regionally and locally.

Quantitative and qualitative quality management objectives and targets are set nationally, regionally and locally. They are found in business and/or quality plans. The NQMI/SD objectives are formal and aimed at ensuring that each Region improves its quality every year. Based on benchmarks, a target is negotiated with each Region. In 2002-03, three Regions out of ten achieved their target and all Regions showed an improvement in quality. However, the use by some Regions of certain regional/local practices instead of the national standards has resulted in a distortion in the reporting of the NQMI/SD results that approximates 5% in average.

Audit Criterion No. 2.1.8: A clear responsibility and accountability framework exists at all levels of entitlement to benefits.

The organization charts for the organization set a coherent responsibility and accountability framework, and performance contracts for senior managers set clear levels of individual responsibility and accountability.

The tools for measuring quantitative performance are the financial budget and the Balanced Scorecard. A number of qualitative elements developed within Employment Insurance performance contracts and business plans are also evaluated.

Average productivity standards are expected for groups of employees involved in entitlement to benefit activities. Measurement of the level of individual productivity, at least in an informal way, is conducted in most Regions.

Recommendation No. 1:

The existing resource allocation model should be clarified with regard to the development and calculation of the block funding allocation.

The unit cost determination project should be applied to the claim processing (variable) allocation as soon as they become available.

Management Response:

1. Regions are in the process of identifying their block funding allocation breakdown with FAS. The appropriateness of allocations will then be determined by ensuring that resources are aligned to priorities.
2. EI resource allocation model will be determined based on the unit costing information available and the block funding allocation exercise.
3. A comprehensive review of the costs associated with the delivery of the EI program at the local, regional and national level is underway.

2.2 Decisions on Entitlement to EI Benefits and Resulting Payments Comply with Legislation, Regulations, Jurisprudence and Policies

Audit Criterion No. 2.2.1: A quality management program is in place in every Region.

Quality management programs are present in all ten Regions. Specifically, the Regions started implementing the NQMI in April 2000. The Strategic Direction (SD), which deals with the overall quality of the benefit process, including the accuracy of benefit payment, became mandatory in April 2001.

Insurance policy advisors (IPA) assigned to the NQMI/SD represented in 2002-03, 52 Full Time Equivalent (FTE). They are expected to review claims chosen randomly from a sample of 80 recent initial claims selected automatically by a computer program each month. The minimum number of claims to be reviewed is 40 by FTE and by month in average. Some Regions have implemented their own Quality Assurance Program. They review a sample of claims that have already been subjected to the NQMI/SD.

To promote quality, one Region has developed an initiative called "Impact of Non-Quality" to sensitize Agents and all other employees to the cost and impact of an error in a claimant's file. A selection of cases with an initial error are traced to measure the additional work that has been required to redress the situation. These cases are discussed during training and development sessions to sensitize employees to the importance of doing the job well the first time.

Audit Criterion No. 2.2.2: The portion of the regional quality management programs that cover initial decisions meets the standards and criteria of the National Insurance Quality Management Initiative.

The guidelines, standards and criteria of the NQMI/SD are followed in the ten Regions, which all use the National Claim File Quality Monitoring Guide. All Regions meet and most exceed the minimum sample size of initial claims prescribed by the NQMI/SD. Some regional/local particularities were identified. These are discussed below.

Audit Criterion No. 2.2.3: In addition to initial decisions, regional quality management programs cover the other decisions taken during the benefit period.

In addition to initial decisions, many Regions also review a sample of renewal or revised claims, appeals, etc., as well as all the decisions of new employees or employees who have recently undergone training. These results are used locally or regionally and all activities are reported quarterly at the national level through quality report.

Audit Criterion No. 2.2.4: The results of the quality management program in a Region are representative of the actual quality of the decisions in that Region; the definition and determination of errors are consistent and error rates are comparable between regions.

The results of the NQMI/SD review are presented quarterly as a percentage of files "in order". A file that is "in order" is one that meets the twelve objectives set out in the National Claim File Quality Monitoring Guide (a summary of these objectives is given in Appendix D). Some of these objectives are related to the financial aspect of the claim while some deal with non-financial elements such as the quality of fact-finding or the quality of communications. Where claims are "not in order", the deficiencies (i.e. the situations in which the objectives are not met) are coded and reported.

A quality assurance of 1,606 initial claims which had already been subjected to NQMI/SD examination between April and August 2002 was conducted by IARMS in the fall of 2002. The quality assurance process used the national NQMI/SD standards. In about 15% of the cases, a file that had been reported to be "in order" in the initial NQMI/SD review was found not to be "in order" in the second examination.

In nearly 5% of these instances, the difference came from regional/local practices which consider to be "in order" certain situations that are "not in order" under the national NQMI/SD standards. National NQMI/SD management was generally aware of the use of regional/local practices instead of national standards but the quality assurance has shown the importance and impact of this practice which accounts for variations of up to 23% in the Regions.

In less than 1% of the cases, a file considered not to be "in order" in the NQMI/SD review was found to be "in order" in the quality assurance review. A detailed description of the quality assurance process and the detailed results can be found in Appendix C.

IARMS has been informed that a working group has been created to analyse the results of the quality assurance and address the issues they raise in order to improve the process.

Audit Criterion No. 2.2.5: Errors detected by quality management programs are promptly brought to the agent involved and corrected if required.

IPAs record the results of the NQMI/SD review of each file on a standard report sheet. In some Regions, the report is given to the agent who adjudicated the file or to the immediate supervisor whether or not a deficiency has been detected. In other Regions, the report is sent only if the file contains deficiencies.

When a deficiency is reported, the codes of the NQMI/SD objectives which are not met are indicated and a short explanation is generally provided. The Agent is required to return the report to the IPA with an annotation indicating whether or not he/she agrees with the finding.

The IPA is not expected to correct the deficiency; this is the responsibility of the Agent who worked on the file. Not all IPAs follow-up to make sure that all deficiencies have been corrected.

Audit Criterion No. 2.2.6: Quality management programs analyze the results of the review of decisions to identify the causes of the errors and the trends, and the identification of the causes of errors and trends leads to effective measures to prevent their repetition and improve the quality of decisions over time.

Each IPA prepares a quarterly report for the Regional Quality Coordinator on the number of files examined under the NQMI/SD and the number and type of deficiency identified. The majority of these reports also include some analysis of local trends.

The Regional Quality Coordinator produces a quarterly report summing up local reports and providing regional-level trend and causal analysis.

The National Quality Coordinator at NHQ rolls up and summarizes the regional reports and conducts a trend and causal analysis at the national level. This report is sent to senior management at NHQ and distributed to Regional Executive Heads. Since April 2002, this report has been produced twice a year.

Although all IPAs perform the basic review of the prescribed minimum number of files, the scope of their involvement in terms of analysis, feedback and prevention under the NQMI/SD review varies depending on their availability and other duties. At the time of the audit, Insurance Quality Services had just completed a comprehensive review of the role and responsibilities of the IPA and intend to provide functional direction relating to IPA responsibilities within and between the Regions.

Audit Criterion No. 2.2.7: Quality management programs are assessed periodically and measures are taken to ensure or maintain their efficiency and effectiveness.

Since its introduction, the NQMI/SD has been reviewed and assessed every year through national seminars and consultations. A Standards and Simplification Committee was created to discuss standards and simplify procedures.

Recommendation No. 2:

The results of the quality assurance conducted during the present audit by IARMS should be used to identify the cause of the difference between the initial and subsequent results "in order" to increase the accuracy and consistency of NQMI/SD examination and reporting. In particular, the use of regional/local practices should be specifically authorized and controlled at the national level and its impact on results should be measured and taken into account in the reports.

Insurance, through NQMI/SD, should ensure that feedback to agents is consistently provided, whether or not their cases are "in order", and that corrective action is taken whenever required.

Management Response:

Actions Undertaken:

1. A web based report was implemented in April 2003 to enhance the capacity to capture error statistics and improve our ability to identify trends. In addition, future enhancements of the Web based reporting system for IPAs will allow for agent feedback to be captured.
2. Memo to field entitled: "Clarification of the Roles and Responsibilities of the IPA" sent July 2003 to Regional Executive Heads; also discussed at the IQS Workshop and the National Insurance Committed meeting in October 2003.
3. A workshop was held with Regional Quality Coordinators in Sept 2003 to discuss improving quality and promoting the balance between productivity and quality; all regions encouraged to use the initiative entitled "Impact of Non-Quality" as well as "Productive Quality Performance"; will follow-up with regional coordinators on bimonthly conference calls.
4. The Standardization and Simplification Committee regularly reviews issues to ensure uniformity and that there is as much consistency as possible across regions.
5. Insurance Program Advisor training package developed and will be piloted in Regina in Dec 2003. This training package will highlight the need to ensure follow-up of errors. It will also emphasize the need to provide agents with feedback in cases where there is an error as well as when claims are in order.

Actions to be Undertaken:

In, addition Insurance, in collaboration with the regions, will analyze the results of the files reviewed during the audit and will develop a comprehensive action plan to address identified discrepancies.

A comprehensive action plan will be developed to improve the accuracy and consistency of Quality Monitoring (QM), ensure policies and procedures are applied consistently across the country, ensure feedback is provided consistently to agents and that corrective actions are followed up on.

Audit Criterion No. 2.2.8: The Comprehensive Tracking System (CTS) gathers reliable and nationally representative data on the accuracy of benefit payment.

The CTS is a national activity aimed at providing an indication of the integrity of the EI fund administration by reporting the dollar value of EI benefit overpayments and underpayments. These underpaid or overpaid benefits are classified according to their cause: error by HRDC staff; false, erroneous or omitted information by the claimant; false, erroneous or omitted information by the employer.

The CTS has been in operation for more than 20 years and has been audited by the Auditor General.

Audit Criterion No. 2.2.9: The results of the Comprehensive Tracking System (CTS) are analyzed and taken into account to validate and improve regional quality management programs.

The NQMI/SD was largely developed around the Comprehensive Tracking System analysis grid and codes, but the two programs differ in several ways.

The CTS procedures require a detailed and comprehensive examination of all the elements that may have an impact on the determination of the amount or duration of EI benefits. This examination may take more than a day and may require direct contacts with HRDC staff and employers. For that reason, the number of cases reviewed from each Region is limited and the results are officially reported at the national level only.

The NQMI/SD is also concerned with the accurate determination of the amount and duration of benefits, but its main objective is to assess the overall entitlement to benefit process. It has a wider scope and covers several elements related to quality of service and the adjudication process that have no direct financial impact (a summary of the objectives used by the NQMI/SD is given in Appendix D). Its results are collected and reported at the local, regional and national levels.

Comparing the results of the CTS and the SD requires caution. Only a portion of the CTS results originate from an error made by an HRDC employee. There is little chance that many of the factors leading to the overpayments or underpayments identified during the CTS review would have been known by the Insurance agent or detected by the NQMI/SD. The reason is that the CTS, as stated above, can require up to a full day on a file, and in-depth investigation with employers and HRDC staff may be conducted to verify and confirm all data and information. Insurance agents, on the other hand, are expected to handle several claims a day. The cost of having every Insurance agents spend up to a day on claims would be prohibitive.

It is therefore normal that the CTS and the NQMI/SD results differ, and over the last two years, both the relative amount of overpayments and underpayments reported by the CTS and the percentage of cases deemed to be "in order" in the NQMI/SD have shown improvement.

Audit Criterion No. 2.2.10: Claim processing employees are informed of the legislation, regulations, jurisprudence, policies and guidelines required to perform their functions; reference documentation is regularly updated and employees are promptly notified of any significant change.

In all the Regions visited, IPA were providing information and explanations to agents on EI legislation, regulations, jurisprudence, policies and guidelines. Generally, changes are announced and explained by e-mail. Formal training is provided periodically and brief information sessions are generally held on a weekly basis, especially in periods of procedural or legislative change. Many employees interviewed wished they had more time to keep up to date on all the changes. They also felt that the benefit of training is maximized when it is received just before the implementation of the change or new legislation.

Audit Criterion No. 2.2.11: Processes are in place to identify training needs with regard to entitlement to benefits.

The results of the NQMI/SD are analyzed at the national level to identify the probable cause of the deficiencies detected. This provides the basis for the determination of training needs from a national perspective. Some sort of cause analysis is also conducted at local and regional levels, but that analysis is not consistent across the Regions.

In addition to NQMI/SD results, Insurance also uses other sources of information to identify training needs. These include statistical data, surveys, monthly conference calls, specific training needs analysis questionnaires and partnerships with regional managers and NHQ partners.

Audit Criterion No. 2.2.12 Training plans and material are developed and updated to meet the training needs identified.

Training programs and material are developed by NHQ, based on the needs identified. In addition, national training material will be produced or updated when there is a change in the Legislation, the procedures or the systems. Training material is available on the Employment Insurance Web site. The Regions can develop their own training plans but they must use the national training material as stipulated in the National Insurance Training Policy to avoid duplication of training efforts and ensure standardization of service delivery.

Audit Criterion No. 2.2.13: Managers, supervisors and employees receive timely and effective training as per the training plans. Formal training in entitlement to benefit is given regularly. This training is generally given to groups of employees performing a specific function when legislative, procedural or system changes occur. On an individual basis, coaching and monitoring are the prevalent ways to meet the specific needs of the employees.

Training programs are available to employees and accessible on the intranet. NHQ had taken several initiatives to inform employees of the content of the training material available.

Audit Criterion No. 2.2.14: The work of the employees in claims processing is supervised and regularly monitored.

Supervision and monitoring of entitlement to benefit activities exist in all regions but to varying degrees, depending on the background of supervisors and the availability of IPA.

2.3 Speed and Quality of Service, Client Satisfaction and Communications Related to Entitlement to EI Benefits meet their Respective Objectives and Goals

Audit Criterion No. 2.3.1: National, regional and/or local objectives and goals have been set with regard to speed and quality of service, communication with the public and client satisfaction; objectives and goals are clear, achievable and are known by the managers and employees concerned.

There are four main objectives set out in the Employment Insurance NQMI: clear communications, fairness, accuracy and speed of service. These four objectives are expected to increase client satisfaction.

Standards and goals set for each objective are as follows:

• Clear communications: Every communication, whether oral or written, must be clear, concise, accurate, easily understood and courteous;
• Fairness: All activities relating to Insurance must be conducted in a professional and just manner, which demonstrates that client rights are respected;
• Accuracy: All calculations and decisions must comply with the EI Act, jurisprudence and national policies and be supported by a complete fact book and the required documents/information. All requests for information are given complete and accurate answers;
• Speed: Applications for EI benefits and/or requests for information must be processed within a time limit which corresponds to the speed of service goals. This principle applies to the processing of benefit applications as well as to all services rendered in person, by telephone, in writing or through new technologies.

Employees and managers involved in entitlement to benefit activities are informed of these objectives and regularly updated.

Audit Criterion No. 2.3.2: Responsibility and accountability in the achievement of the objectives and goals are clearly established.

Employment Insurance managers are expected to achieve all the quality of service goals. The speed of service targets are incorporated into the performance contract of managers and reported in the Balanced Scorecard².

Fairness, accuracy and quality of communications are assessed through various quality management programs and client satisfaction is measured through periodic client surveys.

Audit Criterion No. 2.3.3: No manager or employee is held responsible or accountable for the achievement of objectives or goals over which he/she has no control.

Several managers expressed concerns about the "speed of payment or otherwise entitled" (SOPOE) target which requires that 75% of claimants receive their first payment within 28 days of the starting date of the benefit period, whether they applied at the beginning or at the end of that period. Managers agree with the principle of quick service but contend that they have little control over applicants' behaviour, particularly over their diligence in submitting the application and the required documentation. At the time of the audit, that goal had been achieved in three Regions across Canada.

Audit Criterion No. 2.3.4: Reliable and accurate measures of the quality and speed of service, the quality of communications and client satisfaction are taken regularly and communicated, and measures are taken when objectives and goals are not met.

Measures and targets related to speed of service are included in the Employment Insurance Balanced Scorecard and reported on monthly. These measures include the time between an application or the commencement of a benefit period and the first payment, the time between the filing of an appeal to the Board of Referees and the scheduling, and the time a person must wait to be served in person or on the telephone.

Executive managers are linked to these targets through their performance contract. When these targets are not met, they must provide a plan to redress the situation.

Regional Call Centres (RCC) monitor a sample of incoming calls and report on the results. The NQMI/SD review included letters sent to claimants via the Corporate Letter System (CLS) to ensure that the proper messages are being sent out.

Client satisfaction is measured through national surveys. The most recent results made public in 2002 were from a survey conducted in June 2001. The survey covered telephone services in RCC and service in person at HRCC. According to the survey, the main area requiring improvement is the waiting times at RCC and in-person at the HRCC. Every Region has been asked to develop an action plan as part of the Service Improvement Initiative.

2.4 The EI Fund, Benefit Programs and Data are Adequately Protected Against Internal Fraud and Abuse

Audit Criterion No. 2.4.1: Claimant files are kept in a secure area and access is restricted to employees who need access to these files

Claimant files contain personal, sometimes highly sensitive confidential information. Unnecessary or unauthorized viewing of such files is a violation of the Privacy Act, and the theft of a claimant file for whatever reason could have serious consequences for both the claimant and the Department.

All in-person points of service visited kept claimant files in a non-public area but the physical barriers and other internal controls to prevent unauthorized access to the files varied from place to place. A person in the non-public area who has no valid reason to access the files might be able to do so without being noticed or challenged.

A file follow-up system called the DTS (Document Tracking System) was in place in all offices visited. Its purpose is to record files withdrawn by employees. There were no mechanisms in the offices visited to ensure that every withdrawal is recorded, and an entry in the DTS can be permanently erased. The DTS is therefore a file circulation control system, but it is not an effective internal control against improper access to and withdrawal of files. If one does perform an unauthorized or unjustified withdrawal, it would likely not record it on the DTS.

Recommendation No. 3:

A risk assessment should be conducted in the various EI points of service to ensure that appropriate control is exercised to prevent or detect unauthorized access to and withdrawal of client files.

Management Response:

The Insurance Branch takes very seriously the need to ensure appropriate controls are in place to prevent or detect unauthorized access to client files. We are taking steps to safeguard information by conducting a Process Mapping and Risk Assessment workshop in the Fall, 2003 to identify risk and develop mitigating strategies. Based on the outcomes of the workshop, senior management will identify issues that must be addressed, formulate mitigating strategies and develop a comprehensive plan to address the identified issues.

Audit Criterion No. 2.4.2: Workstations linked to benefit programs and EI databases are protected from unauthorized access or use.

User IDs and passwords are the basic logical controls used to prevent unauthorized use of workstations and access to programs and data. Although a workstation can potentially be used by several persons, the policy entitled "Requirements relating to Passwords" stipulates that "a password is given for the exclusive use of its holder ... and ... it is prohibited to access files, databases, computers and other system resources through the use of common passwords." The auditors were informed of instances where groups of employees are using in common a certain number of workstations already open, especially at the front-end in periods of rush.

Recommendation No. 4:

The use in common of workstations by a group of employees should be discontinued.

Management Response:

The Insurance Branch agrees with this recommendation and to reinforce the importance of this, the Regional Executive Heads will be asked to remind their managers of the policy concerning the usage of usercode/passwords and to ensure that this practice is discontinued.

A review of the current policies, procedures and monitoring process will be undertaken to ensure the appropriate use of usercodes/passwords.

Using a screen saver with a password was recommended but not mandatory in most of the offices visited. This increases the risk of someone using an unattended workstation already on line under the identity of the person who has logged on. With the migration to Outlook, every departmental workstation should be equipped with a password-protected screen saver, reducing the risk of unauthorized use.

Audit Criterion No. 2.4.3: Access to entitlement to benefit programs and data is restricted to employees who need this access to perform their functions.

At the time of the audit, most managers and employees involved in entitlement to benefit activities had been given the same broad logical access to most EI programs and data. This concept is generally referred to as the "Universal Agent" approach. This does not imply that Claim Preparation and Insurance Agents must perform all the operations on the same claim i.e. creation, activation and termination, but they have the technical capacity to do so.

This approach is now under review and an Insurance group has been created to look at the real logical access needs for the various functions associated with entitlement to benefit activities and to restrict employees to the accesses they need to perform their specific duties.

The issue of logical access to EI programs and data is covered in depth in the Privacy audit and will be reported in the Privacy audit report that will be issued shortly.

3. CONCLUSION

The auditors conclude that:

• In general, the planning, organization and direction of entitlement to benefit activities and resources support the achievement of entitlement to benefit objectives and goals. The base and assumptions related to the resource allocation model should be clarified and reviewed against the results the cost analysis as soon as they become available.
• Quality management initiatives, especially the National Quality Management Initiative and the Strategic Direction for Improving the Quality of Decisions and Benefit Payments (NQMI/SD), are present in all Regions, comprehensive and well managed. The audit has identified improvement opportunities with regard to the completeness and accuracy of the review of individual files by Insurance Program Advisors. In particular, the impact of regional/local practices should be measured and the use strictly controlled at the national level.
• Goals relating to quality and speed of service and client satisfaction are in place and results are monitored regularly (the IARMS audit did not cover the reliability and use of performance indicators and results which were to be covered in depth by the OAG).
• Protection of EI funds and data against internal fraud and abuse is generally good but could be improved in certain areas.

Overall, it is the opinion of the auditors that the entitlement to benefit activities reviewed are effectively managed and that the National Quality Management Initiative and the Strategic Direction for Improving the Quality of Decisions and Benefit Payments are contributing to the quality of the process.

In our professional judgment, sufficient and appropriate audit procedures have been conducted and evidence gathered to support the accuracy of the conclusions reached and contained in this report. The conclusions are based on a comparison of the situation as it existed at the time against the audit criteria. The conclusions apply only to the entitlement to benefit activities examined.

This internal audit was conducted in accordance with the Treasury Board Policy on Internal Audit and the Institute of Internal Auditors' Standards for the Professional Practice of Internal Auditing.

APPENDIX A: AUDIT OBJECTIVES, CRITERIA AND METHODOLOGY

OBJECTIVE 1

Entitlement to EI Benefit Activities and Resources are Planned, Organized and Directed so as to Achieve Established Objectives and Goals.

1.1	Planning
	1.1.1	Annual planning and budgeting for entitlement to benefit activities are conducted at the national, regional and local levels.
	1.1.2	Resource allocation for entitlement to benefit activities, including resources allocated to quality management, is based on clearly defined models and principles which are known and understood by all managers involved.
	1.1.3	Efficiency and economy objectives and goals for resource utilization are set and communicated to all managers involved.
1.2	Organization
	1.2.1	Resources are organized in order to support the effective performance of entitlement to benefit activities while meeting the objectives for efficiency and economy.
	1.2.2	Tasks and responsibilities of managers, supervisors and employees in entitlement to benefit are clearly defined and communicated.
	1.2.3	Resources allocated to quality management are sufficient and used for that specific initiative.
	1.2.4	National, regional and local quantitative and qualitative standards objectives and goals for quality management are set and are met.
	1.2.5	A clear responsibility and accountability framework for entitlement to benefit is present at all levels.

OBJECTIVE 2:

Decisions on EI entitlement to benefit and resulting payments comply with legislation, regulations, jurisprudence and policies.

2.1	Quality Management
	2.1.1	A quality management program is in place in every Region.
	2.1.2	The portion of the regional quality management programs that cover initial decisions meets the standards and criteria of the national Insurance Quality Management Initiative.
	2.1.3	In addition to initial decisions, regional quality management programs cover the other decisions taken during the benefit period.
	2.1.4	The results of the quality management program in a Region are representative of the actual quality of the decisions in that Region.
	2.1.5	The definition and determination of errors are consistent and error rates are comparable between Regions.
	2.1.6	Errors detected by quality management programs are promptly brought to the agent involved and corrected if required.
	2.1.7	Quality management programs analyze the results of the review of decisions to identify the causes of the errors and the trends.
	2.1.8	The identification of the causes of errors and trends leads to effective measures to prevent their repetition and improve quality of decisions over time.
	2.1.9	Quality management programs are assessed periodically and measures are taken to ensure or maintain its efficiency and effectiveness.
2.2	Comprehensive Tracking System
	2.2.1	The Comprehensive Tracking System (CTS) gathers reliable and nationally representative data on the accuracy of benefit payment.
	2.2.2	The results of the CTS are analyzed and taken into account to validate and improve regional quality management programs.
2.3	Training and development
	2.3.1	Claim processing employees are informed of the legislation, regulations, jurisprudence, policies and guidelines required to perform their functions.
	2.3.2	Reference documentation is regularly updated and employees are promptly notified of any significant change.
	2.3.3	Processes are in place to identify training needs with regards to entitlement to benefit.
	2.3.4	Training plans and material are developed and updated to meet the training needs identified.
	2.3.5	Managers, supervisors and employees receive timely and effective training as per the training plans.
2.4	Supervision and Monitoring
	2.4.1	The work of the employees in claims processing is supervised and regularly monitored.

OBJECTIVE 3:

Speed and quality of service, client satisfaction and communications related to entitlement to Employment Insurance benefits meet their respective objectives and goals.

3.1	Quality objectives
	3.1.1	National, regional and/or local objectives and goals have been set with regards to speed and quality of service, communication with the public and client satisfaction.
	3.1.2	Objectives and goals are clear, achievable and are known by the managers and employees concerned.
	3.1.3	Responsibility and accountability in the achievement of the objectives and goals are clearly established.
	3.1.4	No manager or employee is held responsible or accountable for the achievement of objectives or goals over which he/she has no control.
3.2	Training
	3.2.1	Managers and employees receive training and are provided with tools that support the achievement of the objectives and goals for which they are responsible and accountable.
3.3	Internal controls
	3.3.1	Reliable and accurate measures of the quality and speed of service, quality of communications and client's satisfaction are taken regularly and communicated.
	3.3.2	Measures are taken when objectives and goals are not met.

OBJECTIVE 4:

The EI Fund, benefit programs and data are adequately protected against internal fraud and abuse.

4.1	Access EI benefit programs and data
	4.1.1	Claimant files are kept in a secure area and access is restricted to employees who need access to these files.
	4.1.2	Workstations linked to benefit programs and EI databases are protected from unauthorized access or use.
	4.1.3	Access to entitlement to benefit programs and data is restricted to employees who need this access to perform their functions.
4.2	Internal Control
	4.2.1	Internal controls are in place to prevent or detect fraud and abuse in the creation or modification of benefit periods.
Note: Some of the above criteria were grouped during the audit to facilitate the fieldwork and the analysis, and to prevent any duplication in the report. Also some terminology changes were made such as Benefit Entitlement replaced by Entitlement to EI benefit to better reflect the terms used and understood by the community

SCOPE

The audit covered all the processes, activities and internal controls related to entitlement to EI benefits. This included the planning and organization of human, financial and operational resources, the entitlement to benefit process as such, the quality of service and communications and the protection of the EI fund against internal fraud and abuses.

The audit was conducted in HRDC ten administrative Regions and at National Headquarters.

METHODOLOGY

Standard audit tools and procedures such as document and data analysis, individual and group interviews, direct observation and walk-through were used to determine the existence and adequacy of internal controls.

To assess the effectiveness of the quality management initiative, a sample of claims which had already been subjected to quality control was drawn in each Region and examined by a group of experts seconded to IARMS to determine the completeness and accuracy of the initial review.

APPENDIX B

MANAGEMENT ACTION PLAN

IARMS Recommendations	Management Action Plan	Expected Completion       Date*	Responsibility
Audit Criteria 2.1.8: A clear responsibility and accountability framework exists at al levels of entitlement to benefits Recommendation 1 : The existing resource allocation model should be clarified with regard to the development and calculation of the block funding allocation. The results of unit cost determination project should be applied to the claim processing (variable) allocation as soon as they become available.	1. Regions are in the process of identifying their block funding allocation breakdown with FAS. The appropriateness of allocations will then be determined by ensuring that resources are aligned to priorities.	Completed	Insurance Branch
	2. EI resource allocation model will be determined based on the unit costing information available and the block funding allocation exercise.	In place for 2005-2006
	3. A comprehensive review of the costs associated with the delivery of the EI program at the local, regional and national level is underway.	October 2004
Audit Criteria 2.2.7: Quality management programs are assessed periodically and measures are taken to ensure or maintain their efficiency and effectiveness. Recommendation 2: The results of the quality assurance conducted during the present audit by IARMS should be used to identify the cause of the difference between the initial and subsequent results 'in order' to increase the accuracy and consistency of NQMI/SD examination and reporting. In particular, the use of regional/local practices should be specifically authorized and controlled at the national level and its impact on results should be measured and taken into account in the reports. Insurance, through NQMI/SD, should ensure that feedback to agents is consistently provided, whether or not their cases are 'in order', and that corrective action is taken whenever required.	Actions Undertaken 1. A web based report was implemented in April 2003 to enhance the capacity to capture error statistics and improve our ability to identify trends. In addition, future enhancements of the Web based reporting system for IPAs will allow for agent feedback to be captured.	Completed	Insurance Branch
	2. Memo to field entitled: "Clarification of the Roles and Responsibilities of the IPA" sent July 2003 to Regional Executive Heads; also discussed at the IQS Workshop and the National Insurance Committee meeting in October 2003.
	3. A workshop was held with Regional Quality Coordinators in Sept 2003 to discuss improving quality and promoting the balance between productivity and quality; all regions encouraged to use the initiative entitled "Impact of Non-Quality" as well as "Productive Quality Performance"; will follow-up with regional coordinators on bi-monthly conference calls.
	4. The Standardization and Simplification Committee regularly reviews issues to ensure uniformity and that there is as much consistency as possible across regions.
	5. Insurance Program Advisor training package developed and will be piloted in Regina in Dec 2003. This training package will highlight the need to ensure follow-up of errors. It will also emphasize the need to provide agents with feedback in cases where there is an error as well as when claims are in order.
	Actions to be Undertaken 1. Insurance, in collaboration with the regions, will analyze the results of the files reviewed during the audit and will develop a comprehensive action plan to address identified discrepancies.	Overall March 2005 (Action plans have been developed and work is ongoing to implement the plan)
	2. Insurance will develop a comprehensive action plan to improve the accuracy and consistency of Quality Monitoring (QM), ensure policies and procedures are applied consistently across the country, ensure feedback is provided consistently to agents and that corrective actions are followed up on.
Audit Criterion 2.4.1: Claimant files are kept in a secure area and access is restricted to employees who need access to these files. Recommendation 3: A risk assessment should be conducted in the various EI points of service to ensure that appropriate control is exercised to prevent or detect unauthorized access to and withdrawal of client files.	The Insurance Branch takes very seriously the need to ensure appropriate controls are in place to prevent or detect unauthorized access to client files. We are taking steps to safeguard information by conducting a Process Mapping and Risk Assessment workshop in the Fall, 2003 to identify risk and develop mitigating strategies. Based on the outcomes of the workshop, senior management will identify issues that must be addressed, formulate mitigating strategies and develop a comprehensive plan to address the identified issues.	March 2005	Insurance Branch
Audit Criterion 2.4.2: Workstations linked to benefit programs and EI databases are protected from unauthorized access or use. Recommendation 4: The use in common of workstations by a group of employees should be discontinued.	The Insurance Branch agrees with this recommendation and to reinforce the importance of this, the Regional Executive Heads will be asked to remind their managers of the policy concerning the usage of usercode/passwords and to ensure that this practice is discontinued.	Completed	Insurance Branch
	A review of the current policies, procedures and monitoring process will be undertaken to ensure the appropriate use of usercodes/passwords.	March 2005
* Updated September 2004

APPENDIX C : ANALYSIS OF NQMI/SD FILES

IARMS extracted a random sample of 1,606 files that had already been revised under the NQMI/SD between April and August 2002.

A group of six IPA were seconded to IARMS from October to December 2002 to carry out the quality assurance, or "second review", of the selected applications. The second reviewers examined the files selected at random from Regions other than their own to determine whether the first examination was complete and accurate and whether the initial results were correctly reported.

The second reviewers used the same tools that were used in the initial review, i.e. the twelve quality assurance objectives included in the National Claim File Quality Monitoring Guide developed by National Headquarters. The twelve objectives are presented in Appendix D. The second reviewers were required to follow to a strict protocol. First, they examined each file as if it were the first review, without knowing the original results. Then, the second reviewers compared their own results with the results of the first review. When there was a difference, the first reviewer, namely the local IPA, was contacted. The second reviewer informed the first of the differences between the two results and asked if he or she agreed. If the response from the first reviewer was positive, the difference was recorded and the process stopped there. If the first reviewer did not agree with the conclusions of the second, the case was submitted for arbitration to a group of experts at National Headquarters.

IARMS compiled and analyzed the results of the second review. The analysis and compilation was done at the file level. Each file was coded as being "in order" or "not in order" regardless of the number of discrepancies³ detected during the second review. This approach is similar to the one used for the NQMI/SD and allows a comparison of results.

Results of the quality assurance review

In the sample of 1,606, the percentage of files that had been reported to be "in order" in the initial review was 70.30%, while the percentage of files reported to be "in order" after the IARMS quality assurance review was 60.77%, a difference of 9.53%.

This difference does not take into account situations where the file was reported "in order" by the IPA during the initial NQMI/SD review based on regional/local practices but was found not to be "in order" during the second review based on national standards. If these cases are added, the difference between the initial IPA review and the quality assurance results is 14.51%.

The following two tables present the revised results of cases deemed to be "in order" by Region, first without taking regional/local practices into account (Table 1.a), and then taking them into account (Table 1.b). The names of the Regions have been purposely omitted but each Region has received the detailed results. The gap between the results of the initial NQMI/SD review and the IARMS quality assurance review measures the accuracy of the process.

Table C.1a
Percentage of files reported "in order" by the 1^st and 2^nd reviewers — Not considering regional/local practices — Total files: 1,606

Regions	A	B	C	D	E	F	G	H	I	J	National Average
Quality rate 1st review	54.31%	66.41%	78.53%	79.89%	72.73%	69.61%	87.20%	58.29%	78.13%	60.44%	70.30%
Quality rate 2nd review	48.22%	59.54%	68.10%	64.80%	61.16%	64.71%	71.34%	48.57%	71.35%	53.30%	60.77%
Discrepancy 1st 2nd review	6.09%	6.87%	10.43%	15.08%	11.57%	4.90%	15.85%	9.71%	6.77%	7.14%	9.53%

Table C.1b
Percentage of files reported "in order" by the 1^st and 2^nd reviewers — Considering regional/local practices — Total files: 1,606

Regions	A	B	C	D	E	F	G	H	I	J	National Average
Quality rate 1st review	54.31%	66.41%	78.53%	79.89%	72.73%	69.61%	87.20%	58.29%	78.13%	60.44%	70.30%
Quality rate 2nd review	46.70%	54.20%	55.21%	63.13%	53.72%	63.73%	63.41%	47.43%	60.42%	53.30%	55.79%
Discrepancy 1st 2nd review	7.61%	12.21%	23.31%	16.76%	19.01%	5.88%	23.78%	10.86%	17.71%	7.14%	14.51%

In about four out of five cases (see Table 2), the results show that the first and second reviews both had the same conclusion. "Same conclusion" means that both reviewers arrived at the same result when a file was deemed to be "in order" and coded the same deficiency when a file was deemed to be "not in order".

Table C.2
Percentage of files with the same conclusion from both reviews — Total files: 1,606

Regions	A	B	C	D	E	F	G	H	I	J	National Average
Files found with same conclusion in both reviews	85.28%	83.21%	77.91%	80.45%	85.12%	84.31%	82.93%	81.71%	83.33%	80.77%	82.38%
Discrepancy 1st 2nd review	14.72%	16.79%	22.09%	19.55%	14.88%	15.69%	17.07%	18.29%	16.67%	19.23%	17.62%

The following graph summarizes, by objective, the discrepancies between the first and second reviews. The blue section of the column shows the discrepancies when regional/local practices are not taken into account, and the portion, when they are taken into account.

APPENDIX D: SUMMARY OF THE OBJECTIVES USED FOR THE NQMI/SD AND THE QUALITY ASSURANCE REVIEWS

Table D.1
Summary of the Objectives Used for The NQMI/SD and the Quality Assurance Reviews⁴

01	Documents/information necessary to substantiate the initial claim for benefits were/was obtained before the decision to pay benefits was made and all essential forms were complete.
02	Inadequacies and inconsistencies in data received from the claimant, employer and other sources were identified and explained satisfactorily (does not apply to the record of employment (ROE).
03	All items on ROEs requiring clarification were verified.
04	All missing ROEs affecting rate, duration and entitlement to benefit were followed up, questioned and cleared satisfactorily.
05	The effective date of the claim (BPC) was correctly determined.
06	The claimant met the required qualifying conditions at the time of the initial decision.
07	The insurable earnings were correctly calculated.
08	The initial decision, contentious or non-contentious, was adjudicated appropriately. (This objective includes any decision that is part of the initial adjudication.)
09	The data shown on SSA screens was entered correctly.
10	Initial decisions were communicated to the claimant and to other parties, when necessary.
11	Subsequent decisions (including recalculations) were made on all matters on file and were adequately documented and properly actioned.
12	The decision to renew the claim was adequately documented and appropriately adjudicated.