Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

Note #1: Sources for International Data:

1. The Tax/Benefit Position of Production Workers 1991-1994, Organization for Economic Co-operation and Development (OECD).
2. Social Security Programs Throughout the World - 1993, U.S. Department of Health and Human Services, Social Security Administration, Office of Research and Statistics, May 1994.

3. International Benefits Information Service, Reference Manual - Germany, Section III: Social Security, (Chicago: Charles D. Spencer & Associates, Inc.), 1995.

Note #2: Questions Used in the 1991-95 Comparison: In order to compare extent of disability between CPPD beneficiaries surveyed in HALS and those surveyed in the 1995 Beneficiaries Survey (BS), a new variable indicating extent of disability was created for each survey by identifying exactly matched questions in both surveys. Sixteen matched questions were selected. Each question was assigned a code of 1 and thus, total score was 16. These 16 questions were as follows:

1. Have any difficulty in hearing what is said in a group conversation (Q18 in BS, A2 in HALS);
2. Completely unable to do this (Q18a in BS, A2b in HALS);
3. Have any difficulty in seeing newsprint (Q19 in BS, A4 in HALS);
4. Completely unable to do this (Q19a in BS, A4b in HALS);
5. Have any difficulty in speaking or being understood (Q20 in BS, A7 in HALS);
6. Completely unable to do this (Q20a in BS, A7b, combined, in HALS);
7. Have any difficulty walking 400 yards/400 meters (Q21 in BS, A8 in HALS);
8. Completely unable to do this (Q21a in BS, A8b in HALS);
9. Have any difficulty walking down/up a flight of stairs (Q22 in BS, A9 in HALS);
10. Completely unable to do this (Q22a in BS, A9b in HALS);
11. Have any difficulty bending down and picking up an object (Q23 in BS, A13 in HALS);
12. Completely unable to do this (Q23a in BS, A13b in HALS);
13. Limited in kind or amount of activity due to physical health problem: at home (Q24a in BS, A20I in HALS);
14. Limited in kind or amount of activity due to physical health problem: other activities (Q24b in BS; A20IV in HALS);
15. Limited in kind or amount of activity due to mental health problem: at home (Q25a in BS, A25I in HALS); and
16. Limited in kind or amount of activity due to mental health problem: other activities (Q25b in BS, A25IV in HALS).

    It is noted that some categories of information overlap, e.g., such questions as 3- 4, 5-6, 13-16, and that some health and other limitations are more severe than others depending on the different occupations of the disabled. Therefore this global indicator is an imperfect proxy for the degree of disability experienced. The sixteen question are an expansion of the 10 HALS questions, e.g., such questions as 3 and 4, 5 and 6, etc., in this survey were combined in HALS.

Note #3: Eligibility of the Female Population:
The fact that historical differences in contributory requirements at QPPD have resulted in a lower coverage of the female population in Quebec is also evidenced by other data. This is suggested by 1993 statistics on new beneficiaries, where a far larger proportion of new QPPD beneficiaries are males (see exhibit below), as compared to CPPD. Most of this difference can certainly be attributed to the specific contributory requirements of each program noted above, which have had a particularly adverse effect on the eligibility of women in Quebec with shorter or irregular employment history.

Note #4: HALS Analysis: Limitations: While far from a "perfect" data source for examining these complex issues, particularly because of the limited QPPD sample, the HALS nonetheless provides a useful laboratory for examining QPPD/CPPD differences as they existed in 1991. This is done by simulating certain experimental "program" situations that help to understand the programs -- "quasi-experimental"¹⁷¹ assessments of key issues.

It must be noted that the HALS sample is imperfect for this purpose in several ways: while the CPPD sample in HALS is very large (over 3,000 beneficiaries), the QPPD sample is extremely small (just under 200 cases receiving QPPD only, and only just under 300, when Quebec Workers' Compensation is included). Yet no other data base appears to exist to allow such a direct comparison of persons who receive both types of pension benefits, and indeed, the numbers of cases examined generally exceed what might be practical with a "live" joint file review. Other limitations also affect the analysis. For example, incidence per se cannot be studied, because HALS did not include a complete work history and related indicators of potential eligibility for CPPD or QPPD (years of contributions). As well, self-reporting of key data such as limitations (disabilities) or causes of disability provides potential for bias (but that is probably similar between the two programs). As well, it must be emphasized that the HALS data allow broad tests of disability hypotheses, but not a direct simulation of the specific adjudication criteria applied by the two programs.

Reliability: A test of the reliability of discriminant classifications was conducted using a split sample for the CPPD sub-populations. The analysis indicated that good predictions could be made between split half samples, with 80% accuracy in predictions.

Note #5:
The 1991 and 1995 surveys were compared to determine if CPPD recipients were more or less likely to receive disability benefits from other income replacement programs such as WCB, PSA or LTDI. It was anticipated that a higher incidence of simultaneous benefits (from other agencies) would tend to be inconsistent with the hypothesis that CPPD administration of benefits became more generous, on grounds that CPPD's award of benefits would be "validated" if other agencies viewed the same individuals as disabled. (This is, of course, unless other agencies also relaxed their criteria for eligibility, e.g., perhaps so as not to have to justify why they were withholding benefits to persons the CPPD had "certified" as disabled.)

The results indicate that CPPD beneficiaries were more likely to be receiving other disability benefits in 1995 (55%) than in 1991 (49.9%) -- possible evidence against the hypotheses of generous administration or extensive "economic grants". (Note, however, that an alternative explanation of these findings could be that other disability agencies such as WCBs, PSA or LTDI have simply improved the efforts of their clients to claim CPPD, to offset their own costs, but this explanation would also be consistent with the generous administration hypothesis).

Note #6:
According to the 1995 CPPD Beneficiaries Survey, seven percent of the respondents had a mild disability, 35.9% had a moderate disability and 57.1% had a severe disability.¹⁷² Severity of disability was defined as follows: (1) a beneficiary's disability severity was not measurable if he/she did not have any reported difficulty or limitation in the activities listed in the questions; (2) a beneficiary's disability severity was mild if he/she had hearing difficulty (Q.18) and/or seeing difficulty (Q.19), but not completely unable to do these; (3) a beneficiary's disability severity was moderate if he/she had hearing (Q.18), seeing (Q.19) difficulty(ies), speaking (Q.20), walking 400 yards (Q.21), and/or walking up and down stairs (Q.22), but not completely unable to do these; (4) a beneficiary's disability severity was severe if he/she had all aforementioned difficulties and was completely unable to do those, plus having difficulty in bending down (Q.23) (whether completely unable to do this or not), and/or activity limitations at home and outside (Q.24 and Q.25).

Severity level data contained in the 1991 HALS (the 1991 Health and Activities Limitations Survey) has a different distribution of levels of severity: 19.2%, mild, 41.3%, moderate, and 39.5% severe.¹⁷³ According to HALS 1991, therefore the respondents (all CPPD beneficiaries) were more likely to have mild disabilities and much less likely to have severe disabilities than was the case in the 1995 CPPD beneficiaries survey.

Gail Fawcett of the Centre for International Statistics of the Canadian Centre For Social Development compared the 1995 CPPD Beneficiary Survey severity index with that of HALS 1991 and found that the match between the two surveys was good.¹⁷⁴ Fawcett constructed a severity index with HALS data using only the screening questions available to the 1995 CPPD Beneficiaries Survey (10 of the 32 questions used in HALS, and following the methodology used to construct the severity index which appears in the CPPD Survey. The severity index obtained in this manner was then compared with that contained in HALS 1991.

Of those cases that were identified as having a disability under the screening questions common to both surveys, 80% had an exact match between the severity level using the 1991 HALS index and the severity index using the CPPD Beneficiaries Survey. The remaining 20% were only one level off. Fawcett's conclusion was that the severity index contained in HALS 1991 and the 1995 CPPD Survey provide equivalent measures of severity level for the vast majority of cases (80%) and therefore comparisons based on the two data sets are reasonable, other things being equal.

However, Fawcett, in a separate study, found the low response rate (54%) for the 1995 CPPD Survey, and the much greater potential for 'selection bias', compared to much higher response rates for other surveys ( e.g., 86% for the 1991 HALS Survey) to be a source of concern.¹⁷⁵ This study indicated that while it is impossible with absolute certainty to ascertain which survey (the HALS, 1991 or the CPPD Beneficiaries Survey, 1995) is most likely to be representative of the actual CPPD population, it is most probable that those who did not respond to the 1995 CPPD Beneficiaries Survey (46%) were more likely to have milder disabilities than those who did respond (54%). Thus the "greater severity" finding for the 1995 survey may be at least in part an artifact of the survey's response rate bias.

The 1995 CPPD Beneficiaries Survey targeted respondents who had been made aware that they had been chosen for the survey because they were CPPD recipients. Therefore, it was suggested that some of those who did not respond, or who provided information that indicated that they were more severely disabled and less capable of work than they actually were, or refused to allow their information to be used for research purposes, did so out of fear that they would be judged ineligible for future benefits.

The particular concerns noted regarding the 1995 CPPD Beneficiaries Survey are a low survey sample size, possible strong selection bias (that some sub-groups constituted a larger proportion of all respondents than others) coupled with a high non-response rate, (54% compared to 86% for HALS 1991), and a prospect that some variables for the 1995 CPPD survey are not adequate for analysis.

Note #7:
A related study ¹⁷⁶ used taxation data to identify recipients of CPPD benefits based on whether the Disability Tax Credit (DTC) was claimed. The incidence of CPPD benefits was compared to DTC for a three-year period. The comparative data, in the table below, shows the extent to which nearly every person claiming a DTC,¹⁷⁷ also received CPPD benefits, while the reverse is not true.

Note #8: Review of Sample of QPPD Decisions Against QPPD Eligibility Criteria:
A sample of 477 CPPD decisions made in fiscal 1993-94 were reviewed by a team of two QPPD physicians using QPPD adjudication guidelines¹⁷⁸.

Footnotes

171	By a "quasi-experiment" is meant a mathematical or computer simulation of a situation which cannot readily exist in the real world, or in a specific time frame. For example, CPPD applicants cannot be adjudicated by QPPD criteria in the real world, but a simulation or "quasi-experiment" may tell us what the result would look like if they were. Various types of simulations are possible, of which this HALS analysis is one. Another type of simulation would be a "live" file cross-adjudication, with QPPD and CPPD criteria applied by QPPD and CPPD adjudicators each to applicants of the other program.
172	These figures exclude those for which there was not enough information to measure severity level. If these 'missing cases' are included, 6.8 % have a mild disability, 34.8% have a moderate disability, 55.6% have a severe disability, and 2.8 % are missing cases.
173	HALS respondents were asked if they received either CPPD or QPPD benefits. There was no way to separate out CPPD from QPPD directly in this survey. But it is reasonable to assume that the vast majority of those collecting either CPPD or QPPD outside Quebec, were actually collecting CPPD, and the vast majority collecting such benefits in Quebec were actually collecting QPPD.
174	See Gail Fawcett, Report #1, Evaluation of the Severity Index Constructed for the CPP Disability Beneficiaries Survey, 1995, May 1996 (unpublished).
175	See Gail Fawcett, Report #2, Overall Analysis of Data Quality of the CPP Disability Beneficiaries Survey, 1995, May 1996 (unpublished).
176	See: A Labour Market Analysis of CPP Disability Claimants, a working paper for the CPPD Evaluation, Abt Associates, January 1996.
177	The Disability Tax Credit is claimed by those "with a severe mental or physical impairment which markedly restricts basic activities of daily living and which is prolonged, that is, has lasted or is expected to last for a continuous period of at least 12 months". See, Special Income Tax Guide, Revenue Canada, 1995.
178	Commentaires et observations (Étude comparative des dossiers RPC/RRQ du 27 mai au 7 juin 1996) par Roger Dorion et Claude Sarra-Bournet, Régime de rentes du Québec.
179	"Grants 60+": grants to persons 60 years and over; the QPPD physicians used the QPP criteria related to whether the individual could do his job rather than any job.
180	"A More research" : these were files where the QPPD physicians could not make a decision without obtaining additional information.