Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

Based on our review of the literature, interviews with key informants currently and formerly involved in social indicators, and consideration among ourselves, we suggest three options regarding the use of social indicators as evaluation instruments. Each option is briefly illustrated with two example programs which will be of immediate concern to HRDC's program evaluation groups: the new measures against child poverty, and the CHST. Following a description of the options, we examine each along several dimensions.

5.1 Option 1: Composite Indices

The simplest and least expensive option for use of social indicators in evaluation would be to concentrate on composite indices. Tracking changes in these indices would provide contextual measurement of changes in social conditions. However under this approach, there would be no facility to attribute changes in the indices to changes in social expenditures. Still, it would be able to capitalize on inter-provincial and regional variation in social expenditures compared to changes in social conditions. Further, using a small set of evaluation-validated indices would have the advantage of tying in with work being done elsewhere in HRDC and Statistics Canada, easily understood by the public, and relatively straightforward to monitor.

Such indices would have to be fairly well accepted and relatively robust, and ensure that these can be tracked at both the national and provincial levels. Further, to ensure that these indicators would be useful from an evaluation and monitoring perspective (e.g. sensitive to change in social policy), it would be critical for program evaluation staff to be directly involved in their development.

For the example of the child poverty measures (new Child Tax Benefit and corresponding provincial programs), many separate indices exist which could be developed into composite indices. The report, The Progress of Canada's Children (CCSD 1996a), for example, has already done much work in this direction, using data from the National Longitudinal Study of Children and Youth (NLSCY). The development of composite indices would have to ensure, however, that provincial and in some cases regional data were available on all of their components.

For evaluation and monitoring of the CHST, this option would require developing composite indicators for the health, post-secondary education components, and social welfare. In the health domain, Statistics Canada is currently in the process of developing utility of health status measures. But these would have to be critically examined, as population health status is not yet strongly related to health system inputs. Still, the POHEM effort appears to be making progress in this respect. In the area of education, Human Resources Development Canada is, as noted, currently in the process of developing pan-Canadian indicators. For social welfare, we have the impression that more remains to be done. Composite indices would need to include indicators of access (eligibility), individual level outcomes, and aggregate outcomes.

5.2 Option 2: Social Benchmarks Plus

A more sophisticated and costly approach to the one just discussed is what we are calling "social benchmark plus." This approach, modelled after the "Oregon Shines" project involves two major components. We are proposing which is suggested by the Oregon Shines effort. In the first component, goals, outputs and inputs would be identified, based on some conceptual model linking program inputs to social outcomes. These indicators would be used to measure the level and changes in social conditions. In the second component, data would be gathered and expert opinion and estimation would be used to confirm the links and suggest ways of attributing outcomes to social expenditures.

To elaborate, the first component would start with an abstract conceptual model of how inputs contribute to outputs in the areas of interest covered by HRDC's share of the CHST: post-secondary education (PSE) and social welfare⁸. Consultation with concerned stakeholders would then be used both to set standards and objectives and to "operationalize" the different aspects of the conceptual model on the basic of social outcome and input indicators. Groups to be consulted would include Delphi experts, officials from all levels and relevant areas of government including program evaluation, service deliverers, and the informed public at large including potential program clients. The indicators would have to be selected with certain attributes in mind: data availability, the capability to be disaggregated by province and by a number of demographic (equity) characteristics, and simplicity and public acceptance. Much in the manner described above in the first option, these indicators would then be tracked over the historical period and over the first year under the new CHST regime. The purpose would be to observe contextual trends in the areas of health, education and social welfare and to note any major changes that may have occurred in the last year.

What would be suitable indicators for evaluating CHST and monitoring changes wrought by the implementation of it? Bearing in mind that we are suggesting that indicators be derived on the basis of a broad-based consensual process, we provide a tentative list of indicators that would reflect the three areas of responsibility under the CHST and that could be tracked at the provincial level and by equity groups. For health, preliminary indicators would include: life expectancy, child mortality rate, disability rate, health benefit coverage, perhaps qualitative perceptions of health, as well as per-capita input measures such as health expenditures, hospital beds, and medical personnel. For education, we would suggest: the incidence of post-secondary education degree or certificate (attainment), literacy rate, some measure of educational achievement, employment rate of PSE graduates, a measure of the relationship between education and the job, and such input measures as expenditures, schools, and teachers per capita. As for social welfare, a less precise concept, possible indicators would include the Gini coefficient (a measure of income inequality), the median income and the proportion within X per cent of the median, the poverty rate (e.g., the proportion below the low-income cutoff or within X per cent of the median), the unemployment rate, and the proportion on income support, in addition to welfare expenditures per capita by level of government.

In the example of the Child Tax Credit and the child poverty issue, some consensual processes have already been undertaken to identify goals, and could be extended to the identification of benchmarks (e.g., Health Canada's Turning Points: National Goals for Healthy Youth and Child Development). Experts and representatives of the public, governments and NGOs could all be consulted. This work could be enriched both by provincial efforts (e.g., British Columbia's program, Québec's goals for children's social adaptation contained in the 1992 Health and Social Services Policy) and the considerable existing research using social modelling for child outcomes. The goal of this consensus-building phase would be to develop a broad consensus on the underlying causal model of determinants, exogenous factors, moderating variables and outcomes of interest in relation to child poverty.

The second component of the approach is concerned with corroborating and enhancing the conceptual model on which the consensus-seeking exercise was based. This could be carried out at two levels of sophistication. The most sophisticated would be the construction of a comprehensive microsimulation causal model capturing how various factors including government expenditures contribute to PSE, welfare and, possibly, health outcomes. With these, we would be able to track the impacts of changes in expenditures and types of programs on mediating variables and social indicators (dependent variables) over time and among jurisdictions. Reporting of these indicators could describe not only changes in them, then, but also the main reasons (i.e, based on empirically-derived structural equation coefficients) for the changes. The goal would be to attribute changes in various social indicators to expenditures at various levels of government.

However, this would obviously be long-term, costly exercise involving the gathering of longitudinal data for which it would take some time to amass a long enough time series to carry out the estimation and test the equations⁹. A less ambitious, "second best" approach would be simply to test a series of econometric equations¹⁰ in the relevant areas, much as we demonstrated above in the literature review in Chapter 3. This too would take time but not as much as the full microsimulation approach.

Along the way, a variety of measures would be used to gather data to be used to carry out the confirmation process. Expert opinion would be used to suggest possible directions for subsequent research and modelling with regard to the impacts of the inputs on the outcomes. Case studies of a handful of "representative" communities could be employed to corroborate the links between inputs and outcomes. Finally, a longitudinal or panel dataset would have to be considered to properly estimate and test the model. At present, there are a number of such datasets that would be able to track changes in areas relevant to the CHST, including the Follow-up Graduate Survey, the National Longitudinal Survey of Children and Youth, the School Leavers' Survey, the National Population Health Model, and the Survey of Labour Income Dynamics. However, while these could be used as input into the estimation process¹¹, not any single dataset could serve the multiple purposes that would be needed to monitor the impacts of CHST.

5.3 Option 3: Social Accounting

The third option we are recommending concerns strengthening our ability to account for social phenomena using an account-based approach. This could take place at three levels of intensity: full-blown Social Accounting Matrices, social satellite accounts, and building a social dimension into one aspect of the national accounts.

The application of SAMs to evaluating the CHST and the child tax credit would involve extending the causal model described in the previous option to include all sectors of interventions and intervening variables affecting individuals (health, education, welfare, justice, environment, etc.) and preferably translating their inputs into a common numeraire. Ideally, determinants (social expenditures), exogenous factors, moderating variables and outcomes would also have to be translated into the common numeraire. The utility of this approach would depend on the capacity to reach a consensus on the identification and quantification of all sources of economic and social activity that impinge upon Canadians. Using a structural modelling approach, variables could be retained in their original metric. Independent and interaction effects would be represented by terms (coefficients) in the structural model, where the system of dependent variables would be represented by changes over time in key social indicators related to poverty, child poverty, health, and education achievement.

The complexity of this process (see Chapter 3, section (d)) means that the construction of full SAMs or even social satellite accounts would take considerable time, effort, expertise and money. For this reason, we are suggesting the Evaluation and Data Development pursue the more modest approach of incorporating a social dimension into one component of the National Accounts the input-output tables or the income and expenditure accounts¹². This would be done for both the federal and provincial accounts with a view to being able to attribute changes in outcome indicators to changes in provincial expenditures. At present, efforts are underway at Statistics Canada to build environment satellite accounts, which, although complex, will demonstrate the feasibility of sensitizing input-output tables to non-economic activities.

5.4 Analysis of Options in Terms of Monitoring and Evaluation Requirements

Each of the three options described above has been analyzed in terms of the eleven dimensions listed below. The dimensions reflect important requirements for program evaluation and monitoring. This analysis is designed to reveal the advantages and disadvantages of each option, and to facilitate discussion and refinement of these options and alternatives within HRDC.

The dimensions used to assess the relative strengths and weaknesses of each option are:

1. Potential for attributing effects to different levels of government expenditure, using methods ranging from community case studies to expert studies to surveys to small-area analysis;
2. Possibility for grounding in a explanatory conceptual model permitting microsimulation and causal explanation;
3. Potential to satisfy accountability demands and requirements;
4. Ease of proactive reporting (report cards) to public; public comprehension and acceptability of the indicators; clear bases for recommendations;
5. Potential for integration/sharing with provincial data collection and reporting systems for those provinces where agreements have been concluded and are compatible;
6. Ability to track differential impacts with respect to equity target groups;
7. Ability to track differential impacts at provincial, regional and community levels;
8. Degree of conceptual and operational interconnectedness with HRDC policy research function and other efforts such as the IDRC and the TB initiative;
9. Cost of data required, i.e., necessity to go beyond currently available Statistics Canada data;
10. Amount of resources required (intellectual and time); and
11. Implementation feasibility, taking into account the previous variables.

A summary of this analysis is presented in Exhibit 5.1

Exhibit 5.1 Analysis of Options in Terms of Monitoring and Evaluation
	Option 1: Composite Indices	Option 2: Benchmarks Plus	Option 3: Social Accounting*
Attribution potential for federal contribution	Low	Moderate	Moderate
Grounding in explanatory model	Low	High	Moderate
Potential to satisfy accountability demands	Moderate	High	Moderate-High
Ease of comprehensible proactive reporting	High	High	Moderate
Potential for integration with provincial data systems	High	Moderate	Moderate
Potential to track impacts on equity target groups	Moderate	Moderate	Low
Potential to track impacts at provincial and community level	Moderate	Moderate	Moderate-High**
Interconnectedness with policy research	High	Moderate	Moderate
Cost of data	Low	Moderate-High	Low
Amount of intellectual resources and time required	Moderate	High	High
Implementation feasibility	High	Low-Moderate	Moderate

Footnotes

8	Consideration should be given to including health in this exercise given the demonstrated links between health and PSE and social welfare. It should be pointed out that another possible starting point for social benchmark selection could be the challenges to Canada identified in the research agenda for the Interdepartmental Policy Research Committee (IPRC) (growth, human development, social cohesion). It also should be pointed out that working toward a national consensus on social issues such as social welfare may be difficult, given major ideological cleavages within Canadian society and between the public and decision-makers (Ekos Research Associates 1997). Still, a consultative process could use the key principles contained or implicit in the CHST (for example, compliance with the principles of the Canada Health Act) to identify some goals or benchmarks on which there would be wide, if not complete, agreement.
9	The power to establish incrementality may increase with time, as more data become available over longer historical periods.
10	Interrupted time series analysis could be applied assess the impact on of specific interventions such as the change in regime under CHST. Specifically, techniques such as ARIMA, transfer functions and Box-Jenkins could be used to model the evolution of the indicators while detecting and correcting for possible autocorrelation due to existence of little year-to-year variation in the inputs.
11	Other Statistics Canada datasets that could be used as input include FAMEX, the Survey of Consumer Finances, the Labour Force Survey, surveys of educational institutions (enrolment), and some of the new surveys on hunger and wealth that are underway. These would be complemented by federal administrative data and provincial data on post-secondary education accessibility and attainment, welfare and labour market program participation, health system utilisation and health status.
12	The main orientation of the latter is towards the "institutions" of individuals, corporations, and all levels of government, which appears to be closer to the needs of the evaluation group.