|
|
     |
||
 |
     |
 |
|
The Use of
Final Report
Prepared for: Strategic Evaluation and Monitoring
August 1998
Prepared by: Ekos Research Associates Inc.
SP-AH064E-08-98
Social indicators, which have had a long history in Canada and elsewhere, have recently attracted a lot of attention as potential monitors of social change. The purpose of this report is to propose options as to how social indicators may be used by the evaluation group of Human Resources Development Canada (HRDC) to measure the impacts of such block federal-provincial funding arrangements as the Canada Health and Social Transfer (CHST). Being able to evaluate such initiatives as the CHST is both important and difficult. Evaluation is important in light of the demands increasingly expressed by Canadians for accountability in government programming. Evaluation is made difficult by the fact that it has never been easy to disentangle the net contribution of public programs to the economy from the effects of other more influential forces such as economic and demographic change. And attribution is all the more difficult in light of the multiple levels of government — federal, provincial, and municipal — involved in new social funding arrangements. This report contains four chapters in addition to this one. Chapter 2 presents a rationale for increasing efforts to make programs, particularly social programs, more accountable to the public. The third Chapter provides a brief review of past and recent social indicator work, with a focus on defining and typologizing social indicators and identifying their potential evaluative properties. Chapter 4 summarizes interviews that Ekos Research Associates has conducted, as part of this project, with past and current key informants in the area of social indicators. In Chapter 5 of this report presents a number of options, drawn from past and current experiences, on how social indicators, in its different forms, may be used to fulfil HRDC's new accountability responsibilities. The final chapter provides a summary and our recommendations.
A number of different rationales appear to have motivated earlier interest in the social indicator movement, which would apply today. Land (1975) identified three of them: (1) a social change rationale, to improve capacity to measure social conditions and change and to supplement economic indicators; (2) a social reporting rationale, to monitor social progress, presumably towards certain societal goals; and (3) a social policy rationale, to evaluate government programs and to establish social accounts and goals. Land (1975) saw these rationales as inter-related: an improved capacity to measure social change is necessary for effective social reporting, which in turn is necessary for guiding social policy. The prevailing economic climate likely determines which of these rationales is dominant. Earlier interest in social measurement was born in a climate of plentitude. Social indicators were sought to monitor social conditions in the face of great economic gains. This contrasts with the climate out of which current interest in social indicators has arisen, one where scarcity is leading to greater interest in a social policy role for social indicators. As funding for social programs is reduced, concern for the state of the social economy (social conditions) and social progress (social reporting) remains strong, but the public demand for accountability and transparency in the spending of the increasingly scarce social budgets (social policy) is even greater. Interest in social indicators for social policy purposes is reflected in recent Rethinking Government polling identifying a desire among Canadians for greater government accountability. In April 1996, about 75 per cent of Canadians felt fairly strongly that accountability for measured results and effectiveness would improve governance (Ekos 1996a). A similar sentiment was expressed in a October-November poll which indicated that 78 per cent of Canadians felt that an appropriate role for government is to demonstrate accountability for the measured results and effectiveness in its operations (Ekos 1996b). This opinion held across age, income, and education groups and regions, though most strongly among older, higher-income, university-educated Canadians and in western Canada. These results appear to indicate that Canadians are less willing than they have been to support expenditures on social and other programs unless the benefits to the economy and to quality of life can be clearly demonstrated. Interest in the social report function of social indicators is also manifested in recent Ekos polling. For example, a majority of Canadians feel that the federal government should issue national report cards that document progress in key social and economic areas, particularly education and health care (Ekos 1997a). Identified as important elements of these report cards were measuring current conditions and setting measurable targets for the future. The latter requirement matches the very strong support that exists in Canada for national standards in social programs (Ekos 1997b).
The term social "indicators" embraces a wide range of measures of social conditions and change. In the literature, we have distinguished four not necessarily mutually exclusive strands of social-indicator research, with rising degrees of sophistication and complexity: compendia of social statistics aggregated into areas of social concern; composite indices based on a number of variables in one or more specific areas of social concern; social modelling efforts, which attempt to explain certain outcomes in specific social areas in terms of program expenditures and sociodemographic and economic variables; and matrix/account-based approaches that attempt to account for both social and economic transactions in an economy. In this chapter, we outline each of these approaches by reviewing how they have been used in the literature. This section concludes with a summary of the problems with past and existing social indicator efforts. (a) Compendia of Social Statistics The least sophisticated of social indicator approaches is the simple presentation of large numbers of social statistics. Examples of such efforts are, in Canada, a compendium of social statistics published regularly in the Canadian Social Trends quarterly (Statistics Canada 1996) and, internationally, in the annual The Human Development Report (United Nations 1991). (See Bunch 1995 for more illustrations covering a wide range of indicators.) These statistics are normally organized to capture conditions in a number of social areas such as education, health, welfare, environment, justice, and culture. Within each major area, certain sub-areas of concern (e.g., access in health) are specified, and for each of these, the appropriate data on inputs and outcomes are selected. By publishing these statistics on a periodic basis, social changes may be tracked over time. These compendia have a number of positive aspects to them. One is that individual indicators are readily understood by the public at large. Another is the comprehensiveness of these collections. Ordinarily consisting of a large number of indicators in a several areas of the social economy, they provide a complete picture of social conditions. A third advantage is that individual statistics often have the capacity to be disaggregated to reflect the demographic (as to, e.g., sex, age, education) and provincial diversity of society, thus permitting measurement of changes in particular (equity and regional) segments of the population. However, it should be pointed out that the statistics are normally presented in aggregate form or are already focused on a specific population group, e.g., children or women. A fourth advantage is their comparative low cost, as such efforts depend on existing data sources and involve little analytical effort. But out of these advantages arise certain disadvantages. One is that the shear volume of statistics presented prevents discernment of the broad outlines of conditions and trends (ECC 1974). Another problem is arbitrariness in deciding what statistics should be included in the collection. The inclusion or exclusion of certain indicators may reflect values and ideologies that are not made explicit. A third problem is that these collections of indicators rarely include measures of how satisfied individuals are in their activities, i.e., of the subjective value placed on objective conditions. Finally, such efforts are rarely based on an explicit conceptual social framework and modelling nor do they include socially desirable goals to which current conditions may be compared. Thus, from an evaluation perspective, they have limited value. Compendia of social statistics are not the appropriate instruments to monitor progress towards goal attainment nor to understand the contribution various inputs, such as the CHST, make towards measured outcomes. (b) Composite Social Indices A more sophisticated approach to social indicators is the construction of composite social indices based on a number of social indicators. By summarizing a number of measures, such indices overcome the difficulty in detecting trends based on a plethora of singular social statistics. One example of such measures is Statistics Canada's prototype Index of Social Health (ISH), which is an amalgam of fifteen socio-economic indicators in the areas of health, insurance coverage, poverty, welfare, housing, inequality and unemployment (CCSD 1996). This index tracks overall social well-being over time relative to the best year in the last twenty-five. Another index is the United Nation's Human Development Index (HDI) which combines measures in health (longevity), education (attainment and literacy), and income (adjusted for poverty). The HDI can be used to compare trends across countries. Also, because the indices can often be disaggregated by age, sex, education and income, it is possible to measure change in particular segments of the population, and, presumably, in the case of the ISH, in different provinces. While such measures offer a convenient way to observe social trends, they suffer from three major shortcomings. First, there is no agreement on what variables should be included in the composite nor on how the variables should be combined (Wolfson 1996a). Component indicators are often selected arbitrarily, and the values behind the selection criteria are not made open for scrutiny.1 Secondly, there may be a problem of interpretation. Because these measures are composed of several indicators in different areas often with different units of denomination, what does a rising or falling index mean for particular aspects of the social economy (e.g., health), except that overall "well-being" or "human development" is increasing or decreasing? There may be more value in constructing indices focused on a particular area (e.g., the HALE, described below) than in trying to construct global indicators spanning several areas of social concern (e.g., the ISH and HDI). Thirdly, because these outcome-based indices are rarely based on a conceptual framework nor have they been econometrically linked to input (expenditure) variables, their value as evaluation instruments monitoring the impacts of social policy changes appears limited. The Health-Adjusted Life Expectancy (HALE), a prototype composite index of functional health produced by Statistics Canada, overcomes some of these problems (Wolfson 1996a). HALE extends traditional life-expectancy measures by incorporating the results of surveys asking individuals about not just their health but also about what they feel is the impact of various conditions on their health. Thus, HALE's main advantage over other purely quantitative approaches is that it incorporates a qualitative dimension: how persons feel about their health. In addition, HALE is based on a numeraire — years — which is easily understood by the general public. Finally, because this measure is embedded in a multi-stage Population Health Model (POHEM), it is possible to measure the impact of socioeconomic status, lifestyle (e.g., smoking) and disease (e.g., lung cancer) on the quality of a person's health. HALE's main disadvantage, however, is that it covers only one aspect of one area of social concern — functional health — among a large number of important areas. The Genuine Progress Indicator (GPI) is the product of attempts to construct a general measure of social welfare based on the logic of the System of National Accounts (SNA) and one of its summary measures, the Gross Domestic Product (GDP) (Cobb et al. 1995a and 1995b; CCSD 1996). The GPI is a measure of national well-being or sustainable economic welfare, expressed in economic (dollar) terms. Instead of treating the flows of all expenditures within the economy (as positive as the SNA computation of the GDP does), the GPI takes into consideration the negative impact of some transactions on well-being and social and natural capital by subtracting out such costs (Cobb et al., 1995b). The GPI has the advantage of being fairly easily understood single measure of social progress, while being rooted in an input-output-like framework. Among the shortcomings of the GPI is the arbitrariness of the weights used to construct them. Another disadvantage is an inability to capture all factors contributing to social welfare, a problem common to all such efforts. Moreover, as the GPI is an economy-wide aggregate, it is questionable whether or not it can be disaggregated by demographic characteristics, specifically targeted equity groups. (c) Social Modelling Over the years, there has been a number of initiatives to model social outcomes using multivariate methods aiming to establish causal relations. Examples include attempts to model educational attainment (Rowley and Leckie 1977), cultural participation (Ekos 1987), functional health (Wolfson 1994), and child development (Bronfenbrenner 1979). In most of these efforts, policy-relevant variables are introduced as one independent or predictor variable, among many others, contributing to the social outcome in question. The outcome is ordinarily represented by variables which can be termed social indicators. Each of the four studies cited above are briefly described below in order to illustrate these concepts. The first example, the study of educational attainment, was carried out as part of the social indicator project at the Economic Council of Canada (Rowley and Leckie 1977). It was confined to the grade 7 students of one Toronto school board, from which comprehensive data on students and school plant were obtained. A conceptual model was first developed that explained children's achievement in school in terms of a number of student -, school -, and community-level factors. The educational achievement indicators (reading and writing) were based on the results of standardized achievement tests obtained from the school board. Among the explanatory variables entered into the analysis were such policy-relevant variables as school resources, teacher experience, and class size and school density, along with controls for peer effects, student characteristics, and socioeconomic status of the surrounding community. A main finding was that aspects of the school plant (resources) positively affect student achievement. Second, in the study of cultural participation, a set of outcome indicators was chosen to reflect the objectives of the program being evaluated (the Cultural Infrastructure Program) (Ekos 1987). These objectives were to assist, and promote public access and participation in, cultural activities. Like the project described above, a conceptual model was first developed that linked cultural "performance" in the community with cultural resources and several "control" variables. The performance indicators included variables capturing cultural participation, opinion and cultural awareness. Explanatory variables introduced into the analysis were the funds provided under the program being studied and under other cultural programs, along with controls for cultural employment, socioeconomic status variables, employment rate and other effects. The major finding was that cultural funding does enhance participation in and opinion of cultural activities, as was the intention of the program. A unique feature of this study was that the multivariate analysis was conducted at not just the individual level, which is normally the case in evaluations, but at also the community level. There were reasons two for this. One was that the delivery and "consumption" of culture takes place at the level of the community, which makes modelling at this level the natural choice. Furthermore, in another community-level evaluation, a strong correlation was found between expenditures under the program in question and other programs, which was not revealed when measured at the individual level. Second, the use of longitudinal data in causal studies is necessary for proper measurement of the impacts of programs. As longitudinal data are cheaper to accumulate at the community than at the individual level, once again community-level models are the ideal. The third example, health, concerns the Population Health Model (POHEM), which takes a total-system, multi-stage approach to modelling individual's life expectancy (Wolfson 1994). In this model, the penultimate health indicator is, or will be, life expectancy (HALE, described above). Eventually to be introduced as factors contributing to the outcome are health-care and treatment costs; socio-economic variables such as educational attainment, labour-market earnings and participation, and marital status; risk factors such as smoking (lifestyle), cholesterol, blood pressure and obesity; and diseases. (To date, only the lung-cancer module has treatment costs as an explanatory variable.) Finally, the external milieu is another set of factors that could be expected to impinge on health, i.e., the physio-chemical, socio-cultural, and economic environments, along with the health system infrastructure and government programs and regulations indirectly affecting health. A fourth example, in the area of child development, shows how social modelling may be used to as a social policy rationale. Applications of Bronfenbrenner's ecological model of child development, such as Bouchard (1991), have proposed wide-ranging reforms to government intervention in the promotion of health physical, cognitive, social and emotional development, taking into consideration the entire set of micro- and structural factors which have been empirically shown to be risk or causal factors. Key child-development indicators such as level of social competence or school-readiness can thus be tracked in the context of all relevant government programs and NGO (non-government organization) and community intervention and other key determinants (e.g., child poverty; secure attachment). Sociodemographic and economic variables are included as exogenous explanatory variables. These examples have highlighted four main advantages of social modelling or simulation. First, simulation models are ordinarily based on some kind of conceptual or causal model where the link between inputs and the outputs (represented by social indicators) are clearly specified, along with the external control variables affecting these outcomes. Second, because the simulation model is based on a conceptual or theoretical model, coherence and efficiency are lent to data collection and research (Wolfson 1994). Without the existence of an analytical framework, data series compiled run the risk of being hodgepodge and research being unfocused. Third, because government expenditures are included as explanatory variables, it is possible to measure the contribution of programs to final social outcomes. This approach may be seen, then, as possibly filling the role for social indicators envisaged by HRDC, i.e., to be used to evaluate social expenditures such as that under the CHST. Fourth, the simulation models have the further flexibility of posing and rigorously answering "what if" questions such as the implications of a change in welfare expenditures. One main problem with social modelling exercises to date is that, like the HALE described above, they are ordinarily confined to one area of social concern. This is due to the fact there is no underlying social theory linking the various areas (Nissell 1995), which would be required to evaluate a program like the CHST. Corresponding date limitations are also a problem. (d) Matrix-Accounting Based Social Indicators The fourth and most comprehensive social indicator approach is based on input/output accounts. Three are described here: Social Accounting Matrices (SAMs), which are adaptations of Systems of National Accounts (SNA) to capture non-monetary, distributive aspects of the economy with dollars as their unit of denomination; Lifepaths, which is the result of efforts to construct a series of time- and person-based accounts covering activities in the population's lifecourse; and the Social Policy Simulation Database and Model (SPSD/M), which is an effort to account for and model all government transfers and taxes and measure their impact on various personal income measures. Work on social accounting matrices (SAMs) consists of attempts to expand the mainly economic focus of traditional National Accounts (NAs) to include social dimensions such as crime, pollution, health, and access to education. It would involve adapting existing National Accounts to permit measurement of all monetary and non-monetary phenomena for various segments of the household sector and the labour market. The construction of SAMs entails adjusting total consumption (the basis of NAs) for inequality, unpaid work, and transactions that impose a cost on society (e.g., pollution and family breakdown). The emphasis is on the role of people in the economy, reflected in finer disaggregation of the household sector and labour markets. Data on various monetary and non-monetary phenomena are conceptually and empirically linked but remain expressed in different measurement units. The resulting system of economic and social accounting matrices would permit general insight into the state of human development and welfare, while using a systems approach. To date, however, there has not been wide spread development of SAMs. SAMs are the most complex and costly approaches for generating social indicators. Their generation would require developing: (1) consensus among all parties (i.e., many federal departments, equivalent departments in all provincial and territorial governments, appropriate regional and municipal departments) that this exercise would be worthwhile and useful; (2) consensus on a performance model that specifies all social and economic processes impacting on social outcomes of interest; (3) development of indicators sensitive to changes at all levels of disaggregation; (4) development of tracking systems for inputs; and (5) joint monitoring and reporting systems nationally. These components could then be integrated into a complete national social accounting matrix for the areas of interest. Alternatively, they could be treated in complex structural models attempting to relate independent and joint effects of government expenditures on relevant variables. This is obviously a long-term proposition, probably requiring a substantial and long-term investment. A more modest social accounting approach is social accounts satellites, which represent attempts to construct accounts in specialized social areas. One effort has been underway for some time within Statistics Canada to construct environmental account satellites. This involves creating different natural resource and waste output accounts as well environmental protection expenditure accounts. The purpose of this effort is to provide statistical support to the perceived need to look beyond conventional economic growth-oriented policy to a target of sustainable development (Smith 1993). It is not intended to generate "green aggregates" or an "environmentally-adjusted net domestic product". Much time has been spent so far on this effort and a number of issues must still be resolved. Among them is how to monetize the different natural resource accounts. SAMs would have all the advantages of NAs. They are constructed on the basis of a logical, coherent input-output structure suitable to matrix-algebra manipulation and providing a bird's eye view of the (social) economy (World Bank 1993). Also, Pyatt (1991) points out NAs/SAMs can be used not just to define performance measures (such as the GDP), but also to provide a framework or capacity to analyze problems and monitor progress. To the extent that the linkages are valid and comprehensive, SAMs could conceivably be used for formal modelling of social phenomena and for monitoring and forecasting the impact of government policies or external influences on non-monetary variables (World Bank 1993). In addition, there appears to be the capability within SAMs to measure change in various sociodemographic groups and at the regional level. There are a number of problems with SAMs and even social account satellites, however. First, it is difficult, like most social indicator exercises, to account for all factors affecting social welfare, particularly the qualitative dimensions (though SAMs would be an improvement in this respect over existing SNAs). Thus, it is not clear how SAMs would be able to disentangle the impact of social expenditures from the myriad of factors that could affect overall welfare. A second problem is the unit of measurement. For many aspects of the economy, particularly for unpaid work and environmental and social costs, the assignment of a monetary value may be problematic. Similarly, many of the accounts that would have to be linked are not compatible. Finally, it is likely that it would take considerable time and money to build social account satellites let along a full-blown SAM that can comprehensively capture the full complexity of the social economy2. A significantly different matrix-based social-indicator approach is the Statistics Canada product Lifepaths3 (Wolfson 1996b). The objective of Lifepaths is not to come up with summary scalar indicators like the ISH, HALE, GDP or GPI discussed. But rather, it is to develop multi-faceted pictures of individuals' lives in different activities and institutions (school, work, commuting, family, etc.), disaggregated by sociodemographic characteristics; there is also a crude attempt to account for person's time on social programs. The Lifepaths exercise entails creating synthetic panels of individuals by linking several existing Statistics Canada datasets including the Time-Use cycles of the General Social Survey (GSS). Lifepaths has a number of advantages over other approaches discussed here. One is that it permits the linking several areas of human activity based on the common numeraires of persons and hours. Composite indices and SAMs include indicators and concepts not all amenable to aggregation based on a single numeraire (dollars in the case of the National Accounts). Another advantage is that the heterogeneity of the population can be captured through disaggregation. Finally, within the Lifepaths framework, a coherent range of social indicators can be derived and used to monitor social change. Three disadvantages of Lifepaths can be identified, however. First, Lifepaths is data intensive, requiring the linking of vast amounts of datasets. The second basic criticism is that Lifepaths, like most social-indicator approaches, provides merely a quantitative view of the life experience not its quality4. The third problem with Lifepaths from the perspective of the evaluation group at HRDC is its inability to attribute measured outcomes to particular inputs, such as program expenditures, though in fairness that was not the intention of the exercise. Another Statistics Canada accounting product is the Social Policy Simulation Database and Model (Bordt et al. 1990). The SPSD/M is essentially a tax and transfer accounting instrument that permits measurement, through simulation, of the impacts of a change in tax/transfer policy on various measures of income and, distribution, for various socioeconomic groups. Whereas the SPSD/M would be a good instrument to evaluate the impacts of a change in social policy (e.g., the change from Family Allowances to the Child Tax Benefit), it is unable to isolate the contribution of government policy from other factors such as changes in family composition and the labour market. Moreover, it is a pure accounting exercise without a behaviourial dimension. A problem with all the matrix-based approaches, from the point of view of evaluation at HDRC, is that they are very difficult to grasp intuitively and thus may not satisfy public desire for report cards on governance and accountability in programming. A second limitation with matrix-based approaches, again from an evaluation viewpoint, is that they were developed from a strict accounting perspective, that is, to count up as much activity as possible in various spheres. This is fundamentally different from an evaluation perspective, which is not only to count but also to explain social phenomena, in relation to organized and directed inputs. The contribution of these approaches to an evaluation mandate, while possibly important, needs still to be satisfactorily demonstrated. A final problem with the matrix-based approaches described here is the considerable resources that are needed to pursue them. (e) Summary: Common Problems In this section, by way of summary, we discuss a number of common problems experienced by the various social indicator efforts. One problem is that most social indicator efforts are purely quantitative in nature which means they cannot say anything about individuals' valuations and judgments: how they feel about their social conditions. Incorporating persons' valuations into the measures would help to determine whether the observed changes are good or bad. This would also provide information regarding the public's aspirations, possible tradeoffs between areas, and general level of living or satisfaction with, or quality of, life. Noll (1996) notes that persons in objectively good living conditions may not feel good about them (dissonance), while persons in bad living conditions may feel content (adaptation). To some extent, the HALE (and POHEM) has overcome this problem, though it is limited to one specific area functional health. More promising is the planned effort to incorporate qualitative data into Lifepaths, which is a more comprehensive research effort. At the same time, however, such judgments have a strong element of subjectivity, which may affect their utility to (objectively) monitor social change. Another common problem is that few social indicator approaches actually include societal goals, which are a reflection of social values and standards, against which performance may be gauged. Efforts that have explicitly incorporated goals are the State of Oregon's benchmark project, "Oregon Shines" (CCSD 1996) and a similar Alberta effort (Alberta Treasury 1996). A crucial element of this approach is the identification and ranking, through consultation and consensus-building, of state - or province - wide benchmarks in a number of social areas; outcomes and inputs, as well as the links between them, are also determined in this process5. This approach has the advantage of relative low cost and likely wide acceptance. The preceding problem is related to a problem often mentioned in autopsies of the social indicators movement: that the measurement systems were developed by intellectual and policy elites with little concern for public input or scrutiny of content, methods or application. Often, indicator design was not accompanied by institutional arrangements for public input, with the result that the whole exercise failed to capture public interest as a tool for improving social policy (Innes 1989; Innes de Neufville 1975, Chapter 10). In other words, for social indicators to be policy drivers, process must be part of the product (Waddell 1995). While this problem exists in most current approaches, others such as Oregon Shines have dealt with it directly by beginning the search for indicators and benchmarks with development of consensus on goals and priorities. Nissell (1995) points out that the persons consulted should include program clients since these persons often feel that current social programs are not appropriate. A final prevalent problem is that many of the social-indicator efforts in this area, other than the social modelling efforts, lack an overall conceptual framework or theoretical model (Henderson 1974, Knox 1975, Innes 1990, among others). This is related to the problem of the lack of capacity for "what if" modelling. Without such a capacity, we would not have the ability to measure the impacts of changes in social expenditures on the income and welfare outcomes of individuals, controlling for other possible influences, such as demographic and economic change. Moreover, being able to attribute social change to a blunt policy instrument like the CHST is made more challenging by the fact that provinces will likely spend their transfers on social, health and education in different proportions. And this task is rendered all the more difficult by the apparent devolution of welfare and health spending power down to the community level6. For these reasons, social indicators to date have accomplished little more than enlighten the public and policy makers about social conditions7 (Innes 1990). In fact, among the large number of social indicator efforts reviewed by Bunch (1995), few have the capacity to address administrative issues, one area of the Canadian Public Research Networks' basic principles for social policy. More importantly, given the present project, none has addressed the administrative issue of public accountability.
To explore these questions further, a series of key informant interviews were conducted with individuals who were either currently working with social indicators or other related approaches, or who had been closely associated with the previous social indicators movement. A summary of each group of interviews is provided below. Names and affiliations of those interviewed are listed in Appendix A, while Appendix B contains the interview guides. 4.1 Current Approaches to Social Indicators Respondents were asked to describe their current work in the areas of social indicators, social accounting or social measurement. The results of this enquiry are listed below grouped by area of activity. It should be pointed out that most, if not all, of the work listed has been mentioned above in the literature review. (a) Activities in Government Agencies The Interprovincial Trade Flows Group, Input-Output Division, System of National Accounts, Statistics Canada, has been involved in work on two indices, the composite Index of Social Health and the Genuine Progress Indicator (GPI) (examples of our second type of social indicator), designed to alter the GDP to better reflect other than purely social aspects of the economy, as discussed above. (Elsewhere in Statistics Canada, work is underway to construct environment-account satellites to the National Accounts, as described above). As well, within input-output division, a permanent group is being created to consider ways of incorporating a "social" dimension into provincial and national accounts. This latter work can be classified as belonging to our fourth category, matrix-based social accounting approaches. In a reactive role, Housing, Family and Social Statistics, Statistics Canada, is providing data and expertise for several other initiatives. These include: supplying Treasury Board with a large number of social indicators for its annual report; contributing to the Secure and Confident Society Subcommittee of the Government-Wide Performance Indicators Project, described below; contributing to the Social Cohesion subcommittee of the Interdepartmental Research Committee; developing gender-based indicators for the Status of Women Council; and developing urban-based measures for the Canadian Federation of Municipalities. We would classify this work as largely falling into our first category, that of compendia of social statistics. The Family and Community Support Systems Division, Analytical Studies Branch, National Accounts, Statistics Canada, also in a fairly reactive role, is involved in projects such as developing a new series of indicators of gender equality in income, earnings, education and training, for the Federal/Provincial/Territorial Committee of Ministers Responsible for the Status of Women. This work makes use of several microdata files including the Total Work Accounts master file, and could be classified as belonging to the social modelling category. The Health Analysis and Modelling (HAM) Group, Social and Economic Studies Division, Analytical Studies Branch, Statistics Canada, is working on the Health Status Index (HSI) or Health Utility Index (HUI), a joint project with McMaster University. This is a composite index of functional health which combines health status and health utility measures, and can be used with morbidity and mortality rates to create the Health Adjusted Life Expectancy (HALE), a variant of the QUALY (Quality-Adjusted Life Years). This measure is a product of a more comprehensive Population and Health Model (POHEM), which would be an example of social modelling in one domain. The Socio-Economic Modelling Group (SEM), Social and Economic Studies Division, Analytical Studies Branch, Statistics Canada is working on two modelling and microsimulation efforts, described above: LifePaths and the Social Policy Simulation Database and Model (SPSD/M). The latter is a simulation tool for estimating the impacts on disposable income of changes in tax or transfer policies. The Lifepaths model takes an approach based on time use over the lifespan, and may eventually include tracking of time spent in social assistance or other social programs. Informants in some other departments were, however, critical of this approach, stating that it has too much emphasis on functionality/productivity and not enough on well-being (subjective utilities). In response to this complaint, the model will soon incorporate qualitative impressions from persons reporting how satisfied they are in various life activities. Staff of Income Security and Social Development Studies, Applied Research Branch, Strategic Policy, Human Resources Development Canada, are involved in and sponsoring a number of initiatives of relevance to social indicators, including: the Index of Social Health, described above; the Canadian Federation of Municipalities efforts to assess the quality of life at the community level; an inventory of analytical and data holdings relevant to the Interdepartmental Research Committee on Social Cohesion; Social Project Inventory for the Standing Inter-Governmental Council on Social Policy; various new surveys including the Wealth Survey, Hunger Survey, Survey of the Voluntary Sector; the Longitudinal Survey of Children and Youth; the longitudinal School Leavers' Survey; Objective Measures of Insecurity; and Lifepaths, described above. At this point, these efforts do not appear to be coordinated within overall plan or to be mounted toward a particular end other than as informative research exercises. At best, they may be used to monitor change in various areas of social concern. Treasury Board Secretariat is currently coordinating an interdepartmental effort called the Government-Wide Performance Indicators Study. This involves developing a set of cross-cutting indicators felt to be important or useful for policy direction, for creating better links between programs and across departments (making departments more sensitive to government-wide concerns, getting rid of the stovepipe approach), and communicating more clearly with Canadians. The main force behind this initiative is improving accountability within government and to the public, i.e., a response to eroded public trust. In this sense, this approach resonates well within the rationale for report cards on government performance discussed above. The indicators are being developed from the government-wide performance framework. This has caused the work to be divided among four interdepartmental committees: (1) Social (Secure and Confident Nation); (2) Economic (Sound and Prosperous Economy); (3) Environment (Safe and Healthy Environment); and (4) Governance (Responsive Government). Each committee is identifying key goals and then reviewing indicators which may be related to those goals. This will result in a report, due October 1997, which will state why the selected indicators are important, and outline what data are currently available to support their use. If there are holes in the data, they will then begin to seek to make these data available. Statistics Canada in not the only source (public opinion polls such as Rethinking Government were mentioned as other sources), but Statistics Canada will be their guide in the data exercise. This effort is voluntary, in the sense that it has not been mandated by Cabinet. The themes of the Treasury Board effort resemble the themes being examined by the Interdepartmental Policy Research Committee, co-chaired by representatives from HRDC and Health Canada. These themes are derived from an analysis of the major challenges to Canada in the next century, and are: growth, human development, and social cohesion. The Health Promotion and Population Health Branch, Health Canada, is becoming very interested in social indicators approaches. The department's recent adoption of a population-health approach has created the need to assess health determinants in the population, many of which are found outside the health sphere (notably, employment, income, and education). The assessment of health determinants is also important to their goal of ensuring that all departments in government contribute to healthy public policy. As well, they are involved in an interdepartmental reference group charged with defining health goals across government; measuring goal attainment will require a set of social indicators in common with other sectors affecting health. This work will be used to assess changes in the health status of the population (monitoring role), and to point to areas where impending declines in health status could be mediated. Despite these major areas of interest in social indicators, they see themselves more as users of indicators rather than developers or providers, partly because the important determinants of health that they would use as indicators are under the direct purview of departments like HRDC. (b) Activities in Non-Government Organizations The Canadian Policy Research Network is involved in four relevant projects:
Behind all these projects is a general influence strategy for changing social policy: by creating awareness and momentum, and by shifting the nature of the prevalent discourse, to exert influence on policy-makers. The CPRN, like the other non-governmental agencies discussed below, is trying to stimulate and shape public interest in social reports. The Canadian Council on Social Development, with the support of the Applied Research Branch of HRDC, held a national symposium on social indicators in late 1996 (CCSD 1996). The conclusion of the symposium recommended that work continue on several fronts, with specific roles for the federal and provincial governments, NGO's and community groups. A need to define the social union and the indicators affecting it was identified, as was a need for better longitudinal data. A specific role for HDRC was mentioned, in establishing frameworks for comparability across sectors and groups. Since the symposium, the CCSD has pursued activities in the area of social indicators. They are currently working on the development of an economic security index, which will consist of a series of indicators combined into an annual composite index. Their report The Progress of Canada's Children (1996a), to be published annually, includes a compendium of indicators on child and youth well-being using data mainly derived from the National Longitudinal Survey of Children and Youth. They are very supportive of the Oregon approach, because of the grassroots involvement in setting benchmarks. They note that its chances of success are high because it responds to needs of both the right (for accountability) and the left (for social policy development), and because it fits well with the new cooperative federalist model. At the same time, they are critical of the CHST as model for governance and pessimistic about its effects on social conditions. The Caledon Institute of Social Policy, while not working on social indicators directly, has released a number of papers relevant to evaluation of the CHST (e.g., Torjman, 1995). Their concern is that with the creation of the CHST and in contrast to the Canada Assistance Program, the fact the provinces are bound only to providing services based on certain principles without possible financial penalty, as opposed to upholding specific conditions or standards, will be detrimental to social conditions. They see it as critical that information, which could be social indicators, continue to be provided in a comparable way across the provinces in the areas covered by the CHST. They also recommend sharing of best practices across programs, so as to provide benchmarks if not standards for service provision. Caledon is leery of Oregon-type approaches where citizens merely vote on social priorities, saying that such approaches will fail to protect the vulnerable or to ensure policy leadership on questions related to social values. They also see a role for community-level monitoring or social audit: given that it will be impossible to track inputs top-down, communities should try to track changes in outputs, bottom-up. They suggest that HRDC monitor such indicators as changes in eligibility requirements for welfare (and resulting exclusions), in order to ensure that there still is a social safety net. (c) Summary: Level of Activity and Need for Coordination It is clear that there is currently a great deal of activity within the federal government and social policy-focused NGOs in the area of social indicators and social accounting. Statistics Canada, with the support of the Applied Research Branch of Human Resources Development Canada, is an important player in many of these efforts, in several roles: development of specific composite indicators such as the Genuine Progress Index and the Index of Social Health, for clients in groups such as Income Security and Social Development Studies and Social Policy groups at HRDC; development in several groups of more comprehensive approaches such as POHEM, Lifepaths, and adding a social dimension to the input-output accounts at the provincial level; and, finally, generation and provision of data and expertise to other groups working to develop systems of indicators, such as the Government-Wide Performance Indicators project led by Treasury Board Secretariat. However, despite this central role, there is little evidence that these efforts are being coordinated or developed within an overall conceptual framework. A related observation common to many key informants was that there is a need to develop an integrated set of indicators that has consensual backing across departments. It was pointed out that there are strong interlinkages among the three main areas of social concern under the CHST, education, welfare, and health. Therefore, it would be counterproductive to develop indicators in just one area. At the same time, the difficulty in developing one overarching social model covering all areas was acknowledged. 4.2 Relation to Previous Social Indicators Movements Overall, the key informant interviews suggested that current efforts have only weak links to the past social indicators movement. Rather, the new movement has emerged rather independently and is driven by much more pragmatic concerns, i.e., measuring and reporting on government performance and the outcomes of social investment. Some respondents contrasted current approaches with the past by pointing out that the latter were too idealistic and detached from social policy. This is consistent with our previous observation that there is currently strong public feeling that an appropriate role for the federal government is to demonstrate accountability for the measured results and effectiveness of its operations. As one key informant pointed out, this generalized interest in outcomes will alleviate one of the problems of the earlier movement, that of failing to get the information into the mainstream. Citizens have a better understanding now of the connections between the economic and the social spheres and expect to be provided with information relating inputs to social outcomes. Non-government or arm's length social policy organizations are also fuelling these expectations. Thus, a key lesson learned from the past approaches, according to these key informants, is that social measurement for the sake of social measurement is not a useful approach to improving social policy. To be useful instruments in the hands of either the public or policy-makers, social indicators must be tied to key areas of government activity and performance. This assessment was echoed by several of the key informants who had been active in the previous social indicators movement. These individuals suggested that one reason for the movement's demise was the lack of modelling to link outputs to relevant inputs, to be monitored from a policy perspective. 4.3 Potential for Social Indicators as Evaluation Instruments A striking communality of opinion across the key informant interviews conducted with those currently involved with social indicators approaches was the lack of systematic linkage to program evaluation and monitoring. None of those interviewed is explicitly developing social indicators so that they can be used for evaluation. While this work is, very broadly speaking, evaluative in that it aims to facilitate judgments about the progress of Canadian society or government, it does not deal with the specific problems faced in program evaluation and monitoring requirements. Chief among these is the "attribution problem". Thus, current work in social indicators within the federal government will not directly provide HRDC with tools for using social indicators approaches to evaluation and monitoring. A second common evaluation-related theme across the key informant interviews, at least among those who were knowledgeable about program evaluation mandates, was the need for an overarching conceptual model. In order for social indicators approaches to be useful for evaluation and monitoring purposes, interviewees felt that the indicators must be derived from an overarching performance model (in Treasury Board terms) or a clear vision of the social union which links values, principles and objectives to outcomes (in CPRN terms). In other words, to be useful for evaluation, social indicators must measure aspects of social changes that have been specified in a model of government actions, including all levels of government where this is relevant, in the context of exogenous social forces. While these key informants felt that this was not an impossible task, it would require several years of groundwork before meaningful indicators could be developed. It was suggested that developing this performance framework was the major intellectual and empirical challenge facing HRDC in its desire to use social indicators as instruments for evaluation and monitoring. A third evaluation theme emerging from the interviews, although not unanimously, was that the demand for and interest in social indicators will continue to grow over the next few years, as medium to long term impacts of cuts in social spending begin to become more evident. Increasing perceptions of disparity and hardship in society will be accompanied by a voracious public appetite for social monitoring instruments, which will in turn create and sustain the political will that has been lacking to create a system of social performance measures. Those agencies that have been preparing social indicators and monitoring instruments in advance of this future swell of interest will be providing information deemed timely and important. A final issue in the use of social indicators in evaluation and monitoring is that of the lack of reference criteria (goals, objectives, norms, standards) against which performance can be judged. As printed out above, proper evaluation of a course of a social program requires some kind of explicit referent to compare program performance against. The fact that the implementation of CHST was not accompanied by any social standards or objectives makes its evaluation difficult. Indeed, the very evaluability of the CHST is at question. At any rate, it is worthwhile to note that two of the current social indicators efforts are dealing with this problem directly at a very broad-based level. The Government-Wide Performance Indicators project coordinated by Treasury Board Secretariat is first developing a consensus on goals for government, and then developing indicators relevant to those goals. The consensus on goals is largely being derived from those within government, although some use is made of opinion polling to have public input. The Canadian Policy Research Network is dealing with the lack of referents using a much more populist approach. Under the CPRN approach, which is in some ways probably similar to that of the Oregon program, citizens define their values and goals for their society. These are then to be translated into indicators which can be used to bring pressure on government to effect policy change. Thus, another lesson learned from previous attempts to use social indicators that is now being applied is a greater recognition of the need to base indicator development on a set of goals, norms or standards. (a) Utility of Social Indicators for the Evaluation of Block Transfer Programs In terms more specifically of the utility of social indicators approaches for evaluating block transfer programs such as the CHST, respondents were very cautious. Three particular issues emerged from the interviews, regarding data availability, purview, and attribution. First, at the level of data, it was generally agreed that some data are already available that would be relevant to evaluating and monitoring programs such as the CHST. These data reside mainly at Statistics Canada, but could be supplemented by public opinion polling and by provincial data (such as provincial health surveys). One respondent felt strongly that much more information could be gleaned from administrative databases, especially for provincial programs (health, social assistance, education) if investment were made in appropriate file-linkage techniques. There were two particular data-level concerns. One was that the time lag for the availability of Statistics Canada data would be longer than ideal for evaluation and monitoring data to be most useful. Superimposed on this practical difficulty is the time lag between social interventions and their outcomes. For example, a measure such as a Child Tax Benefit, aimed at reducing child poverty in the medium term, would be expected to have outcomes, in terms of educational and occupational attainment, in 10 to 20 years. For several informants, this suggested the need for longitudinal tracking systems, and, for some, reinforced the need to have overarching performance models in place that specify proxy and mediating variables available in existing information systems. Social indicators would then closely monitor short-term outcomes, which would have to be demonstrated to be empirically predictive of the desired long-term social outcomes. Such approaches will require social modelling using structural equation techniques. The other data-level sentiment expressed by the key informants was the need to track expenditures through provincial accounting systems, especially in social accounting approaches to social indicators. Several key informants agreed that, to be meaningful, evaluation and monitoring efforts for programs such as the CHST must be able to model both federal and provincial inputs. The second concern that was raised about social indicators approaches in evaluating programs such as the CHST was that of purview. It is generally recognized, especially by those working from a population-health perspective in Health Canada and Statistics Canada, as well as by the NGOs, that the determinants of the main social outcomes of interest to HRDC, and government generally, cross-cut departmental jurisdictions and levels of government. Government interventions in social systems constitute but one set of forces for changes, interacting with many other social, economic and demographic trends. Moreover, as one key informant remarked, there is probably a tendency to overestimate the impact of government programs. This suggested to several informants that the set of indicators to be developed should be representative of all the major determinants and be based on a consensus at least across the federal government and with the provinces. These indicators could then be used to monitor changes in social outcomes as changes are made in social expenditures in the respective jurisdictions. The implication of this for HRDC is that development of social indicators to monitor and evaluate programs like the CHST should be a collaborative effort, involving other relevant federal and provincial departments. The final major concern raised about social indicators as evaluation instruments for programs like the CHST was that of attribution. There was unanimous agreement among key informants that attributing social outcome to particular federal expenditures would be extremely difficult. This is not only because of the difficulty in tracking federal funds as they are transformed into provincial and community programs, but also because in some areas, such as health, the links between expenditures and outcomes (i.e., health system spending and population health) have never been clearly established. Added to this is the problem of lags between expenditures and outcomes cited above, with the possibility that countervailing or facilitating social and economic trends will intervene in the period between expenditure and outcome. No key informant had solutions for this problem, and some were pessimistic about the possibility of linking outcome measures to inputs. One respondent also suggested that using social indicators for accountability purposes would be dangerous, in part because of the limited potential for formulating and testing causal hypotheses and making attributions. (b) Social Standards There was a clear split among government and non-government respondents to our key informant survey regarding the question of national social standards. National social standards may be defined as nationally consistent benchmarks by which to judge the adequacy of public programs to deliver benefits leading to a certain minimum level of living in the area addressed by the respective program (Torjman and Battle 1995). Many of those in government, especially those at Statistics Canada, seem to see little connection between social indicators and social standards, and had little comment on this issue. The question of standards, which is very central to public interest in social indicators, is largely absent from their thinking. Also, respondents in Treasury Board Secretariat felt that, should accountability and transparency increase through efforts like theirs, there would be less perceived need for standards. Our sample of non-governmental respondents, in contrast, were very concerned about the lack of explicit social standards. All three agencies are to some degree unhappy with the CHST because it weakens the foundation for national social standards, and all are acting to fill an information gap they perceive as being caused by the new transfer model. It was predicted by some, but not all, that public interest in information showing the state of social standards in the coming post-CAP years will grow. One respondent who had been involved in the previous social indicators movement suggested that a barrier to the use of social indicators for monitoring indicators of or related to social standards was the lack of political will in government to create instruments that could make them look bad. The political will can arise only through public acceptance of social measures which, in turn, would come about if concerted research and modelling, carried out in arm's length organizations, were able to empirically demonstrate what policy relevant inputs contribute to desirable and agreed-upon social outcomes. To date, Canada has not put an effort commensurate with economic modelling on social modelling. (c) Other Areas of Social Programming Where Meaningful Indicators Could be Developed It was generally agreed that labour/employment and education were two areas where meaningful and social indicators could be easily developed. For labour and employment, this is in part because some economic indicators such as the unemployment rate already exist in these areas, and social indicators could easily complement them. In education, the availability of national standardized testing data will facilitate indicator development, which is currently taking place within the Applied Research Branch at HRDC. Several respondents suggested that it would be useful but more difficult to develop indicators of the impacts of the new measures against child poverty. These measures would have to look at not only economic benefits (e.g., change in absolute level and proportion of income going to children in poor families) but also social (e.g., school readiness at kindergarten age; educational attainment) outcomes. Finally, it was suggested by two of the NGO agencies that HRDC make a serious attempt to develop indicators relevant to the CHST, given the information gap that they are strongly concerned about. (d) Social Indicators and Equity Groups Over the entire sample of respondents, there was a strong consensus that using social indicators to track effects on equity groups was both desirable and possible. The main constraint in nearly everyone's mind was availability of the necessary data on equity group membership in the primary data sources. For some equity groups, such as women, the data are usually available and pose no measurement problem. For others, such as aboriginals or persons with disabilities, both availability and measurement will be problematic.
Based on our review of the literature, interviews with key informants currently and formerly involved in social indicators, and consideration among ourselves, we suggest three options regarding the use of social indicators as evaluation instruments. Each option is briefly illustrated with two example programs which will be of immediate concern to HRDC's program evaluation groups: the new measures against child poverty, and the CHST. Following a description of the options, we examine each along several dimensions. 5.1 Option 1: Composite Indices The simplest and least expensive option for use of social indicators in evaluation would be to concentrate on composite indices. Tracking changes in these indices would provide contextual measurement of changes in social conditions. However under this approach, there would be no facility to attribute changes in the indices to changes in social expenditures. Still, it would be able to capitalize on inter-provincial and regional variation in social expenditures compared to changes in social conditions. Further, using a small set of evaluation-validated indices would have the advantage of tying in with work being done elsewhere in HRDC and Statistics Canada, easily understood by the public, and relatively straightforward to monitor. Such indices would have to be fairly well accepted and relatively robust, and ensure that these can be tracked at both the national and provincial levels. Further, to ensure that these indicators would be useful from an evaluation and monitoring perspective (e.g. sensitive to change in social policy), it would be critical for program evaluation staff to be directly involved in their development. For the example of the child poverty measures (new Child Tax Benefit and corresponding provincial programs), many separate indices exist which could be developed into composite indices. The report, The Progress of Canada's Children (CCSD 1996a), for example, has already done much work in this direction, using data from the National Longitudinal Study of Children and Youth (NLSCY). The development of composite indices would have to ensure, however, that provincial and in some cases regional data were available on all of their components. For evaluation and monitoring of the CHST, this option would require developing composite indicators for the health, post-secondary education components, and social welfare. In the health domain, Statistics Canada is currently in the process of developing utility of health status measures. But these would have to be critically examined, as population health status is not yet strongly related to health system inputs. Still, the POHEM effort appears to be making progress in this respect. In the area of education, Human Resources Development Canada is, as noted, currently in the process of developing pan-Canadian indicators. For social welfare, we have the impression that more remains to be done. Composite indices would need to include indicators of access (eligibility), individual level outcomes, and aggregate outcomes. 5.2 Option 2: Social Benchmarks Plus A more sophisticated and costly approach to the one just discussed is what we are calling "social benchmark plus." This approach, modelled after the "Oregon Shines" project involves two major components. We are proposing which is suggested by the Oregon Shines effort. In the first component, goals, outputs and inputs would be identified, based on some conceptual model linking program inputs to social outcomes. These indicators would be used to measure the level and changes in social conditions. In the second component, data would be gathered and expert opinion and estimation would be used to confirm the links and suggest ways of attributing outcomes to social expenditures. To elaborate, the first component would start with an abstract conceptual model of how inputs contribute to outputs in the areas of interest covered by HRDC's share of the CHST: post-secondary education (PSE) and social welfare8. Consultation with concerned stakeholders would then be used both to set standards and objectives and to "operationalize" the different aspects of the conceptual model on the basic of social outcome and input indicators. Groups to be consulted would include Delphi experts, officials from all levels and relevant areas of government including program evaluation, service deliverers, and the informed public at large including potential program clients. The indicators would have to be selected with certain attributes in mind: data availability, the capability to be disaggregated by province and by a number of demographic (equity) characteristics, and simplicity and public acceptance. Much in the manner described above in the first option, these indicators would then be tracked over the historical period and over the first year under the new CHST regime. The purpose would be to observe contextual trends in the areas of health, education and social welfare and to note any major changes that may have occurred in the last year. What would be suitable indicators for evaluating CHST and monitoring changes wrought by the implementation of it? Bearing in mind that we are suggesting that indicators be derived on the basis of a broad-based consensual process, we provide a tentative list of indicators that would reflect the three areas of responsibility under the CHST and that could be tracked at the provincial level and by equity groups. For health, preliminary indicators would include: life expectancy, child mortality rate, disability rate, health benefit coverage, perhaps qualitative perceptions of health, as well as per-capita input measures such as health expenditures, hospital beds, and medical personnel. For education, we would suggest: the incidence of post-secondary education degree or certificate (attainment), literacy rate, some measure of educational achievement, employment rate of PSE graduates, a measure of the relationship between education and the job, and such input measures as expenditures, schools, and teachers per capita. As for social welfare, a less precise concept, possible indicators would include the Gini coefficient (a measure of income inequality), the median income and the proportion within X per cent of the median, the poverty rate (e.g., the proportion below the low-income cutoff or within X per cent of the median), the unemployment rate, and the proportion on income support, in addition to welfare expenditures per capita by level of government. In the example of the Child Tax Credit and the child poverty issue, some consensual processes have already been undertaken to identify goals, and could be extended to the identification of benchmarks (e.g., Health Canada's Turning Points: National Goals for Healthy Youth and Child Development). Experts and representatives of the public, governments and NGOs could all be consulted. This work could be enriched both by provincial efforts (e.g., British Columbia's program, Québec's goals for children's social adaptation contained in the 1992 Health and Social Services Policy) and the considerable existing research using social modelling for child outcomes. The goal of this consensus-building phase would be to develop a broad consensus on the underlying causal model of determinants, exogenous factors, moderating variables and outcomes of interest in relation to child poverty. The second component of the approach is concerned with corroborating and enhancing the conceptual model on which the consensus-seeking exercise was based. This could be carried out at two levels of sophistication. The most sophisticated would be the construction of a comprehensive microsimulation causal model capturing how various factors including government expenditures contribute to PSE, welfare and, possibly, health outcomes. With these, we would be able to track the impacts of changes in expenditures and types of programs on mediating variables and social indicators (dependent variables) over time and among jurisdictions. Reporting of these indicators could describe not only changes in them, then, but also the main reasons (i.e, based on empirically-derived structural equation coefficients) for the changes. The goal would be to attribute changes in various social indicators to expenditures at various levels of government. However, this would obviously be long-term, costly exercise involving the gathering of longitudinal data for which it would take some time to amass a long enough time series to carry out the estimation and test the equations9. A less ambitious, "second best" approach would be simply to test a series of econometric equations10 in the relevant areas, much as we demonstrated above in the literature review in Chapter 3. This too would take time but not as much as the full microsimulation approach. Along the way, a variety of measures would be used to gather data to be used to carry out the confirmation process. Expert opinion would be used to suggest possible directions for subsequent research and modelling with regard to the impacts of the inputs on the outcomes. Case studies of a handful of "representative" communities could be employed to corroborate the links between inputs and outcomes. Finally, a longitudinal or panel dataset would have to be considered to properly estimate and test the model. At present, there are a number of such datasets that would be able to track changes in areas relevant to the CHST, including the Follow-up Graduate Survey, the National Longitudinal Survey of Children and Youth, the School Leavers' Survey, the National Population Health Model, and the Survey of Labour Income Dynamics. However, while these could be used as input into the estimation process11, not any single dataset could serve the multiple purposes that would be needed to monitor the impacts of CHST. 5.3 Option 3: Social Accounting The third option we are recommending concerns strengthening our ability to account for social phenomena using an account-based approach. This could take place at three levels of intensity: full-blown Social Accounting Matrices, social satellite accounts, and building a social dimension into one aspect of the national accounts. The application of SAMs to evaluating the CHST and the child tax credit would involve extending the causal model described in the previous option to include all sectors of interventions and intervening variables affecting individuals (health, education, welfare, justice, environment, etc.) and preferably translating their inputs into a common numeraire. Ideally, determinants (social expenditures), exogenous factors, moderating variables and outcomes would also have to be translated into the common numeraire. The utility of this approach would depend on the capacity to reach a consensus on the identification and quantification of all sources of economic and social activity that impinge upon Canadians. Using a structural modelling approach, variables could be retained in their original metric. Independent and interaction effects would be represented by terms (coefficients) in the structural model, where the system of dependent variables would be represented by changes over time in key social indicators related to poverty, child poverty, health, and education achievement. The complexity of this process (see Chapter 3, section (d)) means that the construction of full SAMs or even social satellite accounts would take considerable time, effort, expertise and money. For this reason, we are suggesting the Evaluation and Data Development pursue the more modest approach of incorporating a social dimension into one component of the National Accounts the input-output tables or the income and expenditure accounts12. This would be done for both the federal and provincial accounts with a view to being able to attribute changes in outcome indicators to changes in provincial expenditures. At present, efforts are underway at Statistics Canada to build environment satellite accounts, which, although complex, will demonstrate the feasibility of sensitizing input-output tables to non-economic activities. 5.4 Analysis of Options in Terms of Monitoring and Evaluation Requirements Each of the three options described above has been analyzed in terms of the eleven dimensions listed below. The dimensions reflect important requirements for program evaluation and monitoring. This analysis is designed to reveal the advantages and disadvantages of each option, and to facilitate discussion and refinement of these options and alternatives within HRDC. The dimensions used to assess the relative strengths and weaknesses of each option are:
A summary of this analysis is presented in Exhibit 5.1
* Incorporating a "social" dimension into the Input-Output tables of the national and provincial accounts. ** Low at the community level.
Our analysis of the current use of social indicators approaches and their potential utility to the monitoring and evaluation mandates of HRDC's program evaluation group has concentrated on three broadly drawn options. Although there are many more nuances to be drawn between these three options, our analysis along several dimensions has led us to the following conclusions about each. Before outlining each of these options, two points must be made. First, whatever option or options the Evaluation and Data Development group of HRDC adopts to evaluate the Canada Health and Social Transfer, we would suggest that EDD prepare detailed work plans which would specify objectives and goals, research issues, approaches and data collection methods. This would be carried with due regard for provincial interests under the new "Social Union". Second, we should emphasize the need to evaluate the ramifications of the CHST in all areas covered by the transfer, not just the areas of responsibility under HRDC's purview. In other words, we are suggesting that health be included in this exercise in addition to post-secondary education and social welfare. Research has shown that health has a significant influence on educational achievement and living conditions, and thus must be treated along with the latter two components. The first option, composite social indices, currently being sponsored by the Applied Research Branch, follows most closely along the path of the previous social indicators movement. Such measures would be used to provide social intelligence on changes in various input and output areas such as social expenditures, physical health, and child poverty. The main weakness of social composites relates to a lesson not learned from the previous social indicators movement: there is little or no connection to either an explanatory causal model of the social states in question, or to explicit goals or benchmarks against which progress can be measured. Moreover, and critical to the mandate of HRDC's program evaluation group, this approach does not permit causal attribution of changes in outputs to changes in inputs, from the federal or any other level of government. The strengths of such an approach lie in its relative simplicity and public acceptability, and in the relatively low investment required to produce indicators from a wide array of existing data. Also, composite indicators are compatible with, indeed perhaps indistinguishable from, current efforts in various NGO and government policy research groups. For these reasons, an approach based on the construction of composite indicators and tracking them over time would be relatively easy to implement and provide contextual evidence on changes in social conditions, though with minimal capability for attribution. However, their utility as evaluation instruments could be significantly enhanced if the Evaluation and Data Development group were to be involved in their selection and construction. The second option, "social benchmarks plus", involves using social indicators derived from conceptual models in relation to consensually derived social benchmarks or goals. Under this approach, a set of social benchmarks and the respective inputs are first selected on the basis of consultation with all relevant stakeholders, including representatives of the general public and potential clients of the programs in question, as well as policy makers, and analysts. For the indices to be useful for program evaluation and monitoring, the program evaluation perspective must be represented in the process of selecting and constructing indicators. For purposes of evaluating CHST, these indicators would be in the three areas of concern under the CHST, namely education, welfare, and, ideally, health. The selected indicators would then be used to measure conditions in each of the respective areas of concern and to monitor change therein. And, over a longer period, the conceptual model would be validated on the basis of expert opinion and microsimulation or econometric analysis and supported by data gathering through a longitudinal (panel) survey of beneficiaries and through case studies. A not insignificant disadvantage of this approach is that, compared to the development of composite social indices, it would be relatively more costly in terms of the resources required to develop possible datasets, the models, the benchmarks, and the indicators. Moreover, this approach would be more complex to implement because of the intellectual challenge posed by the nature of the CHST program and the need to incorporate the views of a variety of persons from outside the department and outside government. Finally, it would take time to amass a long enough time series to complete the estimation process, especially if that involved establishing a full causal model. Still, this approach, potentially, might be more likely than the first option to meet the needs of HRDC's program evaluation group. For one thing, this option at least partially fills the void created by the lack of stated objectives under the CHST. For another, it could ultimately permit attribution of changes in social states to changes in government expenditures and programs and enable empirical assessment of Treasury Board's evaluation questions of relevance, success and cost-effectiveness. However, while this option would be likely to satisfy public needs for accountability and report cards, by addressing directly the attribution problem, its implementation will be limited in the short and medium owing to the constraints outlined above. The feasibility of the third option, the social accounting approach, for provincial attribution purposes should be shown through a special feasibility or demonstration project carried out jointly by Statistics Canada and the EDD group at HRDC. The main advantage of the approach is that it could draw on the considerable analytical strength and intelligence (data) existing within Statistics Canada and the international community. Work is currently under way at Statistics Canada to build a social dimension into the Input-Output component of the National and provincial accounts, which would go a long way to being able to attribute changes in social conditions to changes in provincial expenditures in the wake of the CHST. But there is one cautionary note that must be sounded with regard to this approach. Over and above the wide acceptance of national accounting systems within government and the academic community, this approach may have less intuitive appeal for non-economists and may be difficult to present in the form of report cards to the public. Care must be taken, therefore, that this approach be packaged and "sold" to the public in a clearly transparent and user friendly fashion. If this is done and if the feasibility of incorporating social dimensions into the provincial and federal accounts and attribution can be clearly shown through an initial demonstration project, then this approach should be pursued.
Interviews Conducted Hans Messinger, Robert Sauvé Abe Tarasofsky Michael Wolfson, Brian Murphy Leroy Stone Lorna Bailey and Karen Kelly Alan Zeesman Heather Hopwood, Martin Ulrich, Tim Schuurman Judith Maxwell Chris Clark Sherri Torjman Marie-Thérèse Chicha Kathy Stewart Hans Adler David Henderson Alan Maslove
Interview Guide 1
II. Future Utility of Social Indicators
Interview Guide 2
II. Future Utility of Social Indicators
Alberta Treasury, Office of Budget and Management 1996. "Alberta's Perspective and Experience to Date", October. Bordt, Michael, Grant J. Cameron, Stephen F. Gribble, Brian B. Murphy, Geoff T. Rowe, and Michael C. Wolfson 1990. "The Social Policy Simulation Database and Model: An Integrated Tool for Tax/Transfer Policy Analysis", Canadian Tax Journal, Vol. 38, No. 1, January-February. Bouchard, C., et al. 1991. Un Québec fou de ses enfants. Rapport du groupe de travail pour les jeunes, Ministère de la santé des services sociaux, Québec. Bronfenbrenner, U. 1979. The ecology of human development (Cambridge, MA: Harvard University Press). Bunch, Mary 1995. "Social Indicators: Annotations from the Literature," Working Paper F-02, Exploring Canadian Values: Foundations for Well-Being, Canadian Policy Research Networks Inc., Ottawa. Canadian Council on Social Development 1996. Measuring Well-Being: Proceedings from a Symposium on Social Indicators, Toronto, October. Canadian Council on Social Development 1996a. The Progress of Canada's Children. Cobb, Clifford, Ted Halstead and Jonathan Rowe 1995a. "If the GDP is up, why is America down?", The Atlantic Monthly, October. Cobb, Clifford, Ted Halstead and Jonathan Rowe 1995b. The Genuine Progress Indicator: Summary of Data and Methodology. Redefining Progress. Economic Council of Canada 1974. Economic Targets and Social Indicators, Eleventh Annual Review (Ottawa: Information Canada). Ekos Research Associates Inc. 1987. "A Final Report of the Rationale of the Cultural Initiatives Program and a Review of its Effectiveness", for the Department of Communications, Ottawa. Ekos Research Associates Inc. 1996a. "Rethinking Government 1995", Final Report, July 12. Ekos Research Associates Inc. 1996b. "Rethinking Government 96-1", Data Tables, December 9. Ekos Research Associates Inc. 1997a. "Rethinking Government", Presentation. Ekos Research Associates Inc. 1997b. "Rethinking Government III", Draft Report, January 8. Henderson, D.W. 1974. Social Indicators: A Rationale and Research Framework, Economic Council of Canada (Ottawa: Information Canada). Innes de Neufville, J. 1975. Social Indicators and Public Policy: Interactive Processes of Design and Application (New Brunswick, NJ: Transaction). Innes, J. 1989. "Disappointments and legacies of social indicators", Journal of Public Policy, Vol. 9, No. 4, pp. 429-432. Innes, Judith Eleanor 1990. Knowledge and Public Policy: The Search for Meaningful Indicators, Second Expanded Edition (New Brunswick, NJ: Transaction Publishers). Knox 1975. Social Well-Being: A Spatial Perspective. Land, Kenneth C. 1975. "Social Indicator Models: An Overview", Chapter 2 of Social Indicator Models (NY: Russell Sage Foundation). Nissell, Muriel 1995. "Social Trends and Social Change", Journal of the Royal Statistical Society, Vol. 158, Part 3, pp. 494-504. Noll, Heinz-Herbert 1996. "The International Experience: Social Indicators", paper presented at the "Symposium on Measuring Well-Being and Social Indicators", Toronto, October. Policy Research Committee 1996. Growth, Human Development, Social Cohesion: Draft Interim Report, October. Pyatt, Graham 1991. "SAMs, the SNA and National Accounting Capabilities," Review of Income and Wealth, Series 37, No. 2, June, pp. 177-198. Rowley, J.C.R. and N. Leckie 1977. "A further look at the determinants of educational achievement", Canadian Journal of Sociology, Vol. 4, No. 2, pp. 339-353. Smith, Philip 1993. "The Canadian National Accounts Environment Component: A Status Report," in System of National Accounts: National Income and Expenditure Accounts, Annual Estimates 1982-1993, Statistics Canada Cat. No. 13-201, Ottawa. Stone, Leroy O. and Marie-Therese Chicha 1996. The Statistics Canada Total Work Accounts System, Statistics Canada Cat. No. 89-549-XPE, Ottawa, August. Torjman, S. 1997. "Is CAP in Need of Assistance?" in: K. Banting and K. Battle (Eds.), A New Social Vision for Canada? Perspectives on the Federal Discussion Paper on Social Policy Reform, (Ottawa: Caledon Institute of Policy Studies), pp. 99-113. Torjman, S. and Battle, K. 1995. "Can We Have National Standards?" Caledon Institute for Social Policy, May. United Nations 1991. World Development Report (New York: Oxford University Press). Waddell, S. 1995. "Lessons from the Healthy Cities Movement for Social Indicators Development", Social Indicators Research, Vol. 34, No. 2, pp. 213-222. Wolfson, Michael C. 1994. "POHEM a framework for understanding and modelling the health of human populations," World Health Statistical Quarterly, Vol. 47, pp. 157-176. Wolfson, Michael C. 1996a. "Health-Adjusted Life Expectancy", Health Reports (Statistics Canada Cat. No. 82-003-XPB), Vol. 8, No. 1, pp. 41-46. Wolfson, Michael C. 1996b. "Lifepaths: A New Framework for Socio-Economic Accounts," paper presented for the International Association for the Review of Income and Wealth 24th General Conference "Accounting for Time", Lillehammer, Norway, August. World Bank 1993. "Social Accounting Matrices", Chapter 20 of System of National Accounts, Inter-Secretariat Working Group on National Accounts.
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![[Table of Contents]](/web/20060126231111im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif)
![[To Top]](/web/20060126231111im_/http://www11.hrdc-drhc.gc.ca/edd-img/top.gif){kind=small}