|
|
     |
||
 |
     |
 |
|
To explore these questions further, a series of key informant interviews were conducted with individuals who were either currently working with social indicators or other related approaches, or who had been closely associated with the previous social indicators movement. A summary of each group of interviews is provided below. Names and affiliations of those interviewed are listed in Appendix A, while Appendix B contains the interview guides. 4.1 Current Approaches to Social Indicators Respondents were asked to describe their current work in the areas of social indicators, social accounting or social measurement. The results of this enquiry are listed below grouped by area of activity. It should be pointed out that most, if not all, of the work listed has been mentioned above in the literature review. (a) Activities in Government Agencies The Interprovincial Trade Flows Group, Input-Output Division, System of National Accounts, Statistics Canada, has been involved in work on two indices, the composite Index of Social Health and the Genuine Progress Indicator (GPI) (examples of our second type of social indicator), designed to alter the GDP to better reflect other than purely social aspects of the economy, as discussed above. (Elsewhere in Statistics Canada, work is underway to construct environment-account satellites to the National Accounts, as described above). As well, within input-output division, a permanent group is being created to consider ways of incorporating a "social" dimension into provincial and national accounts. This latter work can be classified as belonging to our fourth category, matrix-based social accounting approaches. In a reactive role, Housing, Family and Social Statistics, Statistics Canada, is providing data and expertise for several other initiatives. These include: supplying Treasury Board with a large number of social indicators for its annual report; contributing to the Secure and Confident Society Subcommittee of the Government-Wide Performance Indicators Project, described below; contributing to the Social Cohesion subcommittee of the Interdepartmental Research Committee; developing gender-based indicators for the Status of Women Council; and developing urban-based measures for the Canadian Federation of Municipalities. We would classify this work as largely falling into our first category, that of compendia of social statistics. The Family and Community Support Systems Division, Analytical Studies Branch, National Accounts, Statistics Canada, also in a fairly reactive role, is involved in projects such as developing a new series of indicators of gender equality in income, earnings, education and training, for the Federal/Provincial/Territorial Committee of Ministers Responsible for the Status of Women. This work makes use of several microdata files including the Total Work Accounts master file, and could be classified as belonging to the social modelling category. The Health Analysis and Modelling (HAM) Group, Social and Economic Studies Division, Analytical Studies Branch, Statistics Canada, is working on the Health Status Index (HSI) or Health Utility Index (HUI), a joint project with McMaster University. This is a composite index of functional health which combines health status and health utility measures, and can be used with morbidity and mortality rates to create the Health Adjusted Life Expectancy (HALE), a variant of the QUALY (Quality-Adjusted Life Years). This measure is a product of a more comprehensive Population and Health Model (POHEM), which would be an example of social modelling in one domain. The Socio-Economic Modelling Group (SEM), Social and Economic Studies Division, Analytical Studies Branch, Statistics Canada is working on two modelling and microsimulation efforts, described above: LifePaths and the Social Policy Simulation Database and Model (SPSD/M). The latter is a simulation tool for estimating the impacts on disposable income of changes in tax or transfer policies. The Lifepaths model takes an approach based on time use over the lifespan, and may eventually include tracking of time spent in social assistance or other social programs. Informants in some other departments were, however, critical of this approach, stating that it has too much emphasis on functionality/productivity and not enough on well-being (subjective utilities). In response to this complaint, the model will soon incorporate qualitative impressions from persons reporting how satisfied they are in various life activities. Staff of Income Security and Social Development Studies, Applied Research Branch, Strategic Policy, Human Resources Development Canada, are involved in and sponsoring a number of initiatives of relevance to social indicators, including: the Index of Social Health, described above; the Canadian Federation of Municipalities efforts to assess the quality of life at the community level; an inventory of analytical and data holdings relevant to the Interdepartmental Research Committee on Social Cohesion; Social Project Inventory for the Standing Inter-Governmental Council on Social Policy; various new surveys including the Wealth Survey, Hunger Survey, Survey of the Voluntary Sector; the Longitudinal Survey of Children and Youth; the longitudinal School Leavers' Survey; Objective Measures of Insecurity; and Lifepaths, described above. At this point, these efforts do not appear to be coordinated within overall plan or to be mounted toward a particular end other than as informative research exercises. At best, they may be used to monitor change in various areas of social concern. Treasury Board Secretariat is currently coordinating an interdepartmental effort called the Government-Wide Performance Indicators Study. This involves developing a set of cross-cutting indicators felt to be important or useful for policy direction, for creating better links between programs and across departments (making departments more sensitive to government-wide concerns, getting rid of the stovepipe approach), and communicating more clearly with Canadians. The main force behind this initiative is improving accountability within government and to the public, i.e., a response to eroded public trust. In this sense, this approach resonates well within the rationale for report cards on government performance discussed above. The indicators are being developed from the government-wide performance framework. This has caused the work to be divided among four interdepartmental committees: (1) Social (Secure and Confident Nation); (2) Economic (Sound and Prosperous Economy); (3) Environment (Safe and Healthy Environment); and (4) Governance (Responsive Government). Each committee is identifying key goals and then reviewing indicators which may be related to those goals. This will result in a report, due October 1997, which will state why the selected indicators are important, and outline what data are currently available to support their use. If there are holes in the data, they will then begin to seek to make these data available. Statistics Canada in not the only source (public opinion polls such as Rethinking Government were mentioned as other sources), but Statistics Canada will be their guide in the data exercise. This effort is voluntary, in the sense that it has not been mandated by Cabinet. The themes of the Treasury Board effort resemble the themes being examined by the Interdepartmental Policy Research Committee, co-chaired by representatives from HRDC and Health Canada. These themes are derived from an analysis of the major challenges to Canada in the next century, and are: growth, human development, and social cohesion. The Health Promotion and Population Health Branch, Health Canada, is becoming very interested in social indicators approaches. The department's recent adoption of a population-health approach has created the need to assess health determinants in the population, many of which are found outside the health sphere (notably, employment, income, and education). The assessment of health determinants is also important to their goal of ensuring that all departments in government contribute to healthy public policy. As well, they are involved in an interdepartmental reference group charged with defining health goals across government; measuring goal attainment will require a set of social indicators in common with other sectors affecting health. This work will be used to assess changes in the health status of the population (monitoring role), and to point to areas where impending declines in health status could be mediated. Despite these major areas of interest in social indicators, they see themselves more as users of indicators rather than developers or providers, partly because the important determinants of health that they would use as indicators are under the direct purview of departments like HRDC. (b) Activities in Non-Government Organizations The Canadian Policy Research Network is involved in four relevant projects:
Behind all these projects is a general influence strategy for changing social policy: by creating awareness and momentum, and by shifting the nature of the prevalent discourse, to exert influence on policy-makers. The CPRN, like the other non-governmental agencies discussed below, is trying to stimulate and shape public interest in social reports. The Canadian Council on Social Development, with the support of the Applied Research Branch of HRDC, held a national symposium on social indicators in late 1996 (CCSD 1996). The conclusion of the symposium recommended that work continue on several fronts, with specific roles for the federal and provincial governments, NGO's and community groups. A need to define the social union and the indicators affecting it was identified, as was a need for better longitudinal data. A specific role for HDRC was mentioned, in establishing frameworks for comparability across sectors and groups. Since the symposium, the CCSD has pursued activities in the area of social indicators. They are currently working on the development of an economic security index, which will consist of a series of indicators combined into an annual composite index. Their report The Progress of Canada's Children (1996a), to be published annually, includes a compendium of indicators on child and youth well-being using data mainly derived from the National Longitudinal Survey of Children and Youth. They are very supportive of the Oregon approach, because of the grassroots involvement in setting benchmarks. They note that its chances of success are high because it responds to needs of both the right (for accountability) and the left (for social policy development), and because it fits well with the new cooperative federalist model. At the same time, they are critical of the CHST as model for governance and pessimistic about its effects on social conditions. The Caledon Institute of Social Policy, while not working on social indicators directly, has released a number of papers relevant to evaluation of the CHST (e.g., Torjman, 1995). Their concern is that with the creation of the CHST and in contrast to the Canada Assistance Program, the fact the provinces are bound only to providing services based on certain principles without possible financial penalty, as opposed to upholding specific conditions or standards, will be detrimental to social conditions. They see it as critical that information, which could be social indicators, continue to be provided in a comparable way across the provinces in the areas covered by the CHST. They also recommend sharing of best practices across programs, so as to provide benchmarks if not standards for service provision. Caledon is leery of Oregon-type approaches where citizens merely vote on social priorities, saying that such approaches will fail to protect the vulnerable or to ensure policy leadership on questions related to social values. They also see a role for community-level monitoring or social audit: given that it will be impossible to track inputs top-down, communities should try to track changes in outputs, bottom-up. They suggest that HRDC monitor such indicators as changes in eligibility requirements for welfare (and resulting exclusions), in order to ensure that there still is a social safety net. (c) Summary: Level of Activity and Need for Coordination It is clear that there is currently a great deal of activity within the federal government and social policy-focused NGOs in the area of social indicators and social accounting. Statistics Canada, with the support of the Applied Research Branch of Human Resources Development Canada, is an important player in many of these efforts, in several roles: development of specific composite indicators such as the Genuine Progress Index and the Index of Social Health, for clients in groups such as Income Security and Social Development Studies and Social Policy groups at HRDC; development in several groups of more comprehensive approaches such as POHEM, Lifepaths, and adding a social dimension to the input-output accounts at the provincial level; and, finally, generation and provision of data and expertise to other groups working to develop systems of indicators, such as the Government-Wide Performance Indicators project led by Treasury Board Secretariat. However, despite this central role, there is little evidence that these efforts are being coordinated or developed within an overall conceptual framework. A related observation common to many key informants was that there is a need to develop an integrated set of indicators that has consensual backing across departments. It was pointed out that there are strong interlinkages among the three main areas of social concern under the CHST, education, welfare, and health. Therefore, it would be counterproductive to develop indicators in just one area. At the same time, the difficulty in developing one overarching social model covering all areas was acknowledged. 4.2 Relation to Previous Social Indicators Movements Overall, the key informant interviews suggested that current efforts have only weak links to the past social indicators movement. Rather, the new movement has emerged rather independently and is driven by much more pragmatic concerns, i.e., measuring and reporting on government performance and the outcomes of social investment. Some respondents contrasted current approaches with the past by pointing out that the latter were too idealistic and detached from social policy. This is consistent with our previous observation that there is currently strong public feeling that an appropriate role for the federal government is to demonstrate accountability for the measured results and effectiveness of its operations. As one key informant pointed out, this generalized interest in outcomes will alleviate one of the problems of the earlier movement, that of failing to get the information into the mainstream. Citizens have a better understanding now of the connections between the economic and the social spheres and expect to be provided with information relating inputs to social outcomes. Non-government or arm's length social policy organizations are also fuelling these expectations. Thus, a key lesson learned from the past approaches, according to these key informants, is that social measurement for the sake of social measurement is not a useful approach to improving social policy. To be useful instruments in the hands of either the public or policy-makers, social indicators must be tied to key areas of government activity and performance. This assessment was echoed by several of the key informants who had been active in the previous social indicators movement. These individuals suggested that one reason for the movement's demise was the lack of modelling to link outputs to relevant inputs, to be monitored from a policy perspective. 4.3 Potential for Social Indicators as Evaluation Instruments A striking communality of opinion across the key informant interviews conducted with those currently involved with social indicators approaches was the lack of systematic linkage to program evaluation and monitoring. None of those interviewed is explicitly developing social indicators so that they can be used for evaluation. While this work is, very broadly speaking, evaluative in that it aims to facilitate judgments about the progress of Canadian society or government, it does not deal with the specific problems faced in program evaluation and monitoring requirements. Chief among these is the "attribution problem". Thus, current work in social indicators within the federal government will not directly provide HRDC with tools for using social indicators approaches to evaluation and monitoring. A second common evaluation-related theme across the key informant interviews, at least among those who were knowledgeable about program evaluation mandates, was the need for an overarching conceptual model. In order for social indicators approaches to be useful for evaluation and monitoring purposes, interviewees felt that the indicators must be derived from an overarching performance model (in Treasury Board terms) or a clear vision of the social union which links values, principles and objectives to outcomes (in CPRN terms). In other words, to be useful for evaluation, social indicators must measure aspects of social changes that have been specified in a model of government actions, including all levels of government where this is relevant, in the context of exogenous social forces. While these key informants felt that this was not an impossible task, it would require several years of groundwork before meaningful indicators could be developed. It was suggested that developing this performance framework was the major intellectual and empirical challenge facing HRDC in its desire to use social indicators as instruments for evaluation and monitoring. A third evaluation theme emerging from the interviews, although not unanimously, was that the demand for and interest in social indicators will continue to grow over the next few years, as medium to long term impacts of cuts in social spending begin to become more evident. Increasing perceptions of disparity and hardship in society will be accompanied by a voracious public appetite for social monitoring instruments, which will in turn create and sustain the political will that has been lacking to create a system of social performance measures. Those agencies that have been preparing social indicators and monitoring instruments in advance of this future swell of interest will be providing information deemed timely and important. A final issue in the use of social indicators in evaluation and monitoring is that of the lack of reference criteria (goals, objectives, norms, standards) against which performance can be judged. As printed out above, proper evaluation of a course of a social program requires some kind of explicit referent to compare program performance against. The fact that the implementation of CHST was not accompanied by any social standards or objectives makes its evaluation difficult. Indeed, the very evaluability of the CHST is at question. At any rate, it is worthwhile to note that two of the current social indicators efforts are dealing with this problem directly at a very broad-based level. The Government-Wide Performance Indicators project coordinated by Treasury Board Secretariat is first developing a consensus on goals for government, and then developing indicators relevant to those goals. The consensus on goals is largely being derived from those within government, although some use is made of opinion polling to have public input. The Canadian Policy Research Network is dealing with the lack of referents using a much more populist approach. Under the CPRN approach, which is in some ways probably similar to that of the Oregon program, citizens define their values and goals for their society. These are then to be translated into indicators which can be used to bring pressure on government to effect policy change. Thus, another lesson learned from previous attempts to use social indicators that is now being applied is a greater recognition of the need to base indicator development on a set of goals, norms or standards. (a) Utility of Social Indicators for the Evaluation of Block Transfer Programs In terms more specifically of the utility of social indicators approaches for evaluating block transfer programs such as the CHST, respondents were very cautious. Three particular issues emerged from the interviews, regarding data availability, purview, and attribution. First, at the level of data, it was generally agreed that some data are already available that would be relevant to evaluating and monitoring programs such as the CHST. These data reside mainly at Statistics Canada, but could be supplemented by public opinion polling and by provincial data (such as provincial health surveys). One respondent felt strongly that much more information could be gleaned from administrative databases, especially for provincial programs (health, social assistance, education) if investment were made in appropriate file-linkage techniques. There were two particular data-level concerns. One was that the time lag for the availability of Statistics Canada data would be longer than ideal for evaluation and monitoring data to be most useful. Superimposed on this practical difficulty is the time lag between social interventions and their outcomes. For example, a measure such as a Child Tax Benefit, aimed at reducing child poverty in the medium term, would be expected to have outcomes, in terms of educational and occupational attainment, in 10 to 20 years. For several informants, this suggested the need for longitudinal tracking systems, and, for some, reinforced the need to have overarching performance models in place that specify proxy and mediating variables available in existing information systems. Social indicators would then closely monitor short-term outcomes, which would have to be demonstrated to be empirically predictive of the desired long-term social outcomes. Such approaches will require social modelling using structural equation techniques. The other data-level sentiment expressed by the key informants was the need to track expenditures through provincial accounting systems, especially in social accounting approaches to social indicators. Several key informants agreed that, to be meaningful, evaluation and monitoring efforts for programs such as the CHST must be able to model both federal and provincial inputs. The second concern that was raised about social indicators approaches in evaluating programs such as the CHST was that of purview. It is generally recognized, especially by those working from a population-health perspective in Health Canada and Statistics Canada, as well as by the NGOs, that the determinants of the main social outcomes of interest to HRDC, and government generally, cross-cut departmental jurisdictions and levels of government. Government interventions in social systems constitute but one set of forces for changes, interacting with many other social, economic and demographic trends. Moreover, as one key informant remarked, there is probably a tendency to overestimate the impact of government programs. This suggested to several informants that the set of indicators to be developed should be representative of all the major determinants and be based on a consensus at least across the federal government and with the provinces. These indicators could then be used to monitor changes in social outcomes as changes are made in social expenditures in the respective jurisdictions. The implication of this for HRDC is that development of social indicators to monitor and evaluate programs like the CHST should be a collaborative effort, involving other relevant federal and provincial departments. The final major concern raised about social indicators as evaluation instruments for programs like the CHST was that of attribution. There was unanimous agreement among key informants that attributing social outcome to particular federal expenditures would be extremely difficult. This is not only because of the difficulty in tracking federal funds as they are transformed into provincial and community programs, but also because in some areas, such as health, the links between expenditures and outcomes (i.e., health system spending and population health) have never been clearly established. Added to this is the problem of lags between expenditures and outcomes cited above, with the possibility that countervailing or facilitating social and economic trends will intervene in the period between expenditure and outcome. No key informant had solutions for this problem, and some were pessimistic about the possibility of linking outcome measures to inputs. One respondent also suggested that using social indicators for accountability purposes would be dangerous, in part because of the limited potential for formulating and testing causal hypotheses and making attributions. (b) Social Standards There was a clear split among government and non-government respondents to our key informant survey regarding the question of national social standards. National social standards may be defined as nationally consistent benchmarks by which to judge the adequacy of public programs to deliver benefits leading to a certain minimum level of living in the area addressed by the respective program (Torjman and Battle 1995). Many of those in government, especially those at Statistics Canada, seem to see little connection between social indicators and social standards, and had little comment on this issue. The question of standards, which is very central to public interest in social indicators, is largely absent from their thinking. Also, respondents in Treasury Board Secretariat felt that, should accountability and transparency increase through efforts like theirs, there would be less perceived need for standards. Our sample of non-governmental respondents, in contrast, were very concerned about the lack of explicit social standards. All three agencies are to some degree unhappy with the CHST because it weakens the foundation for national social standards, and all are acting to fill an information gap they perceive as being caused by the new transfer model. It was predicted by some, but not all, that public interest in information showing the state of social standards in the coming post-CAP years will grow. One respondent who had been involved in the previous social indicators movement suggested that a barrier to the use of social indicators for monitoring indicators of or related to social standards was the lack of political will in government to create instruments that could make them look bad. The political will can arise only through public acceptance of social measures which, in turn, would come about if concerted research and modelling, carried out in arm's length organizations, were able to empirically demonstrate what policy relevant inputs contribute to desirable and agreed-upon social outcomes. To date, Canada has not put an effort commensurate with economic modelling on social modelling. (c) Other Areas of Social Programming Where Meaningful Indicators Could be Developed It was generally agreed that labour/employment and education were two areas where meaningful and social indicators could be easily developed. For labour and employment, this is in part because some economic indicators such as the unemployment rate already exist in these areas, and social indicators could easily complement them. In education, the availability of national standardized testing data will facilitate indicator development, which is currently taking place within the Applied Research Branch at HRDC. Several respondents suggested that it would be useful but more difficult to develop indicators of the impacts of the new measures against child poverty. These measures would have to look at not only economic benefits (e.g., change in absolute level and proportion of income going to children in poor families) but also social (e.g., school readiness at kindergarten age; educational attainment) outcomes. Finally, it was suggested by two of the NGO agencies that HRDC make a serious attempt to develop indicators relevant to the CHST, given the information gap that they are strongly concerned about. (d) Social Indicators and Equity Groups Over the entire sample of respondents, there was a strong consensus that using social indicators to track effects on equity groups was both desirable and possible. The main constraint in nearly everyone's mind was availability of the necessary data on equity group membership in the primary data sources. For some equity groups, such as women, the data are usually available and pose no measurement problem. For others, such as aboriginals or persons with disabilities, both availability and measurement will be problematic.
| |||||||
![[Previous Page]](/web/20060126231109im_/http://www11.hrdc-drhc.gc.ca/edd-img/back.gif){kind=small}
![[Table of Contents]](/web/20060126231109im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif)
![[Next Page]](/web/20060126231109im_/http://www11.hrdc-drhc.gc.ca/edd-img/next.gif){kind=small}