Understanding the rural - urban reading gap

Appendix C: Analytical methodology

Analytical methodology

The analysis of rural-urban differences in student performance was carried out in three phases. The first stage was an examination of a variety of individual, family, school and community characteristics to identify any significant and systematic differences between students in rural and urban schools. The second phase then used a hierarchical (multilevel) regression model to identify the characteristics that best explain the difference between the rural and urban PISA results. Finally, a variety of school characteristics were explored to identify factors which could be further analysed as potential tools for improving rural student performance.

Phase 1

Examination of differences in the school populations of rural and urban schools

In order to answer the question why there are differences between the reading performance of the two populations, the first phase of the analysis was to identify other ways in which the populations differed.

The variables in this analysis came from three sources: 1) student reported variables on individual behaviour and family background from the PISA and YITS student questionnaires, 2) principal reported variables from the PISA school questionnaire, and 3) community level variables from the 1996 Census aggregated for Census Sub-division (CSD) geographic units linked to schools using postal code information.

Given the complex sample of the PISA assessment, group mean characteristics at the student level were estimated using replication methods. The statistics were estimated using 80 different weightings of the sample. The set of replication weights, produced by Westat, was designed to be consistent with PISA sampling (see below). For each statistic, the variance between the 80 different estimates was proportional to the variance of estimation. For further information on the analytical treatment of PISA sampling, see the PISA 2000 Technical Report (Adams and Wu, 2002). These methods were implemented using the software WesVar 4.0 (2001).

At the school level, the comparison of rural and urban schools was complicated by the nature of the school sample. Since schools were the first-stage sampling unit (see below), statistics estimated using the school design weights produced estimates that generalised to the population of schools that enrolled 15-year-olds, but did not generalise to the school environment of 15-year-olds. To illustrate the distinction, consider that a small minority of 15-year-olds repeated a grade and were attending lower secondary schools. Similarly, a small minority of 15-year-olds were attending schools with extremely small populations. These schools would be considered equal in estimating average school characteristics to larger, upper secondary schools that are more typical for 15-year-olds. Using the school design weights to estimate average school characteristics would produce average school characteristics that were actually unrepresentative of the school environment experienced by the average 15-year-old.

In order to remedy this inconsistency, school weights were constructed using school aggregations of student weights. Thus, statistics produced using these weights are generalizable to the environments experienced by 15-year-olds. Unfortunately, this method does not come with a sample-appropriate statistical method for estimating the precision of statistics. Thus, in order to estimate variances to be used in statistical tests, the aggregated weights were normalised across the sample of schools (divided by a constant, such that the sum of the normalised weights was equal to the number of schools sampled). Using normalised weights, variance estimates were calculated under the assumption of simple random sampling of schools. However, given that this method did not take into account any sample design complexities, it is likely that the variance estimates were an underestimation of true variances¹³. Therefore, the test level of significance was changed from the typical 5% error rate to a 1% error rate. One expects that increasing the stringency of the statistical test should offset the possible underestimation of variance.

Having identified many variables that distinguished rural and student populations, several hypotheses were developed about possible causes for the group differences across a wide variety of characteristics, particularly in our variable of interest, reading performance. For example, student career expectations, parental occupational status and education, and community-level factors all showed consistent differences in favour of urban communities. However, other variables, such as school characteristics and student attitudes (within-community variables) were inconsistent in terms of group differences. The systematic difference in community characteristics between rural and urban schools led to an analysis of community level variables in order to explain the systematic differences at the individual level.

Phase 2

Analysis of impact of individual and community level variables on rural student performance (Model Set 1)

In order to test the effects of community level variables on group-averaged individual performance, it was necessary to produce a complex model that could account for some of the complexity of the system, while still allowing us the statistical power to estimate relationships between the variables.

The goal in this phase of the analysis was to predict the average outcome value for one group (focus group — rural school populations) if they had the average characteristics of another group (reference group — urban school populations). In order to accomplish this, all explanatory variables used in the model were centred on the average value of the reference group (so the average of the transformed values is equal to the mean of the reference group). This results in a much more interpretable model, since the intercept term of the regression equation now refers to the predicted average of the population if the average of the predictor(s) for the population was the reference group average. Another interpretation is that the intercept term is the predicted value for all cases that have the predictor value equal to the average of the reference group. In the case of this analysis, the intercept would be the mean for the urban school population.

However, it was important in this analysis to examine rural-urban differences within provincial jurisdictions since education systems are governed provincially and it is more meaningful to compare rural school populations to urban school populations within provinces. This means that the model needed to account for provincial differences without explaining provincial differences. In order to accomplish this, all explanatory variables were transformed so that they did not reflect differences between provinces. This required that, for each province, all explanatory variables were adjusted within-province. Since the objective of this analysis was to predict outcomes for students with average individual and community characteristics of the reference groups, variables were adjusted around the mean of the reference group for each province. As a result, each predictor variable used in this set of models was of the form:

where
x^*_jk is the final, transformed value of variable x for case j in province k that was used in the regression models

x_jk is the raw, untransformed value of the variable for case j in province k

x_uk represents the average value of the variable for the urban cases in province k

Because the predictors had been centred around the urban mean for each province, but no similar adjustment had been made to the outcome variable (reading performance), it was necessary to include dichotomous variables that indicated province, as well as an indicator variable indicating if a school was rural within each province.

Figure 6
An illustration of provincial differences in predictor and outcome, with a constant relationship
Figure 6 An illustration of provincial differences in predictor and outcome, with a constant relationship

The justification for these adjustments is that many of the community characteristics used to describe differences in rural and urban performance vary between provinces. Unfortunately, the educational policies that determine outcomes, perhaps to a greater degree than community conditions, are also systematically different between provinces. It is possible that a province with systematically higher socio-economic conditions will have systematically different educational policies. This confound is illustrated in Figure 1, which presents 3 hypothetical provinces, each with systematically different locations in terms of both predictor and outcome. The traditional scatterplots are represented here as ovals. The relationship between predictor and outcome is the same in each province, indicated by the parallel lines bisecting each oval. However, since the provinces are systematically different in terms of these variables, the observed effect if all the provinces were considered simultaneously would be much different (shown here as the solid line). However, this overestimated relationship would be an artefact of systematic differences between provinces, instead of the actual relationship between variables.

By centring the predictor within-province, the effect of provincial membership is negated (see Figure 2). Since the average value of the predictor is now identical for each province, the distributions have been aligned horizontally. The distributions remain displaced vertically, which has the effect of attenuating the observed relationship, shown here by the flatter solid line. In order to account for this displacement, it is necessary to include variables that account for provincial membership. Since the distributions are already aligned according to the predictor, the variable accounting for provincial membership only describes the differences between provincial means. The resulting distributions, with both within-province centring and provincial indicators, are shown in Figure 3. The distributions have been effectively overlain on top of each other, and observed relationship is now equal to the actual relationship within each province.

Figure 7
The effect of centring variables within-province
Figure 7 The effect of centring variables within-province

Figure 8
The combined effect of centring and use of provincial indicators
Figure 8 The combined effect of centring and use of provincial indicators

For every model, there were a total of 9 province variables (10-1) and 10 rural-urban variables (1 for each province). The provincial indicators were dummy-coded (for an explanation of dummy coding, see Cohen and Cohen, 1983, pp. 183-220) against the reference group of Alberta, and the rural-urban indicators were dummy coded against the urban group within each province. This meant that the intercept term for the regression equation represented the average performance of urban students in Alberta. The regression coefficient for each indicator variable represented an adjustment from the urban Alberta average. The averages of students in urban school in other provinces would be adjusted by the value of the provincial indicator coefficient, while averages of students in rural school would be adjusted by both the provincial indicator coefficient and the rural-within-province indicator coefficient. These adjustments to the model allowed us to account for the differences between provinces without explaining them. For example, since the difference between average Alberta urban performance and average Newfoundland and Labrador urban performance is perfectly accounted for by the variable indicating which schools are in Newfoundland and Labrador, we can account for the interprovincial variation in performance without explicitly defining why it exists.

If we represent the vector of average performance within each school as B₀, the vector of provincial indicators as P and the vector of within-province rural indicators as R, the basic model that accounts for the differences between rural and urban groups in the different provinces is:

where G₀₀ is the average performance of students in urban Alberta schools, G₀₁ is the vector of differences between the urban performance in Alberta and the other provinces, G₀₂ is the vectors of values describing the difference between rural and urban performance in each province, and U is a vector of school residuals from their predicted provincial-geographic group means. The summation term across G_0l represents the combined effects of any community characteristics, S1_l, used in the model. However, in order to account for the wide variation within each school in terms of performance, the model above (2) was combined with a student level model, Equation (3), that describes individual students' performance, A, as a sum of the school average performance, B₀, and within-school residuals for each student, E. The resulting model obtained by substituting (2) into (3), accounts for both differences between school averages and between individual students within each school (4).

Another consideration in the analysis of macro level variables on groups of individuals is that there is a risk of confounding group characteristics with the aggregate effect of individual characteristics. For example, a group characteristic may have a relationship to the average group outcome, but this relationship may simply be the aggregate effect of correlated characteristics of individuals within the group. In other words, if average income seems to be related to average performance, it is important to be certain that this is not simply because individual income is related to individual performance. In order to control for this situation, all models used in this analysis controlled for the socio-economic conditions of individuals. Again, these variables were centred within-province around the average of the reference group. This required an adjustment to Equation (2), so that the jth element of B₀ represents the predicted average performance of students in school j if all the students in the school had families with average urban characteristics. The new individual-level model is represented as:

where the X_p represents a matrix of individual variables describing individual family socio-economic characteristics, centred on the urban average, and B_p is a matrix of the regression coefficients of these variables onto individual performance, estimated within each school. The two individual level variables used to describe family socioeconomic characteristics were the highest parental occupational status and the highest parental educational attainment. The final model used to describe differences in student performance, combining (2) and (5) is:

The elements in G₀₀, G₀₁, and G₀₂ now describe the predicted average performance of the relevant groups if the average characteristics (S1_l and X_p) of each group were equal to the within-province average urban characteristics. If all S1_l and X_p are empty, which is equivalent to modelling the group differences without any predictor variables, combining the elements in G₀₀, G₀₁, and G₀₂ will produce the observed averages for each group. Since the urban schools in Alberta had the highest average performance of any other of the provincial geographic groups, the elements in G₀₁ will all be negative, describing the deviation of each province's urban average from Alberta's. Furthermore, since the urban averages were higher than rural in all provinces, the elements in G₀₂ will also be negative, describing the deviation of each provincial rural average from the provincial urban average. By increasing the number of predictors, we are attempting to reduce the absolute values in G₀₂ by explaining the differences between rural and urban performance. Since the reference urban groups already have average urban characteristics, elements in G₀₀ and G₀₁ will not change in value. However, if the characteristic, S1_l, has some power in explaining the difference in performance between students in rural versus urban schools, then the values in G₀₂ will become less negative. In general, as the elements of G₀₂ increase in relative value, we are explaining more of the difference between rural and urban performance. If the elements become positive, then it suggests that students in rural schools are performing better than predicted, given the conditions we have modelled. The predicted performance of rural students in all provinces is calculated by adding the corresponding elements in G₀₁P and G₀₂R to G₀₀.

Phase 3

Analysis of the potential impact of school-level variables on rural student performance (Model Set 2)

This last phase of the analysis took into consideration that the community context of schools is not easily changed. However, given that schools do exist within specific context(s), it is useful to identify school variables which were not reported at the highest levels and which were associated with high student achievement. This model set looked specifically at the predicted performance of rural students, given the particular characteristics of rural communities and their effects noted in Model Set 1.

When talking about the effects of school conditions, there is a risk that there are a variety of confounding variables. In particular, because the population for PISA is 15-year-olds and is not school grade specific, many schools in the PISA sample were early-secondary or middle schools where the 15-year-olds were likely students who had been held back a grade at some point in their academic history. Thus, the average of these schools is expected to be lower, simply because of the systematic differences in their sample of students. In order to control for this spurious relationship, a variable describing each student's school grade was introduced. The model appeared as follows:

where GRADE describes the grade of a student, a discrete whole number variable, centred on 10. Thus, the elements in G₀₀, G₀₁, and G₀₂ ¹⁴ now describe the predicted performance of grade 10 students in each provincial geographic region. S2_m and B_p represent vectors of a school socio-economic characteristics and individual socioeconomic characteristics, respectively, centred on the rural average for each province. This model set includes all predictors identified in Model Set 1. The G₀₀ coefficient describes the predicted average performance of rural grade 10 students in Alberta. Since G₀₂ indicates rural or urban location of each school, the summation term across all S2_m and B_p vectors does not affect the predicted provincial rural averages. These variables are included in order to reduce the chances of observing spurious relationships between performance and the variables tested in this second model set.

Using this base model, several school policy variables were introduced in order to determine their effect on student performance, while controlling for the important community variables identified in Model Set 1. In order to identify important variables, the variables tested in the model were centred on their maximum values (in other words, the variables described the difference between the actual value and the desirable value):

where
x^* is the final, transformed value of variable x for case j that was used in the regression models

x is the raw, untransformed value of the variable for case j

x_maximum represents the maximum value of the variable

For certain variables, such as the number of 15-year-olds in a school, the relationship appeared to be non-linear. For variables with non-linear relationships, the following transformation was performed:

This 2^nd order term was combined with the 1^st order term in order to estimate the non-linear effect of the variables.

The following table presents the values used as maximum for each variable. These values were determined by examining the bivariate relationships and scatterplots of each variable with school average performance. For linear relationships, the maximum value was fixed as the minimum or maximum observed value for a variable, depending on whether the bivariate relationship between the variable and performance was negative or positive, respectively. For non-linear relationships, maximum values were specified based on literature, if available, and observed maxima in scatterplots.


Variable	Maximum	Value
Teacher support	Maximum observed score	1.61
Disciplinary climate	Minimum observed score	-1.54
School activities	Maximum proportion of students reporting that school offers activities	1.00
Achievement press	Maximum observed score	1.38
Professional development	Maximum % of teachers with recent professional development	100%
Instructional hours	Minimum observed total hours of instruction	100
Student-teacher ratio	25 students per teacher	25
Teacher behaviour problems	Minimum observed score	-2.41
Student behaviour problems	Minimum observed score	-2.61
Teacher morale	Maximum observed score	1.78
School autonomy	Maximum observed score	1.72
Teacher participation	Maximum observed score	3.70
Teacher specialisation	Maximum proportion of teachers teaching in their specialisation	1.00
Number of 15-year-olds	Various values between 100 and 400 15-year-olds were tested	100-400
Standardised testing	Using standardised tests either less than or more than 2 times per year	na

Thus, the elements of G₀₀, G₀₁, and G₀₂ in the full model sum to produce the predicted average performance for rural students in each province if all schools were to have maximum values on the predictors included, while controlling for socio-economic characteristics. For example, if the Alberta rural average for proportion of teachers specialising in their instructional content area were 80%, the model predicts what average rural performance would be if that proportion were 100%. This final model is defined as

where S3_n represents the difference between a school variable and its maximum value. This model was used to identify variables which were not reported at the highest levels and, b) predict a significant change in student performance if they were given maximum values.

Constraints on this analysis

The first consideration in modelling these data was that the issue of estimating standard deviations from a complex sample had not yet been resolved. As a result, the traditional interpretation of effect sizes, which rely on accurate estimation of population standard deviations, to interpret the relative relationships between variables, could not be done. Thus, the analysis was constrained to fitting models and predicting values according to the fitted models, rather than reporting and interpreting effect sizes. An unexpected benefit of this method is that the results are much more communicable to a lay audience, since predicted averages are closer to common experience than are standard deviations and regression coefficients.

All coefficients in Phases 2 and 3 were estimated using hierarchical ordinary least squares (OLS) estimation for both within-school and between-school effects. The models were replicated across the 5 plausible values describing the posterior density function for each individual's reading performance, and the results of the five analyses were averaged to produce the final reported estimates. Within school (individual) effects were allowed to vary randomly between schools. The software used for estimation of coefficients was HLM 5 (Raudenbush, S., Bryk, T., & Congdon, R, 2000). Although this software also produces optimal estimates using Bayesian estimation for individual level effects, vector B_i, a consequence of this technique is that school averages shrink towards the grand average (see Chapter 3, Bryk and Raudenbush, 1992). As a result, the predicted means for provinces would no longer be equivalent to the observed means, which have already been published. The potential cost of using the OLS estimates is instability in the estimation of predicted school averages, given the individual level model. Analysis of the distribution of OLS predicted averages indicates a greater variability in predicted means than was observed in the actual data. This variability disappears when within-school effects are fixed, rather than left to vary. However, the predicted provincial/ geographic averages (the only reported statistics from these sets of models) are stable whether the within school effects are treated as fixed or not. Thus, the tradeoff between consistency and optimisation was decided in favour of consistency, and the OLS estimates were used rather than the Bayesian estimates. If individual-level effects varied significantly between schools according to a chi-squared test of significance, the effects were left to vary; otherwise, they were treated as fixed. All statistical tests in these models were at the 0.05 level.

Other algorithms for modelling these effects were considered, in particular, the disaggregation of school and community characteristics onto individual students. Aside from the violation of the assumption of independence, given the clustering of students in schools, the results produced using this method were relatively unstable, in that the urban averages, which should remain constant across models given the data transformations described above, varied noticeably.

Both model sets rely on several assumptions, particularly that the variables being measured (or their second-order transformations) have a linear relationship with performance. This is a basic assumption of regression, and the tenability of it was examined through scatterplots. It was also assumed that the slope of these relationships did not differ between provinces (see explanation of Model Set 1, above). This assumption provides the analysis with greater power to estimate school effects. Although there is no reason to believe that effects should differ between provinces, excepting random sample-dependent fluctuations, this assumption was untested in this analysis.

A major limitation of this analysis is the limited number and descriptive power of the variables available. It is possible that other variables describing socio-economic conditions or social capital would produce a clearer picture of community effects. Likewise, many aspects of school environments, such a principal leadership, school climate, and community-school interaction, were unavailable for this analysis. As more data are released as part of the ongoing YITS/PISA projects, better indicators of school and community variables will become available, enriching this type of analysis.

¹³These considerations were weighed against the fact that the sampling fraction of the number of schools in many provinces approached a census. Although sampling fraction was not considered here for consistency with previous PISA analyses, adjustment for the sampling fraction of schools would have reduced the variance estimates for many jurisdictions to be near zero.
¹⁴Although it did not change the properties of the model, the elements of G02 were reversed, such that the intercept G00 refers to the average predicted grade 10 performance in rural Alberta. This adjustment simplified the interpretation of the final model.

Contents

Understanding the rural - urban reading gap - November 2002