Nieuw onderzoek KU Leuven over directie en leerkrachten katholiek onderwijs:
Generatiekloof onder personeel in katholiek onderwijs inzake geloof en identiteit is niet meer!
- De generatiekloof inzake religie en levensbeschouwing in het katholiek onderwijs is verdwenen. De oudere generatie is niet langer de drager van de katholieke traditie. Die generatie is intussen op pensioen.
- De katholieke dialoogschool heeft bij alle leeftijdscategorieën duidelijk de voorkeur als optie voor de identiteit van de school.
- Er speelt bijna geen leeftijdseffect meer bij de ondersteuning van de katholieke dialoogschool. Alleen in de jongste generatie leerkrachten is er concurrentie van de kleurrijke school als tweede, opkomende tendens.
- In alle leeftijdsgroepen vindt men een mix van gelovige, ongelovige of andersgelovige profielen.
- De bodem van de secularisatie is in de katholieke scholen bijna bereikt. Alleen de lijn van de kleurrijke school is nog enigszins aan het hellen.
- Ook in deze context blijft de voorkeur van alle leeftijden gaan naar een geëngageerde levensbeschouwelijke openheid voor de christelijke traditie zolang deze maar niet monolitisch of reconfessionaliserend wordt gepresenteerd/opgedrongen.
- Men kan niet meer rekenen op een oudere generatie die spontaan de traditie present stelt en aan de volgende generaties zal doorgeven. Deze generatie is weg. Het tijdvenster is definitief verschoven. Identiteit is een kwestie van iedereen.
- Met het verdwijnen van de oudere generatie is de interesse voor de christelijke traditie helemaal niet verloren, ook de jongere generaties willen zich engageren, maar met respect en openheid voor diversiteit.
- Om de motor van de vernieuwing van de katholieke identiteit in beweging te houden of te krijgen is goede vorming in en presentatie van de christelijke traditie noodzakelijk. Er zullen hiervoor nieuwe inspanningen moeten gebeuren maar de openheid is er voor een post-kritisch geloofsaanbod.
- Dit alles betekent dat het religieuze veld opnieuw helemaal open komt te liggen, dat alles mogelijk is, ook een nieuwe reveil van de zoektocht naar levensbeschouwelijke vragen vanuit een hedendaags christelijk perspectief, en dat met àlle generaties van het personeel in katholieke scholen.
1. Introduction
The purpose of the present study is to investigate whether age is a significant factor in determining respondents´ scores on the PCB, Melbourne, and Victoria Scales. Data on a total of 6460 subjects were used. These are teachers, administration staff, and school board members (groups 6 and 7 in the ECSIP categorization table) affiliated with different Catholic schools across Flanders.
1.1. Sample Size
The sample was collected between January 2009 and October 2017. Subjects who did not provide information on their age at the time of participation and subsequently were excluded from the analysis. In addition, an expected dropout was observed as the survey progressed. The questionnaires were always presented in the following order: PCB Scale, Melbourne Scale, Victoria Scale. Thus, there is a natural decrease in sample size as we move through the scales in this order. A drilldown table was shown in Table I.2.1. Based on Table I.2.1, the total sample sizes for the three scales as used in the current analysis were as follows:
- PCB Scale – 5460
- Melbourne Scale – 3750
- Victoria Scale – 3870
Table I.2.1. Age descriptive statistics across PCB, Melbourne, and Victoria Scales.
Filter | Number of staff |
All respondents in Belgium | 60866 |
All adults respondents in Belgium | 22597 |
All staff respondents in Belgium (RG=6,7) | 9511 |
Exclude staff respondents with missing age | 8973 |
Exclude CAR, CYM, PAR | 7682 |
Exclude TER | 7072 |
Exclude intensive courses and GO! | 6941 |
Exclude 17,18 years old staff | 6928 |
Exclude PLP2014 | 6918 |
1.2. Independent variable
The age of the respondents at the time of completion of survey was the predictor variable of interest. Even though a slightly different sub-sample was used for each scale's analysis, age distribution remained quite constant, as can be verified by the descriptive statistics given in Table I.2.2 and Figure I.4.1. Further, shape was almost identical across the three sub-samples, showing a bimodal right-skewed curve.
Table II.2.2. Age descriptive statistics across PCB, Melbourne, and Victoria Scales.
scale | mean | standard deviation | min – max values |
PCB | 39.90 | 11.42 | 19 - 89 |
Melbourne | 41.09 | 11.15 | 20 - 88 |
Victoria | 40.77 | 11.21 | 19 - 88 |
Figure I.4.1. Boxplot of age respectively for PCB Scale, Melbourne Scale and Victoria Scale.
1.3. Response
A short description of the three scales treated as a response in the present study follows:
- PCB Scale – a questionnaire consisting of 33 items measuring 4 distinct sub-scales: literal belief, external critique, relativism, and post-critical belief; a single score on each sub-scale is calculated as the average response on all items pertaining to the particular sub-scale, using a 7-point Likert scale (from 'strongly disagree' to 'strongly agree')
- Melbourne Scale – a questionnaire consisting of 25 items measuring 5 distinct sub-scales: Secularisation, Reconfessionalisation, Values Education, Recontextualisation, and Confessionality; these sub-scales, with the exception of Confessionality, are measured on 2 levels – factual and normative, which is reflected in having 2 versions of the same question asked simultaneously; individual sub-scale scores are calculated as the average response on all items pertaining to the particular sub-scale, using a 7-point Likert scale (from 'strongly disagree' to 'strongly agree')
- Victoria Scale – a questionnaire consisting of 20 items measuring 4 distinct sub-scales: Monologue (school), Colourless (school), Colourful (school), and Dialogue (school); all sub-scales are measured on 2 levels – factual and normative, which is reflected in having 2 versions of the same question asked simultaneously; individual sub-scale scores are calculated as the average response on all items pertaining to the particular sub-scale, using a 7-point Likert scale (from 'strongly disagree' to 'strongly agree')
Only the normative level of the Melbourne and Victoria Scales were of interest here. Thus, we work with a total of 12 discrete responses, or dependent variables, corresponding to the 12 sub-scales contained across our 3 questionnaires.
1.4. Statistical methods
A separate analysis was performed on each of the 12 responses as identified above. First, a simple regression model was fitted in order to assess influence of age on the response. Next, the analysis of variance (ANOVA) technique was utilized to compare score means among different age groups. For this purpose, age was categorized into the following 5 groups: 19 to 24 years old, 25 to 35 years old, 36 to 45 years old, 46 to 58 years old, older than 58 years. These groups were chosen based on the overall age distribution of participants in the study, as presented in Figure I.4.2, as well as what was considered to be conceptually meaningful by the researchers. A significance level of .05 was applied throughout.
Figure I.4.2. Overall age distribution with red lines indicating break-points in categorization.
1.5. Preliminary analysis
It is interesting to know how are the scores from PCB Scale, Melbourne Scale and Victoria Scale evolving for staff with their age increasing. We have plotted the age with PCB Scale, Melbourne Scale and Victoria Scale (Figure I.4.3 - 5). From Figure I.4.3, for the PCB Scale, we see that the LB and PCB are slowly increasing with the increase in aging though the LB has a higher increasing rate comparing to PCB. To the contrary, the EC scores are slowly decreasing with the increase in aging. And for the REL it almost remains as a horizontal line. From Figure I.4.4, for the Melbourne Scale, we see that the RECONF and RECONT are slowly increasing with the increase in aging. To the contrary, the SEC scores are gradually decreasing with the increase in aging. And for the VALED it almost remains as a horizontal line. From Figure I.4.5, for the Victoria Scale, we see that the DIA and MONO are slowly increasing with the increase in aging though the MONO has a higher increasing rate comparing to PCB. To the contrary, the COLFULL scores and COLLESS scores are gradually decreasing with the increase in aging. The mean scores for COLFULL are decreasing more when age progresses, compared to COLLESS.
Figure I.4.3. Plot of Age vs. PCB Scale.
Figure I.4.4. Plot of Age vs. Melbourne Scale.
Figure I.4.5. Plot of Age vs. Victoria Scale.
1.6. Study limitations
In the present analysis, both independent and response variables were treated as continuous. This can be problematic with our responses, as by nature they are ordinal variables. Numerical values used are simply codes and thus mathematical properties behind our models are suboptimal. Nevertheless, this approach is commonplace in the social sciences research and in this case we follow suit.
As an expected consequence of the above, almost never are model assumptions satisfied. A few common transformations were attempted with no success. Residuals´ normality and constant variance violations will be reported on a continuous basis throughout the report. These violations are of less importance in the ANOVA part of the analysis as ANOVA is known to be a fairly robust test.
Finally, we are dealing with multilevel data. Individual school staff respondents are nested within schools. In that sense, our observations are not completely independent of each other. There might be a school effect present which we did not test for. We recognize that a mixed model would have been indeed more appropriate in the situation.
When interpreting the results of the current analysis, it is important that the hereby listed shortcomings are kept in mind.
1.7. Report structure
The remaining sections in this report follow the order in which the instruments are implemented in the survey: PCB Scale first, next the Melbourne Scale, and the Victoria Scale at the end. Within each of these, results from the regression and the ANOVA analyses are presented in turn, discussing simultaneously findings for all sub-scales of the particular measure. This way, attention is kept on the research problem within the context of the specific measure, with possibilities to compare and contrast findings across the different sub-scales, as well as between the two different statistical methods applied.
2. PCB Scale
2.1. Regression
Results on the four PCB sub-scales obtained via fitting a simple regression model with age as an independent variable are presented in Table II.1.1. Age tuned out to have a highly significant effect on all sub-scales. It affects Literal Belief (LB), Relativism (REL) and Post-Critical Belief (PCB) positively, while it has a negative effect on External Critique (EC). Despite these highly statistically significant results though, we notice that the effect estimates are quite small. The largest one we observe here is that for Post-Critical Belief, .0120. It shows that for every year increase in a person´s age, their score on the LB sub-scale is expected to go up by .0120 of a point, on average. Exemplifying the same principal for the negative relationship between age and External Critique, we predict a person´s score on the EC sub-scale to decrease by .0068 (this is just over 0.5%) of a point, on average, as they get older by 1 year. What this means, practically, is that, indeed, age influences LB, EC, REL and PCB sub-scales. However, this influence is barely visible, as also reflected by the adjusted coefficient of determination (R2 adjusted). Even in the case of the PCB sub-scale that shows the strongest linear relationship with age, age taken by itself accounts for just 2.6% of the total variability we see in scores. Thus, we can conclude that, in practical terms, age is not really a meaningful predictor of any of the sub-scales of the PCB questionnaire.
Table IIII.1.1. Regression results with age as an IV for all sub-scales in the PCB questionnaire.
sub-scale | age effect estimate | std error | p-value | R2 adj |
LB | .0114 | .0011 | <2e-16* | .0207 |
EC | -.0068 | .0013 | 7.41e-8* | .0051 |
REL | .0032 | .0008 | .0001* | .0026 |
PCB | .0120 | .0010 | <2e-16* | .0260 |
(*) significant at alpha = .05
To visually illustrate this last point, let us turn to Figure II.1.1 on the next page. The scatterplots depict the relationship between age (always on the x-axis) and each of the four PCB sub-scales. The slopes of the fitted regression lines (in red) indicate the strength of this relationship. We can clearly see that the line is moving upward for LB and PCB, in a downward direction for EC and for REL the line on the graph is almost flat. Nevertheless, the main clouds of points even for these four sub-scales appear quite randomly scattered, with no distinct pattern with advancement of age. It may be argued that a relatively small group comprising our oldest respondents (60 years and older) are to some extend responsible for 'pulling' the line in their direction, thus augmenting an otherwise very subtle slope.
As per the model assumptions and only considering the sub-scales with significant age effect, the normality of residuals is not satisfied for any of the four responses – LB, EC, REL, or PCB , which can be seen from the QQ plots on Figure II.1.2. The purpose of QQ plots is to find out if two sets of data come from the same distribution, e.g. in our case, we would like to test if the residuals are normally distributed. QQ plots plot two quantiles again each other, one of the quantile is calculated from residuals and the other quantile is calculated from normal distribution. If the residuals come from a normal distribution the data points (black dots in Figure II.1.2) will fall on the reference line (red line in Figure II.1.2), as we can see from Figure II.1.2 indeed show deviations. As our sample size is larger than 5000 data points, instead of applying Shapiro-Wilk test, the Kolmogorov-Smirnov test yields extremely small p-values for all the sub-scales (<2.2e-16 for LB, EC, REL and PCB).
Figure II.1.1. Scatter plots of age v sub-scales of the PCB measure with regression lines fitted.
Figure II.1.2. QQ plots of residuals from the fitted regression models for LB, EC, and PCB sub-scales.
Further, Figure II.1.3 below shows scatterplots of the predicted values versus residuals, again for the three sub-scales with significant age effect. Residuals seem randomly (though usually not completely symmetrically) scattered around zero for the most part. However, there is a perceptible pattern in all three graphs of residuals varying less towards either one or the other end of the predicted values spectrum.
Figure II.1.3. Predicted values v standardized residuals for the LB, EC, and PCB sub-scales.
2.2. ANOVA
Tackling the problem of whether age has an influence on scores on the PCB measure from the angle of age groups comparison, we apply the analysis of variance method. As a reminder, respondents were divided into five age categories, thus giving us five groups of unequal sizes. For each group, the means and standard deviations of scores on the four PCB sub-scales are listed in Table II.2.1. A visual idea of the groups scores distributions can be obtained from the boxplots in Figure II.2.1. At first glance, average scores on all sub-scales appear quite close to each other across groups. The ANOVA test shows, however, that not all group means are equal. We get the highly significant p-values of <2e-16, 7.41e-08, 0.0001, <2e-16 for LB, EC, REL and PCB scores, respectively.
Table II.2.1. Means and standard deviations on PCB sub-scales according to age group.
age group | n | LB mean(sd) | EC mean(sd) | REL mean(sd) | PCB mean(sd) |
19-24 | 430 | 2.46 (.90) | 3.45 (1.01) | 4.86 (.66) | 4.94 (.84) |
25-35 | 1804 | 2.31 (.86) | 3.35 (1.05) | 5.04 (.70) | 5.11 (.89) |
36-45 | 1279 | 2.49 (.88) | 3.32 (1.03) | 5.02 (.69) | 5.17 (.86) |
46-58 | 1763 | 2.62 (.92) | 3.23 (1.11) | 5.08 (.68) | 5.34 (.87) |
59+ | 184 | 2.88 (.94) | 3.05 ( .97) | 4.99 (.69) | 5.54 (.82) |
Figure II.2.1. Boxplots of PCB scores according to age group
Before proceeding with the investigation of where these differences lie exactly, first, a quick look at the model assumptions. Normality of the residuals is violated for every scale. QQ plots is almost identical to the ones obtained from the regression analysis in the previous section. However, since the residuals distributions are not extremely skewed, this is not considered a serious problem. Additionally, the Levene test of homogeneity of variances was performed for all sub-scales across age groups. Score variances turned out unequal for LB and EC (p-values of 2.5e-3 and 3e-4, respectively). Looking at Figure II.2.1 though, this inequality is not that severe and thus also deemed inconsequential.
Applying the Tukey multiple comparisons adjustment to the p-value, we tested all possible pairwise comparisons within each of the four PCB sub-scales. Differences that are statistically significant at the .05 level are presented in Table II.2.2. To facilitate interpretation, Table II.2.2 is organized in such a way that the cited difference is always in favor of the age group listed row-wise. For example, the mean score on the LB sub-scale of group (59+) is .42 points higher than the mean score on the same sub-scale of group (19-24). Considering scores on the EC sub-scale though, we see that the average of group (19-24) is .40 points higher than the average of group (59+). Generally speaking, what can be gathered from the listed significant differences is that, older groups are likely to score differently from the youngest groups. If we have to pinpoint one age group that is most unlike the rest as measure by the PCB questionnaire, that would be the oldest group (59+). It usually, though not always, displays the greatest differences in absolute value, with the biggest number of other groups. Respondents in the (59+) group, on average, score higher on the LB and PCB sub-scale than any other age group, also, they score lower on EC compared to the three youngest groups.
As interesting as this is, if we consider the magnitude of these significant differences in the context of the measurement scale that is being used here, we see that results are actually not that impressive. Overall, differences rarely exceed or even come close to half a point on the 7-point Likert scale. We have to state that age differences on the PCB Scale are too tiny to practically matter, much in accord with our conclusions in section II.1.
Table II.2.2. Significant pairwise comparisons (alpha=.05) of score means across age groups for all PCB sub-scales.
19-24 | 25-35 | 36-45 | 46-58 | 59+ | ||
LB | 19-24 | .15 | ||||
36-45 | .18 | |||||
46-58 | .16 | .31 | .13 | |||
59+ | .42 | .57 | .40 | .26 | ||
EC | 19-24 | .22 | .40 | |||
25-35 | .12 | .29 | ||||
36-45 | .27 | |||||
REL | 25-35 | .19 | ||||
36-45 | .16 | |||||
46-58 | .22 | |||||
PCB | 25-35 | .17 | ||||
36-45 | .23 | |||||
46-58 | .40 | .23 | .17 | |||
59+ | .60 | .44 | .37 | .21 |
3. Melbourne Scale
3.1. Regression
As already mentioned in the introduction, four sub-scales of the five comprising the Melbourne questionnaire are tested as responses in this analysis. All are taken at their normative level. Results from the four corresponding linear regression models are presented in Table III.1.1. Age does not have a statistically significant effect on scores of the Values Education (VALED) sub-scale, while it has a highly significant effect on the remaining three sub-scales. This effect is negative for Secularisation (SEC), with a magnitude of .0178. This is interpreted the following way: as age increases by 1 year, score on SEC is expected to decrease on average by .0178 of a point. On the contrary, as age increases, a corresponding increase in scores on both the Reconfessionalisation (RECONF) and the Recontextualisation (RECONT) sub-scales is observed. In the case of RECONF, this increase is roughly 1% of a point on average, and in the case of RECONT, the increase is of about 0.7% of a point on average.
Similarly to our discussion in section II.1 regarding the PCB instrument, the results here are of very limited practical importance. Age on its own appears to have only minuscule explanatory power, as quantified by the adjusted R-square statistic. The largest age effect observed, that on the SEC sub-scale, is still only responsible for 2.39% of the total variability in scores. This proportion goes down to less than 1% for RECONF and about 0.7% for RECONT.
Table III.1.1. Regression results with age as an IV for all sub-scales in the Melbourne questionnaire.
age effect estimate | std error | p-value | R2 adj | |
sub-scale | ||||
SEC | -.0178 | .0018 | <2e-16* | .0239 |
RECONF | .0095 | .0016 | 1.38e-9* | .0095 |
VALED | -.0009 | .0013 | .736 | --- |
RECONT | .0069 | .0012 | 1.27e-8* | .0083 |
(*) significant at alpha = .05
The above should not be surprising if we also look at the scatterplots depicted on Figure III.1.1. The non-significant age effect on the VALED sub-scale is reflected by an almost flat regression line. For the remaining three sub-scales, we can indeed see that the line goes upwards for RECONF and RECONT, and downwards for SEC. This last one also shows the steepest slope, indicating that the relationship there is strongest among the three (absolute value of the age effect estimate is .0178, compared to .0095 for RECONF and .0069 for RECONT). Here again, this slope seems to at least some degree attributable to the responses of the participants with highest age. On SEC, older participants tend to score low, while on RECONF and RECONT they tend to sore high.
In terms of the model assumptions, an identical overall picture presents itself as that of the PCB measure. The quantile plots on Figure III.1.2 point to a slightly non-normal distribution of the residuals of the three significant models, confirmed by the Shapiro-Wilk test (p-values of <2.2e-16 for SEC, 5.061e-13 for RECONF and <2.2e-16 for RECONT, respectively). Additionally, residuals variance is not constant, as visible on the scatterplots in Figure III.1.3. Most pronounced for the RECONT sub-scale, we see that variance decreases towards one end of the graph.
Figure III.1.1. Scatter plots of age v sub-scales of the Melbourne measure with regression lines fitted.
Figure III.1.2. QQ plots of residuals from the fitted regression models for SEC, RECONF, and RECONT sub-scales.
Figure III.1.3. QQ plots of residuals from the fitted regression models for SEC, RECONF, and RECONT sub-scales.
3.2. ANOVA
Paraphrasing the research question about the effect of age on scores of the different sub-scales within the Melbourne questionnaire in terms of average scores across the five age groups we defined earlier, we first take a look at the descriptive statistics for each of those groups. As visible in Table III.2.1, means and standard deviations of all groups within a given sub-scale appear relatively similar. This observation about the scores distributions across age groups is further confirmed by the boxplots on Figure III.2.1. While for the other three sub-scales the median (represented by the black line inside each red rectangle) moves slightly up or down according to age group, for the most part this cannot be said for VALED. Indeed, when we run the ANOVA we see that the model is not significant for VALED (just as it was not in the regression analysis above) and thus we will not be discussing this sub-scale further. Models for SEC, RECONF, and RECONT yielded p-values of <2e-16, 1.38e-9, and 1.27e-8, respectively, signifying that average scores on these sub-scales are not all equal across age groups.
Table III.2.1. Means and standard deviations on Melbourne sub-scales according to age group.
age group | n | SEC mean(sd) |
RECONF mean(sd) |
VALED mean(sd) |
RECONT mean(sd) |
19-24 | 201 | 3.70 (1.23) | 3.60 (1.06) | 4.56 (.79) | 5.07 (.75) |
25-35 | 1190 | 3.60 (1.30) | 3.37 (1.07) | 4.46 (.89) | 5.15 (.83) |
36-45 | 902 | 3.44 (1.28) | 3.54 (1.09) | 4.50 (.87) | 5.09 (.82) |
46-58 | 1311 | 3.22 (1.23) | 3.64 (1.03) | 4.51 (.86) | 5.23 (.84) |
59+ | 146 | 2.93 (1.15) | 3.84 (1.08) | 4.49 (.85) | 5.55 (.82) |
Figure III.2.1. Boxplots of Melbourne scores according to age group.
Checking the assumptions, we admit again that the residuals of the three models of interest are not normally distributed. The Shapiro-Wilk test gives the highly significant values of <2.2e-16 for SEC, 5.061e-13 for RECONF and <2.2e-16 RECONT. The normal quantiles plots obtained under ANOVA are identical to those from the regression analysis and displayed on Figure III.1.2. On the other hand, the Levene test of homogeneity of variances was performed for all sub-scales across age groups. Score variances turned out unequal for SEC (p-values of 0.018). Looking at Figure II.2.1 though, this inequality is not that severe and thus also deemed inconsequential.
Lastly, we ran all possible pairwise comparisons for each sub-scale individually, using again the Tukey adjustment for multiple testing. Significant differences at the .05 level are presented in Table III.2.2 in a matrix format (age groups listed row-wise are the ones with higher scores in a particular comparison). Trying to discern a general trend, we see that as far as SEC is concerned, the two oldest groups, (59+) and (46-58), exhibit significantly lower scores compared to the three youngest groups, (19-24), (25-35), and (36-45). On the contrary, older groups score higher on both the RECONF and RECONT sub-scales, though this is only roughly speaking as not all pairwise comparisons obviously hold here. But specifically on the RECONT sub-scale, group (59+) may again be identified, like on the PCB Scale, as "the different one", because it indeed shows significantly higher mean score than any of the other groups. For RECONF, the noticeable thing is that group (25-35) scores significantly lower that all other groups, almost half point when comparing with group (59+).
A word of caution against over-interpretation is justified here as well. These statistically significant differences are still in the range of 10-50% of a single point, for the most part. The biggest difference we observe is that between mean score of group (59+) and mean score of group (19-24) on the SEC sub-scale. This is three-quarters of a point. Hence, once more, age seems to be only a weak determinant of respondents´ scores on the scales in question.
Table III.2.2. Significant pairwise comparisons (alpha=.05) of score means across age groups for SEC, RECONF, and RECONT sub-scales.
19-24 | 25-35 | 36-45 | 46-58 | 59+ | ||
SEC | 19-24 | .43 | .73 | |||
25-35 | .19 | .40 | .70 | |||
36-45 | .22 | .51 | ||||
RECONF | 19-24 | .23 | ||||
36-45 | .17 | |||||
46-58 | .27 | |||||
59+ | .47 | .30 | ||||
RECONT | 46-58 | .14 | ||||
59+ | .47 | .40 | .46 | .32 |
4. Victoria Scale
4.1. Regression
The four sub-scales taken on their normative level were used as dependent variables for the regression analysis of the Victoria questionnaire. Age showed to have a statistically significant effect on all four (Table IV.1.1). This effect is positive for Monologue (MONO) and Dialogue (DIA), meaning that as age increases so do scores on these sub-scales. On the contrary, as age increases scores on the Colourful (COLFUL) and Colourless (COLLESS) sub-scales decrease, as indicated by a negative estimate of the age effect. These relationships can be visualized as on the scatterplots presented on Figure IV.1.1. The fitted regression line has an upward slope for MONO and DIA, but a downward one for COLFUL and COLLESS. We could speculate again that it is the group of oldest respondents that is, to a certain extent, influencing the slope, since if we concentrate only on the main cloud of points, for the most part we are not able to discern a clear direction.
Table IV.1.1. Regression results with age as an IV for all sub-scales in the Victoria questionnaire.
age effect estimate | std error | p-value | R2 adj | |
sub-scale | ||||
MONO | .0125 | .0014 | <2e-16* | .0199 |
COLLESS | -.0115 | .0017 | 1.34e-11* | .0115 |
COLFUL | -.0257 | .0016 | <2e-16* | .0614 |
DIA | .0071 | .0012 | 3.93e-9* | .0087 |
(*) significant at alpha = .05
Figure IV.1.1. Scatter plots of age v sub-scales of the Victoria measure with regression lines fitted.
The strongest age effect found, within this instrument but also across all scales, is that on the COLFUL sub-scale – negative .0257. Thus, we can say that with every year increase in a respondent`s age, we expect an average decrease of their COLFUL score of .0257 of a point. Age here accounts for about 6.1% of the total variance in scores. Indeed, this is the 'most impressive' adjusted R2 value that we have seen throughout. For comparison, age explains 2% of MONO scores variability and about 1.0% for both DIA and COLLESS. However, to put things in perspective and exemplify what this means in practical terms, for MONO score we have to point out that a person would need to get 40 years older before they change their score on this sub-scale from 4 to 3, from 3 to 2, or similarly. Same thing said in a different way, we would need to look at respondents who have, on average, 40 years age difference, in order to observe a score difference of 1 point (with the older subjects having lower scores). Thus, the question of how much practical significance these results have, despite their high statistical significance, is no less vital here than with the PCB and Melbourne Scales.
With regards to the model assumptions, the QQ plots on Figure IV.1.2 show less than ideal residuals distributions, although they do not appear extremely non-normal either. The Shapiro-Wilk test returns p-values very close to 0 (namely, <2.2e-16 for MONO, 3.646e-13 for COLLESS, 7.889e-14 for COLFUL and <2.2e-16 for DIA). The constant variance assumption is also not fully satisfied, with residuals spread smaller towards one end of the graphs in Figure IV.1.3. The most severe violation seems to be that of the residuals of the DIA model.
Figure IV.1.2. QQ plots of residuals from the fitted regression models for Victoria sub-scales.
Figure IV.1.3. Predicted values v standardized residuals for Victoria sub-scales.
4.2. ANOVA
In order to explore not the overall age effect but how our five distinct age groups differ from each other with respect to their responses on the Victoria measure, we need to first check the scores distributions within each group, as shown on Table IV.2.1. Again, on first look they do not appear to differ drastically. Turning to the boxplots displayed on Figure IV.2.1 though, we suspect something more interesting going on – it is clear that the medians (which lie quite close to the means, our statistic of interest) are shifting as we move from the youngest to the oldest groups. Performing the ANOVA test on the four Victoria sub-scales, we get significant results for all, as indicated by the very low p-values: <2e-16 for MONO, 1.34e-11 for COLLESS, <2e-16 for COLFUL, and 3.93e-9 for DIA. In other words, we do have at least two age groups within each sub-scales that differ significantly from each other.
Table IV.2.1. Means and standard deviations on Victoria sub-scales according to age group.
age group |
n |
MONO mean(sd) |
COLLESS mean(sd) |
COLFUL mean(sd) |
DIA mean(sd) |
19-24 | 246 | 2.52 (1.05) | 4.05 (1.07) | 5.05 (.93) | 5.29 (.90) |
25-35 | 1211 | 2.33 (.94) | 3.94 (1.15) | 4.74 (1.07) | 5.38 (.85) |
36-45 | 938 | 2.51 (.97) | 3.88 (1.17) | 4.50 (1.14) | 5.40 (.85) |
46-58 | 1333 | 2.64 (.99) | 3.76 (1.24) | 4.26 (1.20) | 5.52 (.82) |
59+ | 142 | 2.91 (1.08) | 3.29 (1.15) | 3.74 (1.15) | 5.65 (.72) |
Figure IV.2.1. Boxplots of Victoria scores according to age group.
In terms of the assumptions of the model, residuals are not normally distributed for any of the sub-scales, as indicated by the Shapiro-Wilk test (p-value <2.2e-16 for MONO, 3.646e-13 for COLLESS, 7.889e-14 for COLFUL and <2.2e-16 for DIA). However, these deviations from normality are not too severe, referring back to the QQ plots on Figure IV.1.2. Non-normality is worst for COLLESS, where the distribution is undoubtedly left-skewed. When testing the variance homogeneity across groups, we see violations only for the COLLESS and COLFUL sub-scales (p-values 0.00476 and 8.265e-08, respectively). Even there, variance differences do not appear huge.
Just like with the PCB and the Melbourne measures before, here again we ran all possible pairwise comparisons applying the Tukey adjustment to the p-value. Statistically significant differences according to sub-scale are listed in Table IV.2.2 (reminder, groups listed row-wise have higher mean scores than groups listed column-wise). For MONO, the noticeable thing is that group (25-35) scores significantly lower that all the other groups, albeit by not much except group (59+). On the COLLESS sub-scale, the lowest scoring group is the oldest – (59+). All differences here exceed three-quarters of a point. Next, the COLFUL sub-scale presents perhaps the most interesting results. All pairwise comparisons within it are significant, meaning that every group´s mean score is different from any other group´s mean score. Visible also on the respective plot on Figure IV.2.1, average COLFUL scores are monotonically decreasing as a function of age. Thus, an older group´s mean score is always significantly lower than a younger group´s mean score. In terms of effect magnitude, we see some diversity, with differences ranging from one-fourth of a point (.25 between (36-45) and (46-58)) to more than a point (1.31 between (19-24) and (59+)). Finally, for the DIA sub-scale, the oldest group (59+) scores highest followed by the group (46-58), on average, compared to the three youngest groups.
Table IV.2.2. Significant pairwise comparisons (alpha=.05) of score means across age groups for all Victoria sub-scales.
19-24 | 25-35 | 36-45 | 46-58 | 59+ | ||
MONO | 19-24 | .20 | ||||
36-45 | .19 | |||||
46-58 | .31 | .13 | ||||
59+ | .38 | .58 | .40 | .27 | ||
COLLESS | 19-24 |
.30 |
.76 | |||
25-35 |
.18 |
.64 | ||||
36-45 | .58 | |||||
46-58 | .46 | |||||
COLFUL | 19-24 | .31 | .55 | .80 | 1.31 | |
25-35 | .24 | .49 | 1.00 | |||
36-45 | .25 | .76 | ||||
46-58 | .51 | |||||
DIA | 46-58 | .23 | .14 | .12 | ||
59+ | .36 | .26 | .24 | .12 |
5. Conclusions
In an attempt to determine whether age has an effect on responses on the PCB, Melbourne, and Victoria questionnaires, data from over 6000 school staff members from different Catholic schools in Flanders were analysed. Two alternative statistical approaches were used – simple regression and one-way analysis of variance.
To summarize results, age showed significant effect on almost all sub-scales as estimated by the regression model and all sub-scales according to ANOVA. However, this effect, despite its high statistical significance, was very tiny throughout. In the best case (Colourful sub-scale on the Victoria instrument), age could account for just 6.1% of the total variability in scores. In terms of comparing mean scores among different age groups, it was again on the Colourful sub-scale that we observed the most and biggest significant differences. Still, only once did we see a difference of 1.3 point, and that was between the youngest and the oldest groups. Apart from this, observed differences could be classified as rather trite.
As a general trend that would require further exploration, we noticed that most often than not it was participants with oldest age that showed response patterns different from all the rest – either youngest, or middle age individuals. Overall though, this trend was also not consistent.
Having a relatively big sample size allowed us to discern effects with high power. Indeed, age does influence responses on the three questionnaires tested here. All the while, this effect seems quite subtle and slow over time. Thus, we have to conclude that age taken by itself is not a very meaningful predictor of scores on the PCB, Melbourne, and Victoria Scales.