Fernanda Oliveira-Ferreira; et al
29 de abril de 2014
School Achievement Test: normative data for a representative sample of elementary school children
Fernanda Oliveira-FerreiraI; Danielle Souza CostaII; Letícia Rettore MicheliII; Livia de Fátima Sílvia OliveiraII; Pedro Pinheiro-ChagasII; Vitor Geraldi HaaseII
IUniversidade Federal dos Vales do Jequitinhonha e Mucuri, Diamantina, MG , Brazil
IIUniversidade Federal de Minas Gerais, Belo Horizonte, MG, Brazil
ABSTRACT
The present study established norms for the spelling and arithmetic subtests of the School Achievement Test (Teste do Desempenho Escolar [TDE]) in two Brazilian cities located in the state of Minas Gerais and compared the results with those obtained from the original normative sample. A stratified proportional sample of 1,034 students from Belo Horizonte and Mariana, from the 1st to 6th grades, was selected. The participants were assessed by the spelling and arithmetic subtests of the TDE. Significant differences were found between the results from Minas Gerais and Rio Grande do Sul, with moderate to high effect sizes. Significant differences were found in percentiles and classification parameters. The educational performance of the children from Minas Gerais was generally classified as less than expected (i.e., inferior) when the original norms were used as a classification parameter. Considering the high variability of educational data in different Brazilian regions, using norms for educational assessment based on only one Brazilian region is inappropriate.
Keywords: school achievement; normative data; school performance; elementary school.
Introduction
School evaluation is a fundamental procedure in child neuropsychological assessment. Behavioral and learning problems are some of the main causes of referral to pediatric care, encompassing professionals from different areas such as psychologists, neurologists, speech therapists, and psychiatrists (Artigas-Pallarés, 2002; Castaño, 2002; Rotta, 2006).
Learning disabilities play a significant role among family complaints, mainly because they involve multiple factors and impact different aspects of children and their families' lives. Learning difficulties are seen as conditions of psychosocial vulnerability (Rutter, 1987). To establish diagnoses of childhood neuropsychiatric or developmental disorders, verifying whether the symptoms manifested by the child provoke poorer school performance and significantly impact the students' lives is essential.
One of the diagnostic criteria for learning disabilities is school performance that is less than the expected average according to age, grade, and intelligence (Hamill, Leigh, McNutt, & Larsen, 1988; Organização Mundial da Saúde, 2002). Educational performance assessment is an important component of child neuropsychological assessment. To accurately perform this assessment, standards are needed based on appropriate population norms that should be considered references for learning evaluation.
The unequal educational conditions in Brazil produce very different results in academic learning in different types of schools (Alves & Franco, 2008). Research developed from data from the Brazilian Assessment of Basic Education in 1995 (Ferrão, Beltrão, Fernandes, Santos, Suárez, Andrade, 2001), which controlled for socioeconomic status, showed that 14% of the variation in student performance in mathematics was attributable to school variations. The authors utilized a multilevel regression model, considering the student as the first-level unit and the school as the second-level unit. Ferrão et al. (2001) verified that the school-effect was different in different Brazilian regions, with a 7.6% effect in the southern region to a 17% effect in the northeastern region. This indicates the possible relevance and influence of the "school" factor in determining students' educational performance (Alves & Franco, 2008). Moreover, regions with larger effects of school require more caution when interpreting the results of comparisons of student performance with normative data, considering that high heterogeneity may occur even within a single region.
A limitation of learning assessment is a lack of standardized, validated, and normalized instruments that measure school performance in the Brazilian population. The use of validated and standardized tests allows the localization of the participants' raw scores relative to the scores of standardization samples (Pasquali, 2003). The only psycho-educational instrument that has been validated and standardized for the Brazilian population is the School Achievement Test (Teste do Desempenho Escolar [TDE]; Stein, 1994) whose norms were developed based on a school sample from Porto Alegre, Rio Grande do Sul. This sample consisted of students from the 1st to 6th grades from six schools in Porto Alegre: four public schools and two private schools. An average of 15 subjects per grade in each school was selected, resulting in a total sample of 538 students.
The limitation of the use of norms from the TDE from only one Brazilian state was described by Lucio, Pinheiro, & Nascimento (2009). Considering that school performance may vary depending on regional differences among Brazilian states, the use of norms established for other Brazilian regions may generate incorrect results and interpretations. For example, if the norms of a test were established for a region that presents higher academic performance than other regions in the country, then a child may be classified as "below the expected average" according to the normative parameters, even if the student's performance is consistent with the average performance of children from the student's own state.
Notably, the TDE manual (Stein, 1994) presents norms that are specific to one Brazilian state, Rio Grande do Sul. Considering the high heterogeneity of Brazilian regions, the authors argued that the establishment of norms for the entire country may be inappropriate for some regions. Authors of the TDE also argued that research should be conducted to develop specific norms for other Brazilian regions. However, no published data currently exist for other Brazilian regions, and the original norms of the TDE are used by a wide range of professionals from different research areas and Brazilian regions.
The main objective of the present study was to present norms for the spelling and arithmetic subtests of the TDE obtained from two Brazilian cities located in the state of Minas Gerais. The study also compared the results of the original normative sample that are presented in the TDE manual and the norms developed for the state of Minas Gerais, considering that these norms were obtained using a larger sample than the original one.
Methods
Sample
Approximately 2.5 million people currently live in Belo Horizonte, the capital of Minas Gerais, one of the largest states in southeastern Brazil. The city has 148 districts subdivided into nine administrative regions. According to the Brazilian Institute of Geography and Statistics (Instituto Brasileiro de Geografia e Estatística;http://www.ibge.gov.br; accessed December 5, 2012), 348,477 students were enrolled in primary schools in Belo Horizonte in 2009, with 279,901 enrollments in public schools and 68,576 enrollments in private schools. Thus, approximately 80% of students are enrolled in public elementary schools (i.e., municipal, state, and federal schools).
To obtain normative data not only for the capital city, which would also be useful for other cities in Minas Gerais, we also selected a small city to include in the sample. Mariana was the first city in the state of Minas Gerais. Located 112 km from Belo Horizonte, Mariana had a population of 54,689 in 2009 (Instituto Brasileiro de Geografia e Estatística; http://www.ibge.gov.br; accessed December 5, 2012). Regarding school enrollment, data from 2009 indicated 10,139 registrations in primary schools. Enrollment in public schools (i.e., state and municipal) accounted for 89% of this total (9,050), whereas only 1,092 students were enrolled in private schools.
To obtain a representative sample, the schools were selected according to the proportional distribution of enrollments in the population. To avoid selection bias due to the location of the school, we selected schools from five different regions of Belo Horizonte (Center–South, Northeast, Northwest, East, and Pampulha) and four different districts in Mariana. The regional and district selection of schools was random. If the selected school could not participate for any reason, then a new school was again randomly selected.
Twelve schools in the city of Belo Horizonte (i.e., seven public schools and five private schools) located in 11 districts and five different regions participated in the study. Mariana's sample was composed of four schools from different districts (i.e., two public schools and two private schools).
A total of 1,034 students participated in the study (i.e., 646 from Belo Horizonte and 388 from Mariana), all of whom were regularly enrolled in a primary public or private school in the 1st to 6th grades from Belo Horizonte and Mariana. This sample represented 95% confidence intervals of 3.85 and 4.88 for the student populations of Belo Horizonte and Mariana, respectively. The mean age of the students was 9.78 ± 2.03 years. Sample characteristics are presented in Table 1. The present study was approved by the Ethics Committee of the Federal University of Minas Gerais (ETIC 42/08).
School Achievement Test
The TDE (Casa do Psicólogo, São Paulo) is the only validated, standardized, and normalized psychoeducational test for the assessment of primary academic performance in the Brazilian population. This test is composed of three subtests that assess spelling, arithmetic, and reading according to expected educational achievement from the 1st to 6th grades. We analyzed only the results from the spelling and arithmetic subtests.
The spelling subtest consisted of 34 single words that were read by the examiner and needed to be written by the children, presented with increasing difficulty and ranging from mono- to polysyllabic words. The words were dictated individually, then in the context of a sentence to avoid possible ambiguity, and again individually. This subtest assesses the ability to decode from the phoneme to the grapheme and the ability to comprehend orthographic rules.
The arithmetic subtest is composed of 38 calculations with varying degrees of complexity. Three calculations were orally presented (i.e., simple single-digit addition, simple single-digit subtraction, and a comparison of two numbers). Thirty-five calculations needed to be solved and recorded on test paper, including simple single-digit addition (e.g., 1 + 1 = ), two-digit addition (e.g., 17 + 21 + 40 = ), four-digit addition (e.g., 452 + 137 + 245 = ), simple subtraction (e.g., 4 – 1 = ), two-digit subtraction (e.g., 43 – 18 = ), four-digit subtraction (e.g., 3415 – 1630 = ), simple multiplication (e.g., 2 × 4 = ), two-digit multiplication (12 × 15 = ), simple division (6 ÷ 3 = ) , two-digit division (e.g., 6630 ÷ 65 = ), fraction multiplication (e.g., 2/3 – 1/3 = ), and exponentiation (e.g., 62 + 33 = ).
According to school grade and age, the TDE provides a classification based on the scores of the children on the test (Stein, 1994). According to the TDE manual, performance <25th percentile of the norms is considered lower than the expected average for age and educational grade (inferior). Performance between the 25th and 75thpercentiles of the norms is considered consistent with the expected average. Performance >75th percentile of the norms is considered higher than the expected average for the child's age and schooling.
Procedure
After the school selection, school principals were contacted and the objectives and study procedures were explained. All parents of the students from the 1st to 6th grades received an invitation letter to participate in the study and a consent form. Only children whose consent forms were signed by the parents or legal guardians participated in the study.
All procedures were conducted in rooms assigned by the schools. Application of the spelling and arithmetic subtests of the TDE occurred collectively, with ~5–10 students per class in each evaluation.
In the present study, data from the reading subtest were not analyzed because this subtest requires individual assessment, which made the composition of a sufficient sample size difficult to normalize the reading subtest. TDE subtests had high correlations with each other (p = .001). The correlation between spelling and reading (r = .865) was greater than the correlation between spelling and arithmetic (r = .791). The spelling and reading subtests presented almost the same level of correlation with the total school performance score (r = .952 and .958, respectively), indicating that the absence of one of these subtests would not impede the assessment of overall academic performance. However, a considerable number of conflicting results have been reported about the neurocognitive mechanisms of reading and spelling (Pammer, Connell, & Kevan, 2010). Therefore, difficulties in reading and spelling in children are not uniform in the literature. This implies that the exclusion of the reading subtest may be one limitation of the present study.
At the end of this stage of the study, the participants received individual reports to inform them of their results on spelling and arithmetic subtests. The participating schools received reports on the overall performance of its students, and lectures were presented to some teachers and parents concerning learning disabilities.
Statistical analysis
To determine possible differences in performance between students from Belo Horizonte and Mariana and students from public and private schools, a two-way analysis of variance (ANOVA) was performed with the data obtained from the Minas Gerais sample, considering the score on each TDE subtest as the dependent variable, with city, type of school, and school grade as factors.
Similar to the original study, to obtain classification standards for the TDE for Minas Gerais state, the first quartile (i.e., 25th percentile) was considered the cutoff for lower performance, and the last quartile (i.e., 75th percentile) was considered the cutoff for superior performance. Scores between the 25th and 75th percentiles were considered performance according to the average.
The participants were classified according to the original norms for the TDE, which were developed for the original Porto Alegre sample, and afterward were classified according to the norms constructed in the present paper. χ2test was used to investigate differences in the frequency distribution of ratings according to each criterion. To verify the existence of performance differences between the original sample from Rio Grande do Sul and the present results, effect sizes (Cohen's d; Cohen, 1988) of these differences were calculated for each school grade for the spelling and arithmetic subtests. Data were normally distributed according to the Kolmogorov-Smirnov test (p > .05).
Results
Discrimination between Minas Gerais cities and school type (public or private)
Significant main effects of city and school type were found on the students' performance on the subtests. For the spelling subtest, Belo Horizonte students had significantly better scores (F1,1034 = 19.08, p < .001, ηp2 = .019) than Mariana students. The same was found for the arithmetic subtest (F1,1034 = 51.81, p < .001, ηp2 = .049). The differences between public and private schools were also significant for the spelling (F1,1034 = 71.35,p < .001, ηp2 = .066) and arithmetic (F1,1034 = 113.65, p < .001, ηp2 = .101) subtests with greater effect sizes. A significant interaction was found between city and school type for the spelling subtest (F1,1034 = 8.38, p = .004, ηp2 = .008). Public schools in Mariana had lower spelling performance than Belo Horizonte public schools. The same results were not found for private schools
As expected, significant differences were found between grades for spelling (F1,1034 = 100.54, p < .001, ηp2 = .332) and arithmetic (F1,1034 = 310.24, p < .001, ηp2 = .606). On both subtests, only the 4th and 5th grades were not significantly different from each other. A significant interaction was found between city, school type, and grade for spelling (F1,1034 = 3.55, p = .003, ηp2 = .017) and arithmetic (F1,1034 = 5.02, p < .001, ηp2 = .024). Private schools had better scores than public schools in all grades, but the differences between the students in the two cities decreased in the higher grades.
In summary, the two-way ANOVA for the spelling subtest revealed that the differences between school grades represented 50.0% of the variance in performance (p < .001), whereas the type of school represented 7.0% of the variance. Considering the arithmetic subtest, school grade represented 63.0% of the performance variance, and the type of school represented 5.0% of the variance. These results are similar to the original results obtained from the Porto Alegre sample and reinforce the need to use specific norms for each school grade. Figure 1 shows the mean and standard deviations according to city and school type for all school grades for the spelling subtest.Figure 2 shows the mean and standard deviations according to city and school type for all school grades for the arithmetic subtest.
Comparison of performance between Minas Gerais and Rio Grande do Sul
Spelling subtest. Table 2 shows the performance of students from Minas Gerais and Rio Grande do Sul on the spelling subtest of the TDE (Stein, 1994), organized according to grade. Additionally, the effect sizes (Cohen, 1988) for performance differences between states are shown.
Generally, students from Rio Grande do Sul had performance that was 20.05% higher than students from Minas Gerais on the spelling subtest. The effect sizes (Cohen's d) ranged from moderate to high for all grades, with the exception of the 1st grade where a small effect was found (Cohen's d = .28). However, the effect sizes were high for the 2nd, 3rd, 5th, and 6th grades (Cohen's d > .80), confirming the existence of large differences between educational data from Rio Grande do Sul and Minas Gerais.
Arithmetic subtest. Table 3 shows the performance of students from Rio Grande do Sul and Minas Gerais on the arithmetic subtest, organized according to grade. The effect sizes for performance differences between Porto Alegre (Rio Grande do Sul) and Belo Horizonte and Mariana (Minas Gerais) are shown.
Generally, students from Rio Grande do Sul performed 12.38% higher than students from cities in Minas Gerais on the arithmetic subtest. The effect sizes ranged from moderate to high, and the most significant difference was found for the 2nd grade (Cohen's d = .88).
Normative data for Minas Gerais
To verify the classification cutoff for each grade in the Minas Gerais sample, percentiles for each classification and grade were calculated. Table 4 shows these results compared with the percentiles used for the classification criteria in the original data from Porto Alegre, Rio Grande do Sul.
Differences were found in the percentiles and school performance classifications between the Rio Grande do Sul and Minas Gerais samples. Generally, the cutoffs for the Minas Gerais sample were lower than the cutoffs for the Porto Alegre sample. To illustrate the comparison between the Minas Gerais data and original Porto Alegre data, the participants were classified according to the two classification criteria (i.e., the original norms and the Minas Gerais norms). Figure 3 shows the absolute frequency of the participants for each classification criterion (i.e., inferior, average, and superior) on the spelling subtest according to both Rio Grande do Sul and Minas Gerais normalization.
Figure 4 shows the absolute frequency of the participants classified as inferior, superior, or consistent with the expected average according to both classification criteria (i.e., norms from Rio Grande do Sul and norms from Minas Gerais) on the arithmetic subtest.
The frequency of the participants in each classification was significantly different between the criteria (Rio Grande do Sul and Minas Gerais norms) on both the spelling subtest (χ2 = 791.23, p < .001) and arithmetic subtest (χ2 = 907.31, p < .001).
Discussion
Psychoeducational assessment is important for characterizing learning profiles and a relevant tool for advancing knowledge of learning and developmental disorders. However, appropriate psychoeducational assessments must be conducted using accurate instruments and proper norms to correctly classify academic performance. Considering the discrepancies in the educational performance of students from different Brazilian regions, developing specific norms for each region is necessary. This would allow an appropriate interpretation of the results obtained from psychoeducational instruments. Learning disorders can be misdiagnosed when the classification is based on norms from a region that is different from that of the original proband. A child could be considered inferior in academic ability according to a test manual, whereas the child's performance is actually consistent with the academic curricula for his/her region and average school performance.
The present results showed that the established norms for the Minas Gerais cities were different from the norms established from the original Porto Alegre data. When classified according to the original normative data, the Minas Gerais sample showed a leftward shift of the classification distribution. This means that when the TDE manual was used, a tendency to underestimate the educational performance of the students from Minas Gerais occurred.
Importantly, the present study revealed small differences between the Minas Gerais and Porto Alegre samples on the spelling subtest for the 1st grade. This result indicates that differences are small at the beginning of the schooling process and increase during schooling. This could be related to a school-effect (Alves & Soares, 2007), indicating that learning differences may be a consequence of the educational process.
Notably, the differences observed between the original norms of the TDE and the norms obtained from Minas Gerais are not consistent with data from the national assessments promoted by the Brazilian Education Ministry (Ministério da Educação; idep.inep.gov.br; accessed December 5, 2012), which are reliable parameters of Brazilian school performance. The Basic Education Developmental Index (Índice de Desenvolvimento da Educação Básica, Ministério da Educação; idep.inep.gov.br; accessed December 5, 2012) was created in 2007 to measure the quality of Brazilian education. This index is calculated based on the assessment of student performance on tests and rates of school failure. This index is measured every 2 years, and the results may vary from 0 to 10. The Brazilian goal is to reach an index of 6.0 by 2022, which corresponds to the index for developed countries. In 2007, Belo Horizonte's index was 4.8, and Porto Alegre's index was 4.1. In 2009, Belo Horizonte's index was 5.3 and Porto Alegre's index was 4.6. These results indicate that school performance in Belo Horizonte and Porto Alegre is similar, and Belo Horizonte has a slightly higher index than Porto Alegre.
Considering these national indices, normative data of the TDE should be similar for Belo Horizonte and Porto Alegre or Belo Horizonte's norms should present slightly higher values than Porto Alegre. However, comparisons of the data from Porto Alegre and Minas Gerais verified that the original normative data presented significantly higher values than the norms from Minas Gerais. These results suggest that the original normative data from Porto Alegre do not reflect the current reality of school performance in that region, which may be related to changes in the educational performance because the original data were collected in 1993 compared with more recent results from national educational assessments obtained in 2007 and 2009.
Norms for the reading subtest were not presented in the present study, which may be one limitation. The reading subtest was conducted as an individual assessment. Because of this, the sample size for this subtest was not sufficiently large (219 participants) to allow the establishment of parameters for this subtest. This sample is being extended to allow future normalization for the reading subtest. With regard to the other subtests, the sample size analyzed in this study (n = 1,034) was almost twice the size used in the original sample from Porto Alegre (n = 538), thus increasing the reliability of our results.
The present study argues for the need to establish specific norms for psychoeducational tests for different Brazilian regions. These different norms for each region should be presented in the tests' manuals to help professionals provide more accurate and reliable diagnoses for learning disabilities.
Acknowledgments
This research was supported by funding from the National Research Council of Brazil (CNPq, protocol no. 307006/2008-5 and 401232/2009-3) and Research Support Foundation of Minas Gerais (FAPEMIG, protocol no. APQ-02755-10). The authors would like to thank Prof. Dr. Lilian Stein for authorizing the publication of the original norms of the TDE from Porto Alegre in Tables 2 and 3 of the present study.
References
Alves, M. T. G., & Franco, C. (2008). A pesquisa em eficácia escolar no Brasil: Evidências sobre o efeito das escolas e fatores associados à eficácia escolar. In N. Brooke, & J. F. Soares (Eds.), Pesquisa em eficácia escolar: Origens e trajetórias (pp. 552). Belo Horizonte: Editora UFMG.
Alves, M. T. G., & Soares, J. F. (2007). As pesquisas sobre o efeito das escolas: Contribuições metodológicas para a Sociologia da Educação. Sociedade e Estado, Brasília, 22(2), 435-473.
Artigas-Pallarés, J. (2002). Problemas associados a la dyslexia. Revista de Neurología, 34, S7-S13.
Castaño, J. (2002). Aportes de la neuropsicología al diagnóstico y tratamiento de transtornos de aprendizaje.Revista de Neurología, 34, S1-S7.
Cohen J. (1988). Statistical power analysis for the behavioral sciences, 2nd edition. Hillsdale, N.J.: Lawrence Erlbaum.
Ferrão, M. E., Beltrão, K. I., Fernandes, C., Santos, D., Suárez, M., & Andrade, A. C. (2001). O SAEB—Sistema Nacional de Avaliação da Educação Básica: objetivos, características e contribuições na investigação da escola eficaz. Revista Brasileira de Estudos de População, 18, 112-130.
Hammill, D. D., Leigh, J. E., McNutt, G., & Larsen, S. C. (1987). A new definition of learning disabilities. Learning Disability Quarterly, 11, 217-223.
Lucio, P. S., Pinheiro, A., & Nascimento, E. (2009). O impacto da mudança no critério de acerto na distribuição dos escores do subteste de leitura do Teste do Desempenho Escolar. Psicologia em Estudo, 14, 593-601.
Organização Mundial da Saúde (2002). Manual diagnóstico e estatístico de transtornos mentais (Trad. Cláudia Dornelles), 4th edition revised. Porto Alegre: Artmed.
Pammer, K., Connell, E., & Kevan, A. (2010) Spelling and reading: using visual sensitivity to explore shared or separate orthographic representations. Perception, 39(3), 387-406.
Pasquali, L. (2003). Psicometria: Teoria dos testes na Psicologia e na educação. Petrópolis: Vozes.
Rotta, N. T. (2006). Transtornos de aprendizagem: Abordagem neurobiológica e multidisciplinar. Porto Alegre: Artmed.
Rutter, J. (1987). Psychosocial resilience and protective mechanisms. American Journal of Orthopsychiatry, 57, 316-331.
Stein, L. M. (1994). Teste de Desempenho Escolar: Manual para aplicação e interpretação. São Paulo: Casa do Psicólogo.
Correspondence:
Fernanda Oliveira-Ferreira
Rua da Glória, 187, Centro, Diamantina – MG, Brazil. 39100-000.
E-mail: ferreira.ufvjm@gmail.com
Artigo original:
