Home Elements for a Diagnosis of the Spanish Educational System

Previous Table of Contents Next

2
Academic performance


Introduction

The analysis of the academic performance is a key factor in the diagnosis of any educational system, in the stage of ESO as well as in others. Although the fact is that the results of education come to light above all in the long term and must not be limited to acquiring knowledge and skills in certain subjects, it is no less certain that the degree to which such acquisition takes place provides most reliable information on the real efficiency of schools as to their most obvious objective: to teach the youth. Evaluation of academic performance in fact has a double aim: on the one hand it shows to what extent the students achieve the specific learning the main effort is aimed at; on the other, it provides us a certain symptom of the efficiency of schooling, as it is difficult for schools to achieve difficult, abstract objectives - such as acquisition of values, forming the character, creation of study and work habits, love of culture, etc. - if one does not achieve at least the more easy, specific objectives, such as the objectives of learning, which in any case is always required of them by society.

Thus, the first effort in diagnosis must be aimed, as is in this case, at checking what the students learn at school in relation to the main subjects. As already explained, these are specifically the students in the last stage of compulsory schooling.

Now, there are special difficulties in ascertaining what the students learn in the school system. It is difficult, if not impossible, to separate the effects caused in them by school learning, that is to say received at school, in relation to the more general and diffuse learning arising from the student’s own immersion in the social system, their families, their diverse social surroundings, the media, etc. Moreover, in the present situation of the Spanish educational system at secondary education level, when major changes are being made due to implementation of the relevant LOGSE reform, it is especially difficult to calculate what the students learn in general. The main reason for this is that different approaches to curricula exist together in the system, some from the old structure of the educational system (EGB, BUP, the Middle Education Reform and traditional Vocational Training) and others due to implementation of the LOGSE (ESO and the new Vocational Training).



Tests, populations, samples

Due to the existence of these different curriculum approaches, it is complicated to prepare valid tests to calculate the learning by all the students in secondary education, on the basis of the curriculum schemes for each one of the educational options. Due to this, the decision was taken to evaluate not the content of the different curricula, or their minimum common factors, but what the team that designed the tests for the study considered basic, desirable learning for 14 and 16 year old students. Thus, the aim of the study is not to determine the effectiveness and efficiency of the curricula. It is not an attempt to know whether the teaching programme followed by the students - very diverse indeed - allows the objectives to be achieved, or to what extent, but to achieve a diagnosis of the learning by Spanish adolescents in the subjects stated. Specifically, administration of the tests is aimed at determining what students of 14 and 16 know and know how to do in the basic fields of Language, Mathematics, Nature Sciences and Geography and History.

However, the different curricula have been considered as implicit expressions of a general common standard, although preparation of the tests was based on what the experts in each one of the fields consider a desirable, correct mastery of the subjects evaluated for the 14 and 16 year old age groups.

As to the population of students examined, one must bear in mind that, when the study refers to students of 14 and 16 years, the population considered is that of students in the relevant courses for 14 and 16 years in the formal scheme of the educational system, that is to say, 8th of EGB and 2nd of ESO on the one hand, and on the other 2nd of BUP and 4th of ESO, 2nd of VT and 2nd of REM. Due to this, the sample of students is based on a mixed definition of the population based on age and academic year. Thus, in 8th of EGB and 2nd of BUP there are students over 14, normally repeating, while there are also 14 year old students in lower years. Likewise, 2nd of BUP, 4th of ESO, 2nd of VT and 2nd of REM have students older than 16, while there are also students of 16 repeating previous academic years. In fact, in the sample examined of 8th of EGB and 2nd of ESO, there is 74.1% of students born in 1983, the foreseen for that academic year, and 24.1% of older students born before 1983, logically students who have repeated a year. The sample for 16 years old has 64.4% of students in the theoretical age for that academic year, that is those born in 1981; 35.3% of students born before 1981, presumably repeaters, and 0.3% of advanced students born in 1982 or 1983.

It is necessary to refer to the number of students who comprised the diverse samples established. In evaluation of Reading Comprehension, Grammar and Literature, and Mathematics, 46,535 students were examined, 20,642 of 14 from 761 schools and 25,893 of 16 from 913 schools. Fifteen Autonomous plus Ceuta and Melilla participated in this evaluation. The Regions of Andalusia, the Canaries did not participate. The Basque Country, claiming technical reasons, only participated in the study for 14 year old students. The samples, of course, were established according to proper scientific criteria, without any kind of bias, with active intervention by experts from all the Administrations involved.

The study of Nature Sciences and Geography and History was implemented in the Autonomous that, at the time of application had still not received their full devolved competencies in education, that is the 10 Autonomous Regions known as the "MEC Territory". Tests were administered to 6,560 students in these areas, 3,374 of 14, who studied at 127 schools and 3,186 of 16, enrolled at 122 schools.

Spelling and written expression were evaluated in a sample of 3,460 students, 1,680 of 14 from 69 schools, and 1,780 of 16 from 73 schools.



Methodological lines

To present the results of evaluation of the academic performance of the 14 and 16 year old students, the methodology used was based on the latest measurement schemes in Psychology and Education, that have many advantages over the traditional techniques. In spite of its novelty, it is a proven, successful methodology that is gradually being implemented by countries which perform periodical evaluations of the educational system; in fact, some of these resources have already been taken advantage of in previous studies by the INCE, although combined with other more conventional ones.

The Report on Academic Performance prepared by the relevant Committee provides a full explanation of the methodology used. Readers who are interested must thus refer to it, as these pages may only provide a very elementary notion of it.

For each one of the subjects aforementioned, the tests for 14 and 16 had some common items that later, thanks to the calculation method used, allowed expression of the performance by the 14 and 16 year old students on a common scale and evaluation of the change between the two points.

Traditionally, performance results have been presented in terms of percentage of correct answers. In order to facilitate understanding of the results this chapter (like the Preview of Results initially published) partially resorts to such presentation. However, the interpretation of that procedure must be made while bearing in mind some exceptions that shall be mentioned hereunder.

The average percentage of correct answers indicates the percentage of correct answers by the students in the items of the different tests. One must not consider 50% of correct answers as a "pass". If the same students had answered another, somewhat easier or more difficult test, they could have obtained different correct answer percentages. It is wrong to identify a certain value of the proportion of correct answers with failure or success in a subject. There is no a priori value that may be considered as insatisfactory performance.

The average percentage of correct answers does not indicate what the students do or do not know. Only a more detailed analysis of the results would be able to determine this to that extent. The percentage of correct answers does not take into account the difficulty of the items. Two students with a same proportion of correct answers may have very different levels of knowledge. A student may have answered the 10 most difficult items correctly and another the 10 easiest. The both would have the same marks and, even worse, the same degree of certainty as to their marks.

The percentage of correct answers may not in any way refer to the content. The percentage of correct answers may be obtained from items with totally different contents. Due to that, one may not say what cut-off point there must be for a satisfactory level of performance. The percentage of answers does not indicate the importance of the items not correctly answered, nor how many of the subjects have not answered them.

Although each subject in the tests for 14 and 16 year olds shared some items, the percentage scales do not allow comparisons to be made between age groups. In this sense, for example, it would be wrong to consider that the results of Grammar and Literature for 16 years old (average percentage of correct answers of 51) is lower than that for 14 years old (average percentage of correct answers of 53). In fact, as checked on the performance scale, the average for 14 years old is 41.38 points lower than that for 16 years old.

All the reasons stated and some of a more technical nature led to adoption in each one of the subjects of the so-called "performance scales". This methodology is based on the Item Response Theory and provides the results on a scale that overcomes many of the disadvantages of those based on the Classical Test Theory.

The performance scale shows the results for each subject that any student would have obtained if he had answered a theoretical test of 500 items. This test would be common for subjects 14 and 16 years old. It is very important to point out that it is a common scale for both age levels and thus may not be interpreted as a satisfactory level on reaching the midway point and the scale, that is 250, or considered insatisfactory if not reached, as the level of adequacy in each group is difficult. To value the results one must know what the students "know" and "know how to do".

It is evident that the greater the mark obtained by a subject in a field, the more tasks he will be able to solve satisfactorily. To give an idea of the types of capacities linked to the different levels on each scale, cut-off points have been established and the tasks defined which subjects who reach those levels are able to perform. In this case, those points are those for the marks 100, 150, 200, 250, 300, 350, 400 and 450. Assignment of meaning to the different points on the scale introduces an element of criteria to evaluation. Attention is no longer paid to the results of the group, but the study concentrates on the different level of achievement of the objectives involved in the process.

These points or levels do not represent judgement of the desirable knowledge and skills of the students. They are not performance criteria or standards. However, by knowing how many subjects pass a certain level and knowing what knowledge and capacities are linked to that level, one may obtain a clear virtue of the performance obtained by the students evaluated. This will allow comparison of what students "know" and "what they know how to do" and what is considered satisfactory performance in the educational system, that is to say, the criteria that represent the desirable knowledge and skills.

As the scale is common for 14 and 16 year olds, the desirable level for those aged 16 must necessarily be higher than one would expect at age 14. Setting that level is a matter of opinion, which will depend on the special circumstances of each group of students. Considering the cut-off points set for the different subjects, one may say than in any case, 14 year old students who do not reach the 200 level have a fully unsatisfactory performance, as what they "know" or "know how to do" is far below what should be expected of them. The same may be said of 16 year old students as to the 250 level. In other words, students aged 14 and 16 who do not pass the respective cut-off points of 200 and 250 clearly seem to have an unsatisfactory performance in the relevant area. However, this does not mean that those who are above these levels, even minimally, may be considered as students with a satisfactory performance. The Scientific Committee at the Institute considered that, once the capacities and skills shown on the performance scales for each subject are evaluated, the average values that would be desirable in a developed educational system, such as the case of the Spanish educational system, would rather have to be at the 250 level for students of 14 and at 300 for those of 16. In other words, only those above those two levels should be considered, according to the Scientific Committee, as students with an undoubtably satisfactory performance level.


Previous Table of Contents Next


© Instituto Nacional de Calidad y Evaluación (INCE)
calle San Fernando del Jarama, 14
E28002 MADRID
Tel: +34 91.745.9200
Fax: +34 91.745.9249
email: info@ince.mec.es