Comparing Schools

REPORTING AND COMPARING SCHOOL PERFORMANCES

Geoff N Masters
Glenn Rowley
John Ainley
Siek Toon Khoo

Paper prepared for the MCEETYA Expert Working Group to provide advice on national schools data collection and reporting for school evaluation, accountability and resource allocation

December 2008

Commissioned by the Reporting and Accountability Branch, National Education Systems
Group, Commonwealth Department of Education, Employment and
Workplace Relations (DEEWR)

i

Reporting and Comparing School Performances

ii

Reporting and Comparing School Performances

CONTENTS
Executive Summary
1. Introduction
1.1 Audiences and purposes
1.2 Forms of information
1.3 Nationally comparable data
1.4 Principles for reporting
1.5 Structure of paper

1
1
2
4
5
5

2. Student Outcomes
2.1 Value-based considerations
2.2 Pragmatic considerations
2.3 What is feasible?

8
8
10
11

3. Measuring Outcomes
3.1 Measurement scales
3.2 Measures of status
3.3 Measures of gain and growth
3.4 Measures of improvement

15
15
16
16
18

4. Physical and Human Resources
4.1 Finances
4.2 Facilities
4.3 Staff
4.4 Leadership

20
20
21
21
22

5. Student Intake Characteristics
5.1 Indigenous status
5.2 Socio-economic status
5.3 Language background other than English
5.4 Geo-location
5.5 Special educational needs

23
23
23
24
25
25

6. Evaluating Outcomes
6.1 Status, gain and growth
6.2 Performance against pre-specified ‘standards’
6.3 Improvement over time
6.4 Simple school comparisons
6.5 Like-school comparisons

26
26
26
26
27
27

7. Measuring School Performance
7.1 The intention to ‘measure’
7.2 Contextualised attainment measures
7.3 ‘Value-added’ measures

33
33
37
40

iii

Reporting and Comparing School Performances

8. Audiences and Purposes for Reporting
8.2 School principals
8.2 Employing authorities
8.3 School communities
8.4 The general public

44
44
45
45
46

9. Public Reporting
9.1 Reporting school data in tables
9.2 Providing school profiles
9.3 What data should be reported?

48
48
52
55

Bibliography

67

Appendix

70

iv

Reporting and Comparing School Performances

EXECUTIVE SUMMARY
This report provides advice on the collection and reporting of information about the performances of Australian schools. The focus is on the collection of nationally comparable data. Two purposes are envisaged: use by education authorities and governments to monitor school performances and, in particular, to identify schools that are performing unusually well or unusually poorly given their circumstances; and use by parents/caregivers and the public to make informed judgements about, and meaningful comparisons of, schools and their offerings.
Our advice is based on a review of recent Australian and international research and experience in reporting on the performances of schools. This is an area of educational practice in which there have been many recent developments, much debate and a growing body of relevant research.
Our work is framed by recent agreements of the Council of Australian Governments
(COAG), in particular, at its meeting on 29 November 2008:
C OAG agreed that the new Australian Curriculum, Assessment and
Reporting Authority will be supplied with the information necessary to enable it to publish relevant, nationally-comparable information on all schools to support accountability, school evaluation, collaborative policy development and resource allocation. The Authority will provide the public with information on each school in Australia that includes data on each school’s performance, including national testing results and school attainment rates, the indicators relevant to the needs of the student population and the school’s capacity including the numbers and qualifications of its teaching staff and its resources. The publication of this information will allow comparison of like schools (that is, schools with similar student populations across the nation) and comparison of a school with other schools in their local community.
(COAG Meeting Outcomes)

Our work also has been framed by the recently endorsed MCEETYA Principles for
Reporting Information on Schooling (see Section 1.4).
Before summarising our specific recommendations, there are some general conclusions that we have reached from our review of international research and experience. The specific recommendations that follow are best understood in the context of these general conclusions:
•

Vigilance is required to ensure that nationally comparable data on individual schools does not have the unintended consequence of focusing attention on some aspects of the purposes of schooling at the expense of other outcomes that are as important but not as easily measurable. Parents/caregivers and the public are interested in a broad range of information about schools, and nationally comparable data should be reported in the context of this broader information.

•

Although it has become popular in education systems in some other parts of the world to use statistical models to develop ‘measures’ of school performance and to report these measures publicly in league tables, we believe that there are very

v

Reporting and Comparing School Performances

sound technical and educational reasons why school measures of this kind should not be used for public reporting and school comparisons.
•

Related to this point, we are not convinced of the value of reporting ‘adjusted’ measures of student outcomes publicly. Measures of student outcomes should be reported without adjustment.

•

To enable the comparison of unadjusted student outcomes across schools, we believe that a ‘like-schools’ methodology should be used. This methodology would allow parents/caregivers, the public, and education systems to compare outcomes for schools in similar circumstances.

•

While point-in-time measures of student outcomes often are useful, it is difficult to establish the contributions that teachers and schools make to point-in-time outcomes. In general, measures of student gain/growth across the years of school provide a more useful basis for making judgements about the value that schools are adding.

•

Measures of gain/growth are most appropriately based on measurement scales that can be used to monitor student progress across the years of school. The NAPLAN measurement scales are an example and provide educational data superior to that available in most other countries. Consideration should be given to developing national measurement scales for early literacy learning and in some subjects of the national curriculum.

•

Initially reporting should build on the understandings that parents and the public have already developed. For example a school’s NAPLAN results should be reported in forms that are consistent with current NAPLAN reports for students.
Although much work needs to be done in defining the most appropriate measures, the principle should be to build on the representations of data that are already familiar to people.

Recommendations
Our report makes the following specific recommendations: student outcome measures
•

Nationally comparable data should be collected on the literacy and numeracy skills of students in each school, using NAPLAN (Years 3, 5, 7 and 9).

•

Nationally comparable data should be collected on the tertiary entrance results of students in each senior secondary school. These data could be reported as the percentage of students achieving tertiary entrance ranks of 60 or above, 70 or above, 80 or above, and 90 or above (calculated as a percentage of the students achieving tertiary entrance ranks).

•

Nationally comparable data should be collected on the percentage of students in each senior secondary school completing Year 12 or equivalent; the percentage of students applying to all forms of post-school education; and the percentage of students completing VET studies.

vi

Reporting and Comparing School Performances

•

Nationally comparable data should be collected on the achievements of students in core national curriculum subjects (English, mathematics, science and history), beginning in 2010. National assessments could be developed initially at Year 10.

•

Nationally comparable data should be collected on the early literacy learning of children in each primary school. These assessments will need to be developed and should be administered upon entry to school and used as a baseline for monitoring progress across the first few years of school.

physical and human resources
•

Nationally comparable data should be collected about sources and amounts of funding received by each school, including all income to the school from State and Commonwealth governments, as well as details of fees payable by parents, including those that are mandatory and any voluntary levies that parents are expected to pay.

•

Nationally comparable data should be collected on the numbers and qualifications of teaching staff in each school.
Basic data would include academic qualifications, details of pre-service teacher education, and details of any advanced certification (eg, Advanced Skills Teacher; Level 3 Teacher).

student intake characteristics
•

Nationally comparable data should be collected on the socio-economic backgrounds of students in each school. Data should be based on information collected at the individual student level, using at least parental occupation and, possibly, parental education levels, under the agreed MCEETYA definitions.

•

Nationally comparable data should be collected on the percentage of students in each school of Aboriginal and/or Torres Strait Islander background under the agreed MCEETYA definition.

•

Nationally comparable data should be collected on the percentage of students in each school identified as having a language background other than English
(LBOTE) under the agreed MCEETYA definition.

•

Nationally comparable data should be collected on the geo-location of each school using a 3-category scale: metropolitan, provincial, and remote.

•

Nationally comparable data should be collected on the percentage of students in each school with special educational needs. A nationally agreed definition of this category will need to be developed.

like-school comparisons
•

In reporting student outcome data for a school, data for like-schools should be provided as a point of comparison. Like-schools will be schools in similar circumstances and facing similar challenges.

•

In determining ‘like-schools’, account should be taken of the percentage of students with Indigenous backgrounds, the socio-economic backgrounds of the students in the school, and the percentage of students from language backgrounds other than English.

vii

Reporting and Comparing School Performances

•

For each school separately, like-schools should be identified as the schools most similar to that school on the above characteristics (rather than pre-defining a limited number of like-school categories).

•

Work should commence as soon as possible on the development of an appropriate like-schools methodology.

public reporting
•

For the purpose of providing public information about schools, a common national website should be used to provide parents/caregivers and the public with access to rich information about individual schools.

•

The national website should provide information about each school’s programs, philosophies, values and purposes, provided by the school itself, as well as nationally comparable data, provided centrally.

•

Nationally comparable student outcome data should, wherever possible, provide information about current levels of attainment (ie, status), gain/growth across the years of school, and improvement in a school over time.

•

The complete database for each state/territory should be made available to the relevant state/territory departments of education and other employing authorities, enabling them to interrogate data for their schools and to make judgments about school performances using aggregated data and national summary statistics.

We believe that almost all nationally comparable data collected centrally could be reported publicly. The exceptions would arise when the public reporting of data may have negative and unintended consequences for schools. For example, we can envisage negative consequences arising from the reporting of the socio-economic backgrounds of students in a school, or of the financial circumstances of struggling, small schools (both government and non-government). We also believe that data reported publicly should be factual data about a school, and not the results of secondary analyses and interpretations that are open to debate (eg, value-added measures). viii

Reporting and Comparing School Performances

1. INTRODUCTION
In education, good decision making is facilitated by access to relevant, reliable and timely information. Dependable information is required at all levels of educational decision making to identify areas of deficiency and special need, to monitor progress towards goals, to evaluate the effectiveness of special interventions and initiatives, and to make decisions in the best interests of individual learners.
The focus of this paper is on the provision and use of information about individual schools. The starting point is the observation that relevant and reliable information about schools is required by a range of decision makers – including parents and caregivers, school principals and school leadership teams, system managers and governments, and the general public – all of whom require dependable information that they can use to maximise opportunities and outcomes for students.
1.1 Audiences and Purposes
Parents and caregivers require valid and reliable information to evaluate the quality of the education their children are receiving, to make informed decisions in the best interests of individual students, and to become active partners in their children’s learning. They require dependable information about the progress individuals have made (the knowledge, skills and understandings developed through instruction), about teachers’ plans for future learning, and about what they can do to assist. There is also considerable evidence that parents and caregivers want information about how their children are performing in comparison with other children of the same age. And, if they are to make judgements about the quality of the education their children are receiving, they require information that enables meaningful comparisons across schools. School leaders require reliable information on student and school performances for effective school management.
Research into factors underpinning school effectiveness highlights the importance of the school leader’s role in establishing an environment in which student learning is accorded a central focus, and goals for improved performance are developed collaboratively by staff with a commitment to achieving them. School managers require dependable pictures of how well students in a school are performing, both with respect to school goals for improvement and with respect to past achievements and achievements in other, comparable schools.
Governments and system managers require dependable information on the performance and progress of individual schools if they are to exercise their responsibilities for the delivery of quality education to all students. Effective management depends on an ability to monitor system-wide and school performances over time, to gauge the effectiveness of special programs and targeted resource allocations, to monitor the impact of policies, and to evaluate the success of initiatives aimed at traditionally disadvantaged and underachieving sections of the student population. Accurate, reliable information allows system managers to measure progress against past performances, to identify schools and issues requiring special attention, to target resources appropriately, and to set goals for future improvement.

1

Reporting and Comparing School Performances

1.2 Forms of Information
Because there are multiple audiences and purposes for information about schools, the forms of information required for effective decision making are different for different stakeholders. Parents and caregivers require a wide range of information, including information relating to their immediate needs (eg, Is the school easily accessible by public transport? Does it have an after-school program? What fees and/or levies does it charge?); the ethos of the school (eg, What evidence is there of bullying/harassment?
What are the espoused values of the school? Do students wear uniforms? What level of discipline is imposed? Who is the principal?); their child’s likely educational experience (eg, Who will be my child’s teacher next year? Will they be in a composite class? How large will the class be? Does the school have a literacy intervention program? What extra-curricular activities are provided?); and the school’s educational results (eg, Does the school achieve outstanding Year 12 results?). School leaders require other forms of information, including information relating to staffing and resources (eg, What resources are available for music next year? How many beginning children have special learning needs?); the effectiveness of initiatives
(eg, Is there any evidence that the extra class time allocated to literacy this year made a difference?); and academic results (eg, How many Year 5 students did not meet the minimum performance standard in Reading? Have our results improved since last year? Are we still below the state average? How did last year’s Year 12 results compare with those of the neighbouring school? ).
System managers and governments require still other forms of information, including information to monitor system-wide trends over time, to evaluate the effectiveness of attempts to raise standards and close gaps, and to identify schools that are performing unusually well or unusually poorly given their circumstances. In general, the schoollevel information required by system managers and governments is less fine-grained than the information required by parents, teachers and school leaders.
Figure 1 displays schematically various forms of information that could be made available about a school, either publicly or to specific audiences (eg, system managers). The forms of evidence represented in Figure 1 are:
A:

student outcome measures that a school could choose to report
Most schools report a wide range of information about the achievements of their students to their school communities. This information is reported in school newsletters, local and community newspapers, school websites, and at school events. The information includes details of Year 12 results, analyses of postschool destinations, results in national mathematics and science competitions, language certificates, awards, prizes, extra-curricular achievements, community recognition, and so on. Most schools take every opportunity to celebrate the achievements of their students and to announce these achievements publicly.

2

Reporting and Comparing School Performances

Figure 1. Forms of information that could be made available about a school
B:

a sub-set of student outcome measures on which it is agreed to collect nationally comparable data
Within the set of student outcome information that might be reported for a school, there could be a sub-set of outcomes on which it was agreed to collect nationally comparable data. A reason for identifying such a sub-set would be to ensure some common measures to facilitate school comparisons – within a local geographical area, across an entire education system, nationally, or within a group of ‘like’ schools. Inevitably, nationally comparable data would be collected for only some of the outcomes that schools, parents and communities value. Performances on common literacy and numeracy tests in Years 3, 5, 7 and 9 are an example of nationally comparable data currently in this category.

C.

physical and human resources measures that a school could choose to report
Schools provide information in various forms and to various audiences about their physical and human resources. Information of this kind includes details of staff qualifications and teaching experience, staff turnover rates, school global budgets, computers and other technology, newly constructed facilities, bequests, results of fundraising drives, and so on. Some of this information may be reported to the school community; some may be kept confidential to the school, education system or government departments.

D:

a sub-set of physical and human resources measures on which it is agreed to collect nationally comparable data
Within the set of physical and human resources measures reported for a school, there could be a sub-set of measures on which it was agreed to collect nationally comparable data. For example, there have been recent calls for greater consistency and transparency in the reporting of school funding arrangements
(Dowling, 2007; 2008) and for more consistent national approaches to assessing and recognising teacher quality (Dinham, et al, 2008).

3

Reporting and Comparing School Performances

E.

student intake measures that a school could choose to report
Most schools have considerable information about their students. For example, they may have information about students’ language backgrounds, Indigenous status, socio-economic backgrounds, learning difficulties and disabilities. This information usually is reported only within education systems or to governments and is not reported publicly, although schools sometimes provide information to their communities about the range of languages spoken by students in the school, the countries from which they come, the percentage of Indigenous students in the school and the school’s special Indigenous programs, or the number of severely disabled students and the facilities and support provided for these students.

F:

a sub-set of student intake measures on which it is agreed to collect nationally comparable data
Within the set of student intake characteristics reported for a school, there could be a sub-set of measures on which it was agreed to collect nationally comparable data. Some progress has been made toward nationally consistent definitions and nationally consistent data collections on student background characteristics.

G.

all other information that a school could choose to make available
Beyond information about student outcomes, student backgrounds and their physical and human resources, schools provide a range of other information to the communities they serve.

1.3 Nationally Comparable Data
Acknowledging the many purposes and audiences for information about schools, and the various forms that this information can take, the specific focus of this paper is on the collection and reporting of nationally comparable data for the purposes of evaluating and comparing school performances. In other words, the focus is on categories B, D and F in Figure 1. We envisage three broad uses of such data:
•

use by parents and caregivers in judging the quality of educational provision and in making informed decisions in the best interests of individual students;

•

use by school leaders in monitoring a school’s improvement and benchmarking the school’s performance against other, comparable schools; and

•

use by education systems and governments in identifying schools that are performing unusually well or unusually poorly given their circumstances.

As noted above, these three stakeholder groups are likely to have different needs. The ways in which nationally comparable data are analysed, combined and reported may be different for different purposes.
We see the process of reaching agreement on the core data that should be available about a school as a national collaborative process, and see little value in arriving at different conclusions about these data for different parts of the country.

4

Reporting and Comparing School Performances

1.4 Principles for Reporting
The Principles for Reporting Information on Schooling (see pages 6-7) adopted by the
Ministerial Council for Education, Employment, Training and Youth Affairs
(MCCETYA) provide an important point of reference for any proposed collection and use of nationally comparable data on schools. These principles recognise the multiple audiences and purposes for information about schools, the need to collect broad evidence about student and school performances, and the desirability of monitoring intended and unintended consequences of reporting information on schools.
Australian governments have undertaken to ensure that data provided for the purposes of comparing schools are reliable and fair and take into account the contexts in which schools work. Governments also have undertaken not to develop simplistic league tables of school performances.
1.5 Structure of Paper
This paper first considers the kinds of nationally comparable data that might be collected about schools for the purposes outlined above. We draw on national and international research and experience, attempt to anticipate the likely requirements of different audiences, and take into account what measures currently exist and what additional measures might be desirable in the future. Each of the three data categories in Figure 1 is considered in turn:
•
•
•

student outcome measures physical and human resources measures student intake measures

(sections 2-3)
(section 4)
(section 5)

We then consider alternative ways of evaluating and comparing school performances.
Two broad methodologies are discussed:
•
•

the direct comparison of student outcomes the construction of measures of school performance

(section 6)
(section 7)

Finally, we consider issues in reporting publicly on the performances of schools:
•
•

audiences and purposes for reporting options for public reporting on schools

(section 8)
(section 9)

5

Reporting and Comparing School Performances

MCEETYA PRINCIPLES FOR REPORTING INFORMATION ON SCHOOLING
There is a vast amount of information on Australian schooling and individual schools. This includes information about the educational approach of schools, their enrolment profile, staffing, facilities and programs, and the education environment they offer, as well as information on the performance of students, schools and systems. Different groups, including schools and their students, parents and families, the community and governments, have different information needs. The following principles provide guidance on requirements for information on schooling, including the types of information that should be made readily available to each of the groups noted above. These principles will be supported by an agreed set of national protocols on the access to and use of information on schooling.
Good quality information on schooling is important:
FOR SCHOOLS AND THEIR STUDENTS
Principle 1: Schools need reliable, rich data on the performance of their students because they have the primary accountability for improving student outcomes.
Good quality data supports each school to improve outcomes for all of their students. It supports effective diagnosis of student progress and the design of quality learning programs.
It also informs schools’ approaches to provision of programs, school policies, pursuit and allocation of resources, relationships with parents and partnerships with community and business. Schools should have access to:
• Comprehensive data on the performance of their own students that uses a broad set of indicators • Data that enables each school to compare its own performance against all schools and with schools of similar characteristics
• Data demonstrating improvements of the school over time
• Data enabling the school to benchmark its own performance against that of the bestperforming schools in their jurisdiction and nationally
FOR PARENTS AND FAMILIES
Principle 2: Information about schooling, including data on the performance of individuals, schools and systems, helps parents and families to make informed choices and to engage with their children’s education and the school community.
Parents and families should have access to:
• Information about the philosophy and educational approach of schools, and their staffing, facilities, programs and extra-curricular activities that enables parents and families to compare the education environment offered by schools
• Information about a school’s enrolment profile, taking care not to use data on student
1
characteristics in a way that may stigmatise schools or undermine social inclusion
• Data on student outcomes that enables them to monitor the individual performance of their child, including what their child knows and is able to do and how this relates to what is expected for their age group, and how they can contribute to their child’s progress
• Information that allows them to assess a school’s performance overall and in improving student outcomes, including in relation to other schools with similar characteristics in their jurisdiction and nationally

1

Any use or publication of information relating to a school’s enrolment profile should ensure that the privacy of individual students is protected. For example, where the small size of a school population or of a specific student cohort may enable identification of individual students, publication of this information should be avoided.

6

Reporting and Comparing School Performances

FOR THE COMMUNITY
Principle 3: The community should have access to information that enables an understanding of the decisions taken by governments and the status and performance of schooling in Australia, to ensure schools are accountable for the results they achieve with the public funding they receive, and governments are accountable for the decisions they take.
Students are an important part of our society and take up a variety of roles within it after leaving school. The community is therefore a direct and indirect consumer of the product of our schools, as well as providing the means of public funding. Information about schools in the public domain fulfils the requirement that schools be accountable for the results they achieve with the public funding they receive, including relative to other ‘like’ schools; it should also give the community a broad picture of school performance and a sense of confidence in our school systems.
The community should have access to:
• Information about the philosophy and educational approach of schools, and their staffing, facilities, programs and extra-curricular activities that enables the community to compare the education environment offered by schools
• Information about individual schools’ enrolment profile, taking care not to use data on student characteristics in a way that may stigmatise schools or undermine social inclusion
• National reporting on the performance of all schools with data that allows them to view a school’s performance overall and in improving student outcomes, including in relation to other schools with similar characteristics
RESPONSIBLE PROVISION OF SCHOOLING INFORMATION
Australian Governments will ensure that school-based information is published responsibly so that: • any public comparisons of schools will be fair, contain accurate and verified data, contextual information and a range of indicators to provide a more reliable and complete view of performance (for example, information on income, student body characteristics, the spread of student outcomes and information on the value added by schools)
• governments will not devise simplistic league tables or rankings and will put in place strategies to manage the risk that third parties may seek to produce such tables or rankings, and will ensure that privacy will be protected
• reports providing information on schooling for parents and families and the community will be developed based on research on what these groups want to know and the most effective ways the information can be presented and communicated.
FOR GOVERNMENTS
Principle 4: Governments need sound information on school performance to support ongoing improvement for students, schools and systems. Government also need to monitor and evaluate the impacts (intended and unintended) of the use and release of this information to improve its application over time.
Good quality information on schooling enables governments to:
• analyse how well schools are performing
• identify schools with particular needs
• determine where resources are most needed to lift attainment
• identify best practice and innovation in high-performing schools that can be mainstreamed and used to support improvements in schools with poorer performance
• conduct national and international comparisons of approaches and performance
• develop a substantive evidence base on what works.
This will enable future improvements in school performance that support the achievement of the agreed education outcomes of both the Ministerial Council for Education, Employment,
Training and Youth Affairs and the Council of Australian Governments.

7

Reporting and Comparing School Performances

2. STUDENT OUTCOMES
Information about the outcomes of a school’s efforts is key information for parents and caregivers if they are to judge the quality of educational provision; for school leaders to monitor a school’s performance and improvement; and for education systems and governments to identify schools in need of additional support.
However, schools work to promote many different kinds of outcomes for their students. For some schools, an important objective is to improve school attendance rates. For others, assisting students to make successful transitions into the workforce is a high priority. Some schools are more focused than others on supporting the social, spiritual and emotional development of students. Still others measure their success in terms of entry rates into highly sought-after university courses.
Decisions about the outcomes to be reported publicly for schools are important because they influence judgements about how well individual schools are performing.
This is particularly true when education systems and governments attempt to construct
‘measures’ of school performance:
Perverse incentives can arise when the [school] performance measure has both a large impact upon actors and focuses on an aspect of schooling that does not reflect the true or overall purpose and objectives of schools. Unfortunately, this can be common in school performance measures if the performance measure is too narrowly defined.
(OECD, 2008, 26)

In practice, the outcomes for which nationally comparable data are collected and reported are likely to be determined by both value-based and pragmatic considerations. 2.1 Value-Based Considerations
Inevitably, decisions about outcomes reflect underlying values. For example, if proficiency in basic skills such as literacy and numeracy is considered the most important outcome of schooling because of its fundamental importance to further learning and life beyond school, then it might be considered adequate to measure and report levels of proficiency in these basic skills only. On the other hand, some secondary schools might argue that their performances should be judged not in terms of basic skills but in terms of tertiary entrance scores and successful admission to selective university courses. And religious community schools that place a priority on developing religious and cultural knowledge and values might argue that these outcomes should be taken into consideration in any judgement of their ‘performance’.
Decisions about student outcome measures need to acknowledge the values that they reflect and explicitly recognise that reported data provide information about only some of the outcomes that schools value.
The first question to be addressed about student outcomes is: Which outcomes are so essential that there would be interest in knowing how well every school in Australia was contributing to their achievement? Some outcomes may be clearly of this kind.
Others may require broad national discussion – by governments, schools and the wider community. Some possible candidates include:

8

Reporting and Comparing School Performances

•

literacy and numeracy skills
There is widespread recognition that skills in reading, writing and basic numeracy are fundamental to success in formal learning settings and to subsequent employment prospects and a range of post-school outcomes and experiences.
Students’ literacy levels are the best available predictor of secondary school completion. Nationally comparable data on literacy and numeracy are already available for all Australian students in Years 3, 5, 7 and 9 through the NAPLAN assessments. •

achievement in core subjects
Current work to develop a national curriculum in English, mathematics, science and history reflects consensus to give priority to these subjects and agreement on the desirability of national curriculum consistency in these areas. Australia’s participation in the IEA Trends in International Mathematics and Science Study
(TIMSS) and the OECD Programme for International Student Assessment (PISA) further underscores the national priority attached to student achievement in the core areas of English literacy, mathematics and science.

•

secondary school completion
Secondary school completion has direct implications not only for the long-term employment prospects of individuals, but also for the Australian economy.
Compared with people who complete Year 12 or equivalent, early school leavers tend to be less likely to work and tend to earn less when they are employed (The
Productivity Commission, 2006). The Business Council of Australia (2003) estimates that, if the percentage of young people completing Year 12 or its equivalent were increased from 80 per cent to 90 per cent, GDP would be $1.8 billion higher in 2020 than it would otherwise have been.

•

employability skills
A key function of schooling is to provide young people with skills and attributes for work and life beyond school. MCEETYA has endorsed the eight employability skills (communication, teamwork, problem solving, initiative and enterprise, planning and organising, self management, learning skills and technology skills) proposed by the Australian Chamber of Commerce and Industry and the Business
Council of Australia, and the Rudd Government’s Job Ready Certificate will recognise the achievement of these skills by students undertaking vocational education and training in schools and Australian Apprenticeships.

•

Year 12 results
For many students and parents, an important outcome of school education is the achievement of a senior secondary certificate and a tertiary entrance score that provides entry to a university course of choice. Many parents choose secondary schools on the basis of their past Year 12 results and the proportions of their students being admitted to competitive university courses.

There are many other outcomes that schools value and seek to achieve. In theory, there is no reason why nationally comparable measures could not be developed beyond those listed above, including in areas such as civics and citizenship, foreign

9

Reporting and Comparing School Performances

language proficiency, ICT literacy, and knowledge about and attitudes toward the protection of the environment.
Any attempt to evaluate a school’s performance on the basis of the outcomes it delivers for its students also must address the question of the stage of schooling at which those outcomes are assessed. For example, parents of pre-school children may be most interested in knowing how well a school addresses the needs of children in their first years of school. What progress do children make in the areas of oral language development, reading and writing in their first three years? How does this compare with typical progress in other schools? How well does the school cater for children with developmental lags and learning difficulties? How effectively does it support students with language backgrounds other than English? Questions of this kind may require measures of literacy development from the time children begin school, rather than commencing literacy assessments at Year 3.
Similarly, a school’s effectiveness in the core subjects of English, mathematics and science could be evaluated at different stages of schooling. A K-12 school that is highly effective in the upper primary years may be much less effective in the junior secondary years – information that is likely to be of interest to parents, but that would require evidence from both these stages of school.
2.2 Pragmatic Considerations
The assessment and reporting of student outcomes in a nationally comparable way also will be influenced by pragmatic considerations, especially the question of what measures are already available, how reliable and credible they are, and how easily and inexpensively additional measures could be obtained.
Literacy and numeracy achievements already are assessed in a nationally comparable way across all Australian schools through the National Assessment Program, Literacy and Numeracy (NAPLAN) at Years 3, 5, 7 and 9.
Tertiary admissions agencies throughout Australia have devised a common measure of overall success in the final stage of secondary schooling. In NSW and the ACT, this measure is called the University Admissions Index (UAI); in Victoria, the
Equivalent National Tertiary Entrance Rank (ENTER); and in South Australia, the
Northern Territory, Western Australia and Tasmania, the tertiary Entrance Rank
(TER). Queensland provides data to allow students’ Overall Positions to be transformed to a finer scale and converted to an Interstate Transfer Index. For purposes of tertiary admissions, the UAI, ENTER, TER and the Interstate Transfer
Index are treated by the selection agencies as equivalent measures (Victorian Tertiary
Admissions Centre, 2008). These indices are already treated as a common measure across the country by tertiary institutions, and often are seen by students with tertiary aspirations and their parents as the single most valued outcome at the end of secondary schooling. At the present time, these measures provide the only available common measure of achievement at senior secondary level.
Work has been undertaken to investigate ways of achieving nationally comparable evidence in relation to the eight employability skills (Matters & Curtis, 2008). These

10

Reporting and Comparing School Performances

may in the future be considered for inclusion in national reporting, but at this point could not be considered.
Other outcomes currently are assessed in ways that provide comparability only across schools within the same state. Examples include Year 12 subject results, achievement in core subjects (eg, through the NSW School Certificate), and secondary school completion rates. If national comparability is an objective, then the challenge in relation to these outcomes is to find ways of making this information comparable across states and territories (see Masters et al, 2006 and Matters & Masters, 2007).
Obvious requirements for the collection and reporting of nationally comparable measures of student outcomes are political will and the resources necessary for data collection. Education systems and schools must be convinced that the benefits of providing nationally comparable data outweigh the costs. This decision has already been made for literacy and numeracy outcomes at Years 3, 5, 7 and 9. Agreement to develop national curricula in English, mathematics, science and history may provide a first step towards nationally comparable outcome measures in these subjects. And the high degree of curriculum consistency across jurisdictions and broad community support for a more common approach to Year 12 certification provide a basis for developing more comparable measures of achievement in senior secondary subjects.
Internationally, most countries provide nationally comparable student outcome data for only a few aspects of the curriculum – usually in the core school subjects of first language, mathematics and science.
2.3 What is Feasible?
Table 1 summarises some currently available student outcome measures. Nationally comparable data are available in literacy and numeracy for all schools through
NAPLAN. National data also are available in the form of tertiary entrance scores and
Year 12 completion rates, although in both these cases, there are some questions about the extent of comparability across states and territories. Some other assessment programs provide nationally comparable data, but only for schools and students who choose, or are sampled, to participate in them. There are no other obvious nationally comparable outcome measures collected for all schools.
One relatively cost-effective way to provide better information about student outcomes would be to make wider use of existing assessment instruments. For example, information about the extent to which schools are addressing the needs of students with inadequate literacy and numeracy levels in the final years of secondary school could be provided by identifying students who perform at unacceptably low levels on NAPLAN in Year 9 and administering the Year 9 NAPLAN tests to them two years later when they are in Year 11. Assessments developed and used by one jurisdiction could be made available to schools in other jurisdictions (eg, NSW School
Certificate examinations). And assessments developed primarily for research purposes could be made available for broader use (eg, the Longitudinal Literacy and
Numeracy Survey (LLANS) and Indigenous LLANS materials could provide better information about how well schools are supporting student learning in the early years of school and Indigenous students with language backgrounds other than English).

11

Reporting and Comparing School Performances

Another option is to develop, over time, other nationally comparable measures of student outcomes. Possibilities include new assessments to track the early oracy and literacy skills of young children. These may be especially useful for students who currently perform well below the minimum Year 3 literacy standard (eg, children in remote Indigenous ESL schools). Other possibilities include nationally comparable assessments of achievement in core school subjects, both at senior secondary level and at earlier stages of schooling. Such assessments are likely to become more feasible with agreement on a national curriculum in these subjects.
Table 1. Current Student Outcome Measures
Student Outcomes

Current Data
Collection
Nationally comparable, annual data collection at
Years 3, 5, 7 and 9

Adequacy
Adequate for making direct comparisons across schools at these Year levels. No nationally comparable data for the early years of school. Literacy and numeracy skills Inadequate for some remote Indigenous schools where English is not the first language.
Provides no information on the numbers of students leaving school with inadequate basic skills.
Annual Year 12 assessments in these subjects Adequate for making comparisons within a state
(for students who choose).
Research has shown a high degree of curriculum consistency across jurisdictions (in some subjects) and a high degree of consistency in the criteria used to assess student achievement.

NSW School Certificate
(Year 10 assessments)

Adequate for making comparisons within NSW.

Achievement in core subjects 12

Reporting and Comparing School Performances

Student Outcomes

Current Data
Collection

Adequacy

Sample surveys (eg, WA
Monitoring Standards in
Education; TIMSS)

Inadequate because based on sampled students only.

Data on secondary school Adequate for making completion rates comparisons across schools within a state.
Secondary school completion Differences in completion requirements make comparison across states more problematic.
No nationally comparable assessments.

Inadequate.
Work has been undertaken to develop and explore ways of assessing these skills in a nationally comparable way.

No nationally comparable assessments at the level of Year 11-12 subjects. Inadequate.

UAI, TER, ENTER and
OP scores in all jurisdictions. Adequate?
Although these indices are constructed to be comparable across states, they do not use a strictly common measure.

Employability skills

Year 12 results

Reflections
The best available nationally comparable student outcome measures at the present time are provided by NAPLAN. These data became available for all Australian schools for the first time in 2008. Students tested in 2008 will be retested in 2010, meaning that it will not be possible to assess the progress those students have made until that time. In the meantime, NAPLAN data from 2008 and 2009 could be used in school evaluations.
It may be possible to treat school completion rates as comparable across Australia at the present time, even though what it means to complete Year 12 or equivalent is not currently defined consistently. Student tertiary entrance ranks also could be treated as comparable across states (as they already are for university entrance purposes).

13

Reporting and Comparing School Performances

School attendance is an interesting issue because, on the one hand, it can be seen as an outcome of schooling, with schools working to increase student attendance rates. On the other hand, high rates of absenteeism can be seen as part of the contextual difficulties confronting some schools. In these contexts, schools, parents and local communities need to collaborate to improve levels of school attendance.
For the purposes of future school evaluation, we believe it would be useful to investigate the development of nationally comparable measures of early literacy skills, including for students from non-English backgrounds, and the collection of better information about literacy and numeracy outcomes post-Year 9 (especially for students with inadequate skills at Year 9).
We also believe there is value in working towards more nationally consistent measures of achievement in core national curriculum subjects, particularly at or near the end of Year 10. For national reporting, such measures would provide information, presently lacking, about the outcomes of middle schooling beyond literacy and numeracy. 14

Reporting and Comparing School Performances

3. MEASURING OUTCOMES
Having considered the kinds of outcomes that could be assessed and reported in a nationally comparable way, we turn now to the question of how best to assess and report student achievement.
3.1 Measurement Scales
An important general concept in the assessment and reporting of achievement is the concept of a measurement scale. A measurement scale can be conceptualised as a continuum of increasing achievement. At any given time, a student can be thought of as being at a particular location on this continuum (eg, a particular level of reading proficiency). The purpose of assessment is to estimate each student’s current level of attainment from observations of their work and performances.
The assessment process assumes that every student is at some point in their learning and so can be assessed on a continuum of increasing achievement. It is further assumed that every student is capable of making progress beyond their current level of attainment given appropriate learning opportunities and if engaged and motivated to learn. A characteristic of a measurement scale is that it usually is not limited to a particular year of school, but extends across several years, enabling student progress to be monitored independently of their age or year level.
Another important characteristic of a measurement scale is that it is not tied to any particular assessment instrument, but is freed of the specifics of the instruments used.
This is achieved by (statistically) calibrating the difficulties of test questions, enabling performances on different tests to be reported and compared on the same (numerical) scale. If a Year 3 reading test and a Year 5 reading test are calibrated on the same measurement scale, then a good performance on the Year 3 test may lead to exactly the same scale score as an average performance on the Year 5 test.
Results of the National Assessment Program-Literacy and Numeracy (NAPLAN) are reported on measurement scales with these properties. NAPLAN reports literacy and numeracy achievements on five measurements scales:
• Reading
• Writing
• Numeracy
• Spelling
• Grammar & Punctuation
All students in Years 3, 5, 7 and 9 are assessed on these scales, with each student receiving a score between 0 and 1000 on each scale.
NAPLAN scores maintain their meaning over time. For example, a NAPLAN
Reading score of 345 will represent the same level of reading proficiency in 2014 as it represented in 2011. Because of this property of NAPLAN scores, it is possible to measure a student’s achievement and growth (or to measure average achievement or growth for a group of students) across Years 3 to 9.

15

Reporting and Comparing School Performances

The NAPLAN measurement scales also are accompanied by descriptions of the kinds of performances and skills typical of students at various levels of proficiency (ten
‘bands’) along each scale. And for each year level, and for each aspect of literacy and numeracy, a national minimum standard is defined and located on the NAPLAN scale. For Year 3, Band 2 is the national minimum standard; for Year 5, Band 4; Year
7, Band 5; and Year 9, Band 6.
3.2 Measures of Status
A measure of status is a measure of attainment or standing at a particular time.
For an individual student, an example of a status measure would be:
• the student’s Year 7 NAPLAN Numeracy score
For a school, examples of status measures would be:
• the mean Year 5 NAPLAN Numeracy score in a given year; and
• the percentage of Year 3 students achieving the national minimum standard in
Reading.
Measures of status can take different forms, providing answers to different questions.
For example, in a given school in a given year, the Year 9 mean score, the Year 9 standard deviation, and the percentage of Year 9 students achieving the national minimum standard are all measures of status, but they provide information about different aspects of the performances of that school’s Year 9 students.
3.3 Measures of Gain and Growth
A measure of gain is a measure of how much progress a student has made on a measurement scale from one occasion to another (or how much progress, on average, a group of students has made from one occasion to another).
For an individual student, an example of a measure of ‘gain’ would be:
• the change in a student’s NAPLAN score from Year 3 to Year 5.
For a school, an example of a measure of gain would be:
• the change in a cohort’s mean NAPLAN Reading score between Year 3 (in 2010) and Year 5 (in 2012).
Although Cronbach and Furby (1970) raised concerns about the reliability of simple measures of gain, more recent research (Zimmerman and Williams, 1982; Rogosa and
Willett, 1983; Williams and Zimmerman, 1996) has shown that measures of gain can be more reliable and more useful than was once believed (Haertel, 2006).
A measure of growth is based on measures of status on three or more occasions, obtained either by averaging two or more ‘gains’ or by modelling growth (curve fitting). In recent years there has been growing international interest in the measurement of how much progress students are making across the years of school, and of the possibility of using these measures to make judgements of school effectiveness:

16

Reporting and Comparing School Performances

A measure of the effectiveness of a school should reflect the gains in achievement over a period of time, not just where the students finish.
(Rowley, 2006)
Growth models assume that student performance, and by extension school performance, is not simply a matter of where the school is at any single point in time, and a school’s ability to facilitate academic progress is a better indicator of its performance.
(Goldschmidt, et al, 2005)

Goldschmidt, et al, (2008) argue that teachers are likely to be more willing to be held accountable for the progress students make than for students’ current levels of attainment which will depend on many factors outside their control (including prior teaching). Measures of gain and measures of growth require the collection of longitudinal data – that is, the same students must be assessed on different occasions. This in turn requires a system for tracking students as they move from year to year and, possibly, from school to school. The issue of unique student identifiers currently is being addressed by the Australian states and territories. Examples include the planned
Victorian Academic Number (VAN) and the planned NSW Student Administration and Learning Management system which will use the Enrolment Registration Number to track student progress over time and across schools. As noted by the NSW
Auditor-General (2008), this system ‘should allow parents and teachers to follow student progress throughout their schooling and enable the Department to systematically ‘case-manage’ students with learning difficulties’.
In the United States, the National Centre for Educational Achievement (2008) has observed that most US states currently rely on ‘snapshot’ (status) data based on aggregated data at a point in time. Through their Data Quality Campaign, the NCEA advocates greater use of longitudinal data to enable schools and parents to track the academic progress of individual students over time and to answer such questions as:
‘Which schools have been most successful in improving the success of students who entered the school with poor reading skills’. In their view, ‘leaders at all levels of school systems need to demand, understand and use longitudinal data to improve instruction and management’.
Few countries have developed measurement scales along which gain and growth can be measured for all students. Few have a system of ‘vertically equated’ tests administered at different year levels, but calibrated on the same measurement scale.
Some countries (such as England) have assessments at multiple time points, but their assessments are not vertically equated and so are of limited value for measuring gain and/or growth across the years of school. The NAPLAN measurement scales enable status, gain and growth to be measured across Years 3, 5, 7 and 9 and, in this sense, represent world’s best practice in the measurement of student progress.
NAPLAN provides a basis for evaluating each primary school’s effectiveness in promoting literacy and numeracy gains between Year 3 and Year 5. In those states in which Year 7 is in the primary school years, NAPLAN also provides a basis for evaluating each primary school’s effectiveness in promoting gains between Year 5 and Year 7. In other states, gains from Year 5 to Year 7 occur across the primary17

Reporting and Comparing School Performances

secondary transition and so are more difficult to attribute to a single school (except in
K-10 or K-12 schools). In most Australian states and territories, NAPLAN provides a basis for evaluating the effectiveness of secondary schools in promoting literacy and numeracy gains between Year 7 and Year 9.
3.4 Measures of Improvement
A measure of improvement is a measure of how much change has occurred in a school over time. For example, how much better or worse is the performance of this year’s
Year 3 cohort than the performance of last year’s Year 3 cohort?
Improvement can occur either in measures of status, eg,
• an increase in the Year 7 NAPLAN Numeracy mean score between 2011 and
2014
or in measures of gain/growth, eg,
• an increase in average Year 3-7 Reading growth over a ten-year period.
Table 2, modified from Flicek (2004) summarises some of the above discussion and makes the point that improvement can occur either in measures of status or in measures of gain/growth.
Table 2. Measures of Status, Gain/Growth and Improvement
Status

Gain/Growth

Single Cohort
Q1
Measure of cohort achievement on a single occasion (eg, mean Year 5 Reading score in 2010)
Q3
Progress a cohort makes across the years of school
(eg, average growth in
Reading between Year 3 and
Year 7)

Improvement (Across Cohorts)
Q2
Improvement/decline in achievement over time
(eg, increase in mean Year 5
Reading score in the period 2010 to 2014)
Q4
Improvement/decline in amount of growth (eg, improvement in average Year
3-7 Reading growth in the period
2010 to 2014)

As Flicek (2004) notes, Q1 measures are an instance of Campbell and Stanley’s
(1963) ‘one-shot case study’, and in the absence of any point of reference are of
‘almost no scientific value’ for evaluating a school’s performance. High achievement levels are, of course, important, but they alone do not provide a measure of school performance. Measures of improvement in status (Q2) provide information about changes in a school’s performance over time, but must be interpreted with caution because they are based on different cohorts of students. It often will not be clear whether increases or decreases over time are due to changes in the quality of educational provision, changes in the cohort of students, or some combination of these:

18

Reporting and Comparing School Performances

Experience has shown that fluctuations in student cohort from one year to the next are large enough to swamp the effect of any improved teaching that may be occurring. Therefore, while improved results are and should remain a key motivation in any school, they provide an unreliable indicator of improvements in school effectiveness.
(Rowley, 2006)
Shortly after No Child Left Behind was enacted, concerns were expressed that determining Adequate Yearly Progress (AYP) by comparing snapshot [status] data for one cohort to a different cohort of students may not be the most accurate means of judging the performance of a school or district.
(National Centre for Educational Achievement, 2008)

Nevertheless, when long term trends show increases or declines in test scores and demographic results suggest stability of student characteristics, the conclusion that instruction is contributing to the test score trend becomes increasingly reasonable
(Flicek, 2004).
Measures of how much a cohort of students has progressed across the years of school
(Q3) can be simple measures of gain from one occasion to another or measures of growth based on trajectories over three or more time points. In general, growth trajectories are more informative and reliable than simple measures of gain. Teddlie, et al, (2000) and Willms (1992) have concluded that measures of growth in student achievement provide the most effective basis for measuring a school’s effectiveness
(ie, the ‘value’ the school adds).
Following concerns in the United States about the limitations of using improvement in status (Q2) as an indicator of school performance, as at January 2008, the US
Secretary of State had approved the use of growth-based measures (Q3) in lieu of
Adequate Yearly Progress (AYP) calculations in eight US states.
It also is possible to track improvements over time in the amount of student growth occurring in a school (Q4). Increases in growth suggest that a school is becoming more effective in promoting student learning. However, once again, caution must be exercised in drawing this inference because different student cohorts are involved.
Reflections
Measures of student gain/growth provide much more direct and useful information about student learning – and thus about the contribution that schools are making to student learning – than snapshot measures of status. However, reliable measures of gain/growth depend on the availability of well-constructed measurement scales that can be used to monitor student progress across the years of school.
The NAPLAN scales on which students’ literacy and numeracy results will be reported, beginning in 2008, use the same general methodology as PISA and as has been used in all Australian state and territory literacy and numeracy programs since the introduction of the NSW Basic Skills Tests in 1989 (Masters, et al, 1990).
Evaluations of school effectiveness in promoting student progress in other areas of learning (such as early literacy learning and progress across the years in national curriculum subjects) would benefit from similarly well-constructed measurement scales. 19

Reporting and Comparing School Performances

4. PHYSICAL AND HUMAN RESOURCES
Parents, caregivers and the wider community also have an interest in the quality of the physical and human resources of schools. School buildings and facilities, the financial resources of the school, the quality of its teaching and support staff, and the quality of the senior leadership team all shape judgements about a school and the kind of educational experience it is likely to provide for students. Information of this kind often can be found on school websites and in schools’ annual reports, but also is gleaned from visits to the school (eg, on open days) and through discussions with other parents.
Education systems and governments, too, have an interest in the physical and human resources of schools for the purposes of identifying appropriate interventions (eg, to demolish existing buildings and commence a new building program; to replace the school principal; to upgrade technology in the school). Beyond this, information about a school’s physical and human resources might be useful in identifying schools that are performing unusually well or unusually poorly given their resources. In other words, information of this kind might be useful contextual information in evaluating school performances.
4.1 Finances
One piece of information that could be useful to parents and governments is information about the financial resources of a school. However, as a number of researchers have observed recently, the provision of information about the quantum of resources available to individual schools from all sources is a radical proposal at the present time (Angus, 2007). Not only does this information not exist uniformly, but some states are incapable of reporting such information at the school level. In his report Australia’s School Funding System Dowling (2007) observes:
School funding, which is the area of education that should be most amenable to quantification and measurement, is plagued by inconsistency. Arguably, the lack of consistency and transparency in this area has a broader impact, as all other aspects of education are dependent on the primary issue of funding. It is theoretically possible to measure and report school resourcing in a clear and logical fashion, yet it remains resistant to greater comparability, transparency, and accountability.
(Dowling, 2007)

Reliable information about the financial resources of individual schools – whether provided as contextual information for parents and caregivers, or as information that could be used in the evaluation of schools’ performances by education systems and governments – first would require agreement across jurisdictions and education sectors to adopt more consistent and comparable approaches to reporting on school finances. We believe it is feasible to provide details of fees payable by parents, including those that are mandatory and any voluntary levies that parents are expected to pay.

20

Reporting and Comparing School Performances

4.2 Facilities
In general, a school’s facilities also are likely to be a determinant of its ability to deliver high outcomes for its students. Schools able to provide ready access to information and communication technologies throughout the school, well-equipped science laboratories, a well-resourced library, careers centre, well-equipped gymnasium and performing arts centre, sports grounds, and technology and vocational training centres may find it easier to deliver high outcomes for their students than schools lacking many of these facilities. For parents and caregivers, a school’s facilities usually will be an important consideration in judging the quality of the educational experience the school provides. For education systems and governments, a question is whether the facilities available to a school should be taken into consideration in making a judgement about the school’s ‘performance’ in delivering outcomes for its students.
4.3 Staff
There is now overwhelming research evidence that the single most important factor influencing student outcomes is the quality of the teaching to which students are exposed. In their report Teaching Talent: The Best Teachers for Australia’s
Classrooms written for the Business Council of Australia, Dinham, et al, (2008) observe: Until the mid-1960s it was widely believed that schools and teachers made little difference to student achievement, which was largely determined by heredity, family background and socioeconomic context. There is now considerable international evidence that the major in-school influence on student achievement is the quality of the classroom teacher. However, research evidence is also clear on a related matter: teacher quality varies considerably within schools and across schools. (Dinham, et al, 2008)

This is consistent with the conclusion reached a year earlier by McKinsey (2007):
The quality of an education system cannot exceed the quality of its teachers. The top-performing school systems [internationally] attract more able people into the teaching profession, leading to better student outcomes.
(McKinsey, 2007)

Research suggests that the most important determinant of the quality of educational provision in a school is the quality of the teaching occurring in its classrooms. This is more important than the school’s wealth, the quality of its buildings and facilities, or any other aspect of the school. At the present time, there is limited information available about the quality of teaching in Australian schools. Research suggests that years of teaching experience, levels of remuneration and formal university qualifications have limited value as guides to the quality of teaching occurring in classrooms. Schemes such as the Advanced Skills Teacher in South Australia and the
Level 3 teacher in Western Australia attempt to differentiate levels of teaching competence, and national work currently underway to develop standards for accomplished teachers also may provide better public information about the capacities of school staff.

21

Reporting and Comparing School Performances

4.4 Leadership
The leadership of a school is a key factor in a school’s capacity and performance.
Leithwood (2004) argues that effective school leadership is second only to classroom teaching in its potential influence on student outcomes:
Recent research suggests that successful leadership can play a highly significant – and frequently underestimated – role in improving student learning. Specifically, the available evidence about the size and nature of the effects of successful leadership on student learning justifies two important claims: that leadership is second only to classroom instruction among all school-related factors that contribute to what students learn at school, and secondly that leadership effects are usually largest where and when they are needed the most. These results, therefore, point to the value of changing, or adding to, the leadership capacities of underperforming schools as part of their improvement efforts or as part of school reconstitution. (Leithwood, 2004)

Highly effective school leaders set directions for a school (accounting for the greatest proportion of a leader’s impact), including establishing a shared vision, motivating the school community to pursue that vision, monitoring the school’s performance, and promoting effective communication. Effective leaders also are strongly focused on developing and mentoring staff and redesigning school structures and practices to maximise student, staff and school performance.
Direct and comparable indicators of the quality of school leadership do not currently exist. Judgements about the quality of a school leader or school leadership team usually are based on observations of a school’s achievements and performance over time. And research suggests that the kinds of behaviours required of school leaders depend on the circumstances of the school (eg, the leadership behaviours required to turn around a struggling school tend to be different from the leadership required to build on to outstanding successes).
Reflections
Our view is that there would be value in having more nationally comparable information about each of these aspects of a school’s resources/capacities: school finances, school facilities, teaching staff, and school leadership. More transparent and consistent data on school funding arrangements and school facilities should be useful to the public and to governments in comparing schools and their performances. The development of national standards for accomplished teaching and school leadership and systems for certifying teachers and leaders who meet these standards also could provide useful information about the human resource capacities of schools.
While information of this kind might provide a useful backdrop in understanding a school’s current performance, we believe it would be a mistake to use a school’s limited physical and/or human resources to justify its low outcomes (or worse, to
‘adjust’ a school’s outcomes to take account of its limited physical and human resources). The development of high quality physical and human resources should be seen as an objective of schools and an indicator of school success.

22

Reporting and Comparing School Performances

5. STUDENT INTAKE CHARACTERISTICS
Other important information about a school relates to its student intake – the kinds of students who attend the school. This may be useful information for parents and caregivers in evaluating a school and its achievements. It also is important information for education systems and governments in evaluating how well a school is performing. Judgements about a school’s performance must take into consideration challenges arising from the school’s student population: for example, the percentage of students who live in poor social and economic circumstances, are newly arrived in
Australia, who come from language backgrounds other than English, have special educational needs, and so on.
There are many different kinds of information that could be collected and reported about the students in a school. Some of these characteristics are known from
Australian and international research to be more highly correlated with student achievement than others. In making judgements about a school’s performance, it is especially important that characteristics that are known to be correlated with student outcomes are taken into consideration. The most significant of these are considered below. 5.1 Indigenous Status
The MCEETYA Data Implementation Manual (2008) provides the following definition of Indigenous Status:
A student is considered to be ‘Indigenous’ if he or she identifies as being of
Aboriginal and/or Torres Strait Islander origin. The term 'origin' is considered to relate to people's Australian Aboriginal or Torres Strait Islander descent and for some, but not all, their cultural identity.

Because of the relatively small percentage of Indigenous students in the Australian student population, Indigenous status does not explain much of the variance in student outcomes in national and international studies of student achievement. However, in a different sense, Indigenous status has the largest effect on, and is the best predictor of, student attainment. In the OECD Programme for International Student Assessment
(PISA), the difference between Indigenous and non-Indigenous literacy levels was well in excess of 80 points on the PISA scale, for which the OECD standard deviation is 100 points (Thomson & Bortoli, 2008). This suggests that Indigenous status will be an important factor in understanding the performances of schools with significant percentages of Indigenous students.
The most appropriate way to measure this variable will be to calculate the percentage of students in a school identified as Aboriginal and/or Torres Strait Islander under the agreed MCEETYA definition.
5.2 Socio-Economic Status
Research consistently shows a correlation between students’ socio-economic backgrounds and their levels of school attainment. For this reason, the socioeconomic backgrounds of a school’s student intake also must be taken into

23

Reporting and Comparing School Performances

consideration in any evaluation of the school’s performance. In PISA 2006, the literacy levels of Australian students from the highest SES quartile were in excess of
80 points higher than the literacy levels of students in the lowest SES quartile. This difference is consistent with results from a range of other studies such as those reviewed by Sirin (2005).
The socio-economic backgrounds of students in a school can be measured either at the level of the school (eg, using data from the ABS census collection districts for the home addresses of the students attending the school) or by aggregating information about the SES backgrounds of individual students in the school. Of these, the latter is the preferred approach. However, there are difficulties in collecting nationally comparable SES data in this way at the current time.
Although the MCEETYA Data Implementation Manual (2008) provides a basis for collecting nationally comparable data on Parental Occupation Group, Parental School
Education and Parental Non-School Education, current definitions of, and approaches to collecting and computing SES data vary across jurisdictions. There are also problems with varied and incomplete response rates when attempts have been made to collect these data from parents. Students usually are able to provide information about parental occupation (which can be classified into occupational groups and is probably adequate as a basis for a nationally comparable measure of SES), but are often not able to provide data on parental education levels.
A measure of the socioeconomic context of a school could be calculated as a mean on a scale or as a percentage of students in a group defined as the bottom fifth of the national distribution or as a predefined group such as Group 4 specified in the
MCEETYA Data Implementation Manual (2008).
5.3 Language Background Other than English
Research also shows a correlation between students’ achievement levels and language backgrounds other than English (LBOTE), although the correlation is much weaker than for Indigenous Status and Socio-Economic Status. In PISA 2006, students from
English-speaking backgrounds performed, on average, 20 points higher on the PISA scale than students from language backgrounds other than English (Thomson &
Bortoli, 2008). This suggests that the language backgrounds of students in a school also should be taken into consideration in any evaluation of school performance.
The MCEETYA Data Implementation Manual (2008) provides guidelines for the collection of nationally comparable data on Main Language Other than English
Spoken at Home. The guidelines provide for the collection of information about the language spoken at home by students as well as by parents and caregivers: ‘if the student or father/guardian1 or mother/guardian2 speaks a language other than English at home, the derived language background indicator code will be LBOTE’.
The most appropriate way to measure this variable will be to calculate the percentage of students in a school identified as LBOTE under the agreed MCEETYA definition.

24

Reporting and Comparing School Performances

5.4 Geo-Location
In national surveys, students in metropolitan areas of Australia consistently perform at higher levels, on average, than students in provincial towns and cities, who consistently perform, on average, above students in rural and remote parts of the country. In PISA 2006, students in metropolitan areas performed about 20 points higher on the PISA scale than students in provincial areas, and about 50 points higher than students in remote areas. For this reason, a school’s location also needs to be taken into consideration in any evaluation of the outcomes it delivers for its students.
Geographic location could be based on the MCEETYA Geographical Location
Classification (Jones, 2004). This classification is related to the ARIA classification developed by the Australian Bureau of Statistics. It provides a structure for classifying locations in three zones (metropolitan, provincial, and remote) that together encompass a more detailed structure of five categories and eight subcategories. The classification can be based on the permanent home address of students and then aggregated to school level or on the location of the school. Our recommendation is that Geo-Location be measured on a 3-category scale based on the location of the school: Metropolitan; Provincial; and Remote. However, a more finegrained classification could be considered.
5.5 Special Educational Needs
The number of students with special educational needs is relatively small as a percentage of the total student population. However, in schools with large percentages of students with special needs, this variable may need to be taken into account in understanding average levels of school attainment. The appropriate measure would be the percentage of students in the school with identified Special
Educational Needs based on a nationally agreed definition.
Reflections
There are significant correlations between some student characteristics and school attainment. Indigenous students, students from lower socio-economic backgrounds, from language backgrounds other than English, and from rural and remote parts of
Australia perform at lower levels, on average, than other students. These factors need to be taken into consideration when comparing the outcomes achieved by different schools. Care also is required in comparing outcomes for schools with significant proportions of special educational needs students.
Progress has been made in reaching national agreement on definitions and data collection processes for some of these student characteristics. Further work is required, especially in relation to socio-economic status and the definition of categories of special educational need, to ensure nationally comparable measures of school intake characteristics. There appear to be variations among jurisdiction in definitions at present that will need to be resolved. In addition, some students with special educational needs are exempted from participation in the National Assessment
Program and so the use of National Assessment Program data might not provide an accurate indication of the percentage of students with special educational needs.

25

Reporting and Comparing School Performances

6. EVALUATING OUTCOMES
So far in this paper we have considered several kinds of information about schools
(information about student outcomes, physical and human resources, and student intake characteristics) that might be collected and reported in a nationally comparable way, and we have reviewed some options for collecting and reporting such information. We turn now to the matter of evaluating the outcomes being achieved by a school. What can be done to assist stakeholders to make judgements about the outcomes being achieved? What outcomes is it reasonable to expect? Are the outcomes being delivered adequate?
6.1 Status, Gain and Growth
Student outcome measures – whether expressed as measures of status, gain or growth
– do not, in isolation, provide an adequate basis for evaluating the outcomes being achieved by a school. In general, evaluation depends on an external frame of reference. 6.2 Performance against Pre-Specified ‘Standards’
One external frame of reference is a pre-specified ‘standard’. The minimum national proficiency standard in Reading for Year 3 students would be an example. This specifies the level of Reading expected of all students in Year 3. When a standard of this kind is available, the percentage of students in a school achieving the standard can be reported and used in evaluating how well the school is performing. performance standards
Performance standards – sometimes also called achievement standards – set expectations (or targets) for status measures. For example, the US No Child Left
Behind legislation (2001) required the setting of minimum performance standards in mathematics and reading. Every school in the US is expected to have 100 per cent of its students achieving these performance standards by 2013-14. However, as the
OECD (2008) notes, while setting targets against performance standards is appealing because it is easily understood, it does not necessarily provide an appropriate improvement target for all students. School targets based on the achievement of minimum standards ‘provide little incentive for schools to meet the instructional needs of students already above [the performance standard] or those who are far below it’ (Flicek, 2004, 8). growth standards
Growth standards set expectations (or targets) for gain/growth. For example, if an education system specifies the amount of progress in Reading expected of students between Year 3 and Year 5, then the average progress made between those two grades by students in a school can be compared with this expectation and used in evaluating the school’s performance.

26

Reporting and Comparing School Performances

6.3 Improvement over Time
Another frame of reference for evaluating the outcomes being achieved by a school is the school’s past performance. Are the outcomes being achieved by the school now better than the outcomes it was achieving five years ago? In other words, have outcomes improved? As noted in section 3.4, improvement over time can occur in a school’s status measures (eg, Year 5 mean; percentage of students meeting the minimum national standard) or in measures of gain/growth in a school. In the United
States, schools are expected to demonstrate ongoing improvement in the percentage of students achieving the minimum performance standard. Schools that do not make
Adequate Yearly Progress (AYP) for two years in a row are identified as ‘schools in need of improvement’ and are subject to immediate interventions by the State
Education Agency in their state.
6.4 Simple School Comparisons
Another obvious frame of reference for evaluating the outcomes being achieved by a school is performance in other schools. Measures of status, growth and improvement can be compared from one school to another. For example,
•
•
•

Was the Year 9 Numeracy mean score higher in School X or in School Y?
Was the average Reading gain between Year 3 and Year 5 greater for School X or for School Y?
Was improvement in the Year 7 Reading mean score between 2011 and 2015 greater for School X or for School Y?

Simple comparisons of student outcomes can be made in this way between any two schools. The outcomes achieved in a school also can be compared with state or national averages.
However, comparisons of this kind take no account of the different circumstances and challenges faced by different schools. In consultations conducted as part of the development of the Victorian Department’s Blueprint, Downes and Vindurampulle
(2007) report a widely held view among school staff that it is inappropriate to make simple comparisons of outcomes for schools in very different circumstances.
6.5 Like-School Comparisons like school categories
One way of addressing this concern is to facilitate the comparison of outcomes across schools in similar circumstances (so-called ‘like’ schools). In this way, an attempt is made to take account of differences in school circumstances by comparing ‘like with like’. Across ‘like’ schools, comparisons can be made of measures of status, gain/growth or improvement over time.
Clearly, like-school comparisons require a prior decision about the contextual/ circumstantial features of schools that are to be used in identifying ‘like’ schools.
Once these have been decided, there are then different approaches to defining ‘like’ schools. One approach is to establish groupings of schools with similar characteristics and to assign each school to one of these pre-defined groups. A second approach is to

27

Reporting and Comparing School Performances

identify, for each school, the schools that are most similar to that school in their characteristics. This second approach leads to a much larger number of like-school groups but has the advantage of ensuring that each school is compared with the schools most similar to it. A third approach would be a 2-stage process that combines these first two approaches in some way.
Because the circumstances under which schools work vary so widely across Australia, a challenge under any of these approaches is to ensure that outcomes in a school are compared with outcomes in schools in similar circumstances. For example, remote
Indigenous schools with high proportions of students who do speak English as their first language perhaps should be compared only with similarly remote schools working in similar circumstances. The number of such schools may be relatively small. It is usual to define like-schools in terms of characteristics (eg, socio-economic backgrounds of the student population; percentage of students from non-English speaking backgrounds) that have been shown to be correlated with student outcomes.
In Victoria, for example, two student background characteristics have been used to define like-schools:
• LBOTE: the proportion of students in a school for whom a language other than
English is spoken at home; and
• EMA: the proportion of students in the school who are granted the Educational
Maintenance or Youth Allowance.
Schools have then been divided into three groups on the basis of the proportion of
LOTE speakers at home. In percentage terms, this amounts to:
• Very low:
0 to 4 per cent LOTE speakers at home
• Low:
4 to 26 per cent LOTE speakers at home
• Medium to high:
More than 26 per cent LOTE speakers at home.
Similarly, schools were divided into three groups on the basis of the proportion of
EMA/Youth Allowance recipients among their students:
• Very low:
0 to 28 per cent EMA/Youth Allowance recipients
• Low:
28 to 43 per cent EMA/Youth Allowance recipients
• Medium to high:
> 43 per cent EMA/Youth Allowance recipients.
This two-way categorisation yielded nine ‘Like School’ groups (with two selective high schools kept separate) which are displayed graphically in Figure 2, using data on all government secondary schools in 2001. Two particular schools are highlighted.
This like-schools definition gained a high level of acceptance in Victoria as schools became accustomed to comparing their results to like-schools as well as to statewide results. Nevertheless, there are some problems with this methodology, arising from the coarseness of the groupings. A school close to a boundary can find itself being compared with schools that are more advantaged than it is, and therefore receive a message of unwarranted underperformance. Equally, a school on the other side of a boundary, being compared with schools in more difficult circumstances, can receive flattering reports that are equally unwarranted.

28

Reporting and Comparing School Performances

Figure 2. Like-school data; Government Secondary Schools, 2001 (Source: VCE
Data Service Demonstration Reports, adapted).
The two schools represented by squares in Figure 2 illustrate this problem. Although these schools are about as similar on both dimensions as two schools could be, one school would be compared with schools of higher EMA/Youth Allowance (ie, lower socioeconomic status), while the other would be compared with schools of lower
EMA/Youth Allowance (ie, higher socioeconomic status). It is likely that the message to the first school would be that it was doing well compared to like schools, while the message to the other school would be that it was doing poorly compared to like schools. Neither conclusion is warranted by the data and such comparisons, if made public, could be misleading and possibly damaging to some schools.
For these reasons, and because of doubts about the continuing availability of the EMA measure, Victoria discontinued its like-schools reporting in 2007. Victoria now provides reports back to schools that use the Student Family Occupation (SFO) measure collected annually (MCEETYA, 2008). Each school is assigned a percentile on a school SFO measure, and invited to compare achievement levels with the 20 per cent of schools nearest to them on the SFO measure2. In effect, Victoria has replaced the two-dimensional (LBOTE-EMA) categorization with a one-dimensional (Family
Occupation) ‘statistical neighbour’ approach. statistical neighbour schools
The term ‘statistical neighbour’ is used by the province of Ontario, Canada, to describe a school reporting scheme that allows schools to make comparisons with schools that are most like them on various measures, including demographic measures
2

http://www.education.vic.gov.au/management/schoolimprovement/performancedata/performancereport
s.htm

29

Reporting and Comparing School Performances

based on the student population and school characteristics, such as location and school size.3 The current Victorian reporting scheme is, in effect, a statistical measure based on a single measure. It makes no use of LBOTE data or ATSI status, which, were it to be applied nationally, might be seen as less credible. We believe that the potential exists to develop a neighbour school based on multiple measures, and will outline briefly how such a scheme might work.
Looking once more at Figure 2, an appropriate comparison group for any school would be the group of schools that surround it on the two-way plot. This would involve an alternative conception of ‘like-schools’ – one in which each school had its own set of comparison schools, and, to the extent possible, would be at the centre of each group. Such an approach could be thought of as identifying ‘neighbour’ schools, where the schools are neighbours in their social makeup, but not necessarily geographically. The term ‘statistical neighbour’ effectively conveys this message, and to use it would lessen the risk that they reports would be misinterpreted as referring to geographical neighbours.
Figure 3 illustrates how this might be achieved. In Figure 3 a different school is highlighted, and a circle drawn around it. Depending on the radius of the circle, this approach identifies some number of ‘near-neighbour’ schools.
Among the issues that would have to be resolved in the implementation of a likeschools methodology are the student and school characteristics to be used in defining like-schools. Some of the data that could be used are identified in the MCEETYA
Data Implementation Manual (2008) and are summarised in Table 3.

Figure 3. Like-school data; Alternative Approach (Source: VCE Data Service
Demonstration Reports, adapted).
3

http://www.edu.gov.on.ca/eng/literacynumeracy/osneng.pdf

30

Reporting and Comparing School Performances

Table 3. MCEETYA Data Available for Use in Like-School Comparisons, 2008

The data in Table 3 are individual student data. These could be used to develop summary data for a school. Other school information (such as geo-location) also could be used in defining like-schools.
There is a question as to whether geo-location should be used as basis for like-school groupings. While it may be useful for parents moving into a new area to be able to compare schools in that area, reasons for not including geo-location in like-school definitions are that schools in geographic proximity do not necessarily represent appropriate comparisons. Comparisons in geographic area would only be appropriate if schools in the same area were serving the same population but clearly that is not the case. For example, schools serving public housing estates are sometimes in close proximity to wealthy private schools. In addition, the appropriate geographic area for a rural school might be large when the nearest school is far away and meaningless in terms of choice because of accessibility.
The development of a national like-schools methodology would be a substantial project that would have to address a range of issues including:
•

the most suitable combination of measures (this approach is not restricted to two measures, so the option to include 3 or 4 or 5 would exist); and
31

Reporting and Comparing School Performances

•

the appropriate size for like-school groups (the larger the group the more stable the estimates of group statistics, but the more heterogeneous the generated likeschool groups would be). The modelling of different sized groups would be important in making this decision.

Reflections
It is important to recognise that changes over time in the outcomes being achieved by a school do not necessarily reflect changes in the school’s performance; they may simply reflect changes in the student population. And there is some evidence that changes in the student population can be a direct consequence of publishing school outcome data, as more affluent parents withdraw their children from schools with poorer outcomes.
If schools are to be compared, and particularly if they are to be compared publicly, then it is important that the different circumstances and the different challenges they face are taken into consideration. We believe that a ‘like-schools’ methodology is the best way to do this, and we prefer an approach that is not based entirely on predefined categories of schools but that compares each school with the schools most similar to it (ie, ‘near-neighbour’ method).
The basis for defining like-schools is a topic requiring further investigation. Western
Australia uses a near-neighbour approach based in a single complex measure of socioeconomic background. Figure 3 uses two variables (LBOTE and EMA/Youth
Allowance). In general, we believe that multiple variables need to be taken into account in defining like-schools, including key student intake characteristics discussed in Section 5, and possibly school wealth (Section 4). There are various rule-based and clustering methodologies that could be used to define like-schools, and further work could be needed to explore alternatives and their implications.

32

Reporting and Comparing School Performances

7. MEASURING SCHOOL PERFORMANCE
In Section 6 we considered ways of comparing and evaluating the outcomes being achieved by schools. The approaches we discussed were based on a direct consideration of student outcomes; in other words, no attempt was made to ‘adjust’ a school’s outcomes or to combine outcomes for some other purpose. Our focus was on assisting stakeholders to interpret student outcome measures in the context of the school’s student intake characteristics and other circumstances.
Over recent decades school systems in various parts of the world have investigated methodologies for developing a ‘measure’ of each school’s performance. Under this approach, rather than being reported directly, student outcomes usually are treated as
‘raw’ inputs to statistical processes that produce a measure for each school.
7.1 The Intention to ‘Measure’
Underpinning these efforts has been the proposition that schools differ in their levels of performance and that these levels of performance can be measured. This proposition is reflected in the measures that some systems now report for their schools. In England, for example, each school’s performance is measured on a scale centred on 1000 (Figure 4).

Figure 4. Measures of school performance expressed numerically (England)4

4

‘contextualised value-added measures’, Key Stage 2 to Key Stage 4, 2007

33

Reporting and Comparing School Performances

In Figure 4, Bilton School in Warwickshire (measured at 984.6) is estimated to be a lower-performing school than Holy Family Catholic College (measured at 1011.8), which in turn is estimated to be a lower-performing school than Greenwood Dale
School (measured at 1068.6). The proposition is that the performance of every school of sufficient size to allow a meaningful measure can be measured on this scale, enabling direct comparison with every other school. Figure 4 shows the distribution of measures for all schools in England.
This fundamental proposition also underpins the reporting of school performances in
New York City (Figure 5). In New York City, rather than being reported as scores, measures of performance are reported as grades. Although New York City does not attempt to make fine-grained distinctions between schools, it too assumes that every school’s performance can be measured on a common scale and compared directly with the performance of every other school.

Figure 5. Measures of school performance expressed as grades (New York City)5
The basic intention to ‘measure’ individual schools on a scale of increasing performance makes this problem a standard measurement problem, meaning that standard measurement questions and considerations apply. The issues raised by the attempt to locate schools on a continuum of increasing performance are the same issues that must be addressed in any attempt to measure.

5

‘overall’ school grade shown, 2008

34

Reporting and Comparing School Performances

defining ‘better’ performance
Foremost among these considerations is the question of the measurement construct itself. What is it that we are attempting to measure? In this case, what do we mean by
‘better’ performance? What could be looked for and used as evidence (indicators) of
‘better’ or ‘worse’ school performance?
At the most basic level: Is the idea that schools differ along a continuum of increasing
‘performance’ a meaningful and useful idea in the first place? Is it possible to assemble evidence supportive of this idea? Is this idea useful in practice? These are routine measurement questions. For example, the idea that children differ in their levels of ‘reading proficiency’ is a proposition that must be tested in any attempt to measure reading proficiency. Is assembled reading evidence consistent with this idea, or does it suggest that children have different proficiencies for different kinds of texts, making the original idea less valid and suggesting, perhaps, that separate reading measures are required for different kinds of texts? In the case of reading, research over many decades has shown that the attempt to treat ‘reading proficiency’ as a single construct is well supported by empirical evidence.
Among education systems that attempt to measure school performance there appears to be general consensus that ‘performance’ should be defined and measured in terms of benefits for students. Higher performing schools are those that provide greater benefits to their students in the form of improved educational outcomes. It is possible to imagine other ways of defining school performance (eg, financial performance; quality of management policies and processes; success in recruiting and retaining able staff; levels of staff morale) but, rather than being incorporated into the definition of school performance, these features generally are viewed as enablers of performance.
Consensus to define school performance in terms of benefits for students raises the next question of the kinds of benefits that should be taken into consideration in developing a measure of a school’s performance. As noted earlier in this paper, schools work to promote many different outcomes for their students, including academic achievement, but also including capabilities and skills for work and citizenship, attitudes, values, interest and engagement in learning, school participation, school completion, successful transitions to post-school destinations, and so on. How broadly should ‘school performance’ be conceptualised? Is it generally true that schools performing well in one outcome area tend to perform well in all areas? If not, is a single measure of performance meaningful and helpful, or would measures of separate aspects of school performance be more appropriate? deciding on evidence
Following clarification of the construct/s to be measured (ie, ‘what’ we intend to measure), a second consideration is the evidence to be used as the basis for measurement (ie, ‘how’ performance will be measured). What observations will provide valid information about the construct/s? Again, this is a routine question in every attempt to measure, and the answer depends on what is being measured. For example, measures of students’ attitudes or values generally require different forms of evidence (eg, questionnaires) from measures of academic achievement (eg, tests and examinations), which in turn require different forms of evidence from measures of school completion/participation.

35

Reporting and Comparing School Performances

Because the essential purpose of school education is to improve outcomes for students
(knowledge, skills, understandings, values, attitudes, readiness for life and work, etc), measures of school performance are most valid when they are based on measures of these outcomes. Other things being equal, a ‘higher performing’ school is one that produces better outcomes for its students.
But this observation raises the question of how improvements in outcomes should be measured. It is common to interpret an increase in, say, a school’s Year 5 mathematics scores from one year to the next as evidence of improved outcomes. The problem with this interpretation is that changes in scores from one year to another may simply reflect differences between student cohorts. A better indicator of the contribution a school is making would be the increase in a single cohort’s mathematics scores between Year 3 and Year 5 – with higher performing schools producing greater achievement gains for their students across these years of school. adjusting for school circumstances
With agreement on what is to be measured and how it is to be measured, a third general consideration in the measurement of performance is the context or circumstances under which observations are made. Unless all observations are made in the same context, the context itself and the level of challenge it presents must be taken into account. Once again, this is a routine consideration in all attempts to measure. For example, standard educational measurement techniques take into account differences in the difficulties of the tasks that individuals undertake. Year 5 students usually are administered easier reading tasks than Year 7 students, but the performances of students in these two year levels can be measured and compared on the same measurement scale through standard techniques that estimate and take account of differences in task difficulty. Each student’s reading measure is based not only on records of success or failure, but also on the estimated difficulties of the tasks he or she undertook.
It is widely argued that measures of school performance should take into account not only what learning gains a school achieves for its students, but also the circumstances under which those gains were achieved. If two schools achieve identical gains, but one school operates under more challenging circumstances, then that school is considered the higher performer.
This raises the question of what circumstances influence a school’s ability to achieve greater learning gains for its students. Some circumstances relate to the school itself
(see Section 4). Some schools struggle financially, have sub-standard facilities and resources, limited access to technology, poor leadership, some minimally competent teachers, high levels of staff turnover and low levels of staff and student morale – all of which influence their ability to achieve high outcomes for their students.
Other circumstances relate to the student intake (see Section 5). It seems likely that schools with higher proportions of students with learning difficulties and lower levels of family support and social capital will face greater challenges. So will many schools that operate in communities in which English is rarely spoken.

36

Reporting and Comparing School Performances

Any attempt to take account of a school’s circumstances and the challenges it faces runs a risk of lowering performance expectations and leading to conclusions such as:
“Given its socio-economic intake, high rates of student absenteeism, low levels of community engagement and support, sub-standard facilities and low staff morale, this school is performing quite well”. By adjusting for school circumstances, the method underpinning school performance measures in England (see Figure 4) can have the unintended consequence of lowering expectations of some schools (eg, schools in lower socioeconomic areas). measuring performances
A fourth consideration is the methodology to be used to bring together decisions about what is to be measured, how it is to be measured, and whether and how differences in context/challenge are taken into account. Decisions about methodology also are a feature of educational measurement more generally. For example, a decision is required about the methodology for comparing scores on different tests (eg, this year’s
Chemistry examination and last year’s Chemistry examination). One approach is to make simple comparisons of students’ raw scores or percentage correct scores. But this approach ignores possible differences in test difficulty: correctly answering 80% of questions on one test will not represent the same level of achievement as correctly answering 80% of questions on an easier test. More advanced methodologies use statistical models to take account of the difficulties of the questions on each test. reporting performance
A fifth and final set of considerations concern the ways in which comparisons and measures of school performance are reported and used. Who has a legitimate interest in a school’s performance? What levels of detail are appropriate and useful for different audiences?
How and to whom should school performances be communicated? What are the likely consequences of reporting a school’s performance? What are possible unintended consequences? These are all standard educational measurement considerations. For example, the Standards for Educational and Psychological Testing developed jointly by the American Educational Research
Association, the American Psychological Association and the National Council on
Measurement in Education include:
Standard 13.1 “When educational testing programs are mandated by school, district, state, or other authorities, the ways in which test results are intended to be used should be clearly described. It is the responsibility of those who mandate the use of tests to monitor their impact and to identify and minimize potential negative consequences. Consequences resulting from the uses of the test, both intended and unintended, should also be examined by the test user.”

Any proposal to measure, compare and report the performances of individual schools must be clear about how such performance measures are intended to be used; what positive consequences are anticipated from reporting and the mechanisms through which those consequences are expected to operate; what possible negative consequences there might be, and how those will be minimised; and what processes are to be put in place to monitor and evaluate the impact of school performance measures. 37

Reporting and Comparing School Performances

7.1 ‘Contextualised Attainment’ Measures
One type of school performance measure is referred to by the OECD (2008) as a
‘contextualised attainment measure’. Performance measures of this kind are constructed by asking the question: ‘How much better (or worse) are this school’s measures of status than might have been predicted?’ (eg, How much better (or worse) is this school’s mean Year 5 Reading score than might have been predicted?):
Contextualised attainment models estimate the magnitude of contributing factors to student performance or attainment at a particular point in time... The adjustment to raw scores made with the inclusion of contextual characteristics provides measures that better reflect the contribution of schools to student learning than the use of ‘raw’ test scores to measure school performance.
(OECD, 2008)

A key question in developing contextualised attainment measures of school performance is the question of what factors should be taken into account in establishing the expected (‘predicted’) outcomes for a school, and so, measuring how much better or worse the school’s outcomes are than predicted from its circumstances.
In England, the variables used to predict each school’s outcomes are drawn from the
Pupil Level Annual School Census (PLASC) through which data are collected annually from each school. The variables that have been used to predict each school’s outcomes in England include:
• Gender
• Age
• Language (other than English?)
• Deprivation (in receipt of free school meal?)
• Deprivation of student’s local area
• Special Educational Needs
• Mobility (recent enrolment?)
• Ethnic group
• In care of local authority (residential/foster/etc?)
Contextualised attainment models also sometimes include measures of students’ general ability. The use of the General Achievement Test (GAT) to predict students’ results in the Victorian Certificate Education is, under the OECD’s definition, an example of contextualised attainment modelling.
Sophisticated statistical techniques have been developed and used to construct school performance measures, including multi-level models of the kind used in England:
More sophisticated cross-sectional models… take into account the hierarchical structure of school systems, with students nested within classes, classes nested within schools and schools nested within districts/local areas. (OECD, 2008, 8)

Essentially, contextualised attainment measures are an attempt to indicate how much better or worse a school’s outcomes are at a particular point in time than might have been predicted from its student intake characteristics. The resulting performance measure is a residual (observed minus expected), taking values around zero. In
England, the decision was made to add 1000 to each of these residuals to centre
38

Reporting and Comparing School Performances

school performance measures on 1000 (see Figure 4). Each residual is assumed to reflect the contribution of the school, although being a residual, it also contains the influence of all other factors not taken into account in the prediction process.
While such measures can be useful for particular purposes, there are dangers associated with their use. There is a risk that contextualised attainment measures will encourage users to make the judgement that low achievement is acceptable if it occurs in schools that suffer from social disadvantage. In tabular form, they encourage the interpretation that a certain margin above (or below) expectation has the same meaning in vastly different schools.
There are also technical problems with contextualised attainment measures that make them unfit for use in the public comparison of schools. Contextualized attainment measures are residuals – they are what is left over when the known and measured influences have been adjusted for. It is tempting to attribute variance in achievement that remains to the school, but actually it is the combined effect of the school, the family, the culture and a myriad of other influences that cannot be quantified.
Contextualised attainment measures may be useful for schools to consider as they strive to identify where they are doing well and where it appears that they could do better. They may be useful for state departments of education to pore over as they strive to identify schools that need attention and/or assistance. But they are not a valid means of ranking schools and they are misleading if they are presented in ways that invite the ranking of schools. Goldstein and Leckie (2008) note that they are an inappropriate basis for comparing and choosing schools:
From the point of view of school choice it seems clear that we should not adjust for any school level factors – those taken account of in the contextual valueadded rankings. The relevant question for a parent is whether, given the characteristics of their child, any particular school can be expected to produce better subsequent achievements than any other chosen school or schools. If a school level factor is associated with achievement this is strictly part of the effect being measured and therefore not something to be adjusted for. Thus, the DCSF contextual value-added estimates are not appropriate for choice purposes.
(Goldstein & Leckie, 2008)

A similar point is made by Rowley (2006):
(If) the school functions in difficult circumstances, with students who are unmotivated and difficult to teach, it will receive a higher rating, even though the learning gains may not be great. Another school, in more favourable circumstances, may receive a lower rating even though the learning gain is as great. (Rowley, 2006)

In our view, the decision in England to publish contextualised attainment measures was misguided and is not an example that Australia should follow. The use of measures of this kind, if they are constructed at all, should be restricted to internal use by schools and employing authorities.

39

Reporting and Comparing School Performances

7.2 ‘Value-Added’ Measures
Another type of school performance measure is a ‘value-added measure’. School performance measures of this kind are constructed by asking the question: ‘How much better (or worse) are this school’s measures of gain/growth than might have been predicted?’ (eg, By how much did the average Year 3-5 reading gain exceed expectation?). The distinguishing feature of value-added measures is that they are based on measures of student gain/growth over time:
Value-added models employ data that tracks the test score trajectories of individual students in one or more subjects over one or more years… Through various kinds of adjustments, student growth data is transformed into indicators of school value-added... A distinguishing feature of value-added modelling is the inclusion of prior performance measures that allow a more accurate estimation of the contribution of the school to student progress.
(OECD, 2008)

Value-added measures have the advantage over contextualised attainment measures of not having to ‘explain’ students’ absolute levels of attainment (Raudenbush, 2004). By focusing on gain/growth, they in effect condition out the unknowable contributors to students’ absolute levels of attainment and make it much more likely that what is being observed are school effects (although measures of gain/growth also may reflect non-school influences).
Because value-added measures report how much better (or worse) a school’s gain/growth measures are than predicted, a school can demonstrate positive growth but have a negative value-added measure because the observed growth was not as great as predicted (Goldschmidt, et al, 2005). As in the case of contextualised attainment measures, these residuals (deviations from expectation) are taken to represent the ‘value’ that each school has added. As residuals, value-added measures also carry the effects of all un-modelled influences on student gain/growth. However, because they focus on gain/growth rather than status, valued-added measures provide a more direct indication of a school’s contribution to the progress of its students. considerations There is a growing literature around the construction of value-added measures of school performance.
One consideration in value-added modelling arises from the way in which the task is conceptualised. Some approaches to value-added modelling see it as not different in principle from contextualised attainment modelling. The task is to predict student outcomes at a point in time (ie, status measures) on the basis of student intake characteristics. Models of this kind incorporate available measures of ability or attainment on some earlier occasion (referred to as ‘prior attainment’) into the mix of student intake variables used to predict current attainment.
When available measures from some earlier occasion are not on the same measurement scale as measures of current attainment, it becomes necessary to use the relationship between these two different sets of measures to predict a student’s current

40

Reporting and Comparing School Performances

attainment from their earlier performance. The difference between each student’s predicted score and actual current score (residual gain) is then used as the basis for estimating a school’s value-add. A problem with this approach is that value-added measures constructed in this way can be highly correlated with students’ observed scores, leading Gorard (2006) to the conclusion that ‘value-added is of little value’.
Value-added modelling as defined here is not focused on predicting attainment at a point in time from students’ background characteristics and measures of prior ability/attainment, but on predicting gains/growth over time. As discussed in section
3, measures of gain and growth require outcome measures for the same students on the same measurement scale on different occasions.
A second consideration – as in the case of contextualised attainment measures – is the set of variables to be used to ‘predict’ expected gain/growth in a school. Variables such as Indigenous status and socio-economic background help to explain students’ absolute levels of attainment, but to what extent are they also predictive of the progress that students make in a given period of time? Sanders (2000) argues that, by taking into account students’ starting points, measures of gain/growth largely eliminate the need to take account of student background characteristics.
Nevertheless, value-added modelling, as being discussed here, assumes that measures of gain/growth can be usefully predicted from student intake characteristics.
A third consideration is the possibility of unintended consequences arising from the use of value-added methods. For example, if socio-economic status is included as a predictor of student growth, then the consequence may be lower growth expectations for students from lower socio-economic backgrounds:
The introduction of contextual variables into the value-added model might have undesired consequences for the incentive effects upon schools. It should be borne in mind, however, that from the perspective of students and their families, school value-added measures might only be of secondary interest in comparison with measures of students’ absolute performance or individual student progress.
(OECD, 2008)

A final consideration concerns the reporting of value-added measures. Because valueadded measures (and contextualised attainment measures) are fundamentally residuals
(deviations from expectation), they can be difficult to explain meaningfully to parents and the public. Downes and Vindurampulle (2007) refer to the ‘tension’ between the objective of producing reliable school performance measures and the objective of reporting in ways that are understood. The same conclusion was reached by Saunders
(1999):
The value-added task began by appearing to promise better information for public consumption, but instead turned out to demonstrate that ‘better information’ and
‘public consumption’ are incompatible, if the latter depends on being able to access ‘simple and straightforward’ measures of progress.
(Saunders, 1999)

41

Reporting and Comparing School Performances

Reflections
There is currently a great deal of enthusiasm in some countries for developing measures of school performance in the form of ‘value-added’ measures. For example, a recent OECD report claims:
Value-added models can provide accurate quantitative indicators of performance that facilitate the identification of areas for improvement within schools and school systems, permit performance benchmarks to be created, and facilitate learning within and between schools. Value-added modelling can also be used to increase the effectiveness of existing institutions such as school inspectorates and enable more informed judgements to be made about schools.
(OECD, 2008)

We do not share this enthusiasm, and believe there are good reasons why value-added measures should not be used as a basis for the public comparison of schools and the creation of league tables of the kind shown in Figure 4.
Our reason for not sharing this enthusiasm is partly technical. Value-added measures are fundamentally residuals (the difference between a school’s observed outcomes and its predicted outcomes). Although these residuals are assumed to represent the contribution of the school, they also reflect whatever other influences there are on student outcomes that have not been captured in the value-added model. Attempts to treat residuals as ‘accurate quantitative indicators of performance’ and to assign them numerical values like 1068.6 and 974.1 that can then be used to rank schools, fail to acknowledge their inherent imprecision and the lack of certainty about what they represent. Another reason for not sharing this enthusiasm lies in the likely unintended consequences of public reporting of this kind. Under a value-added approach, expectations are lowered for certain kinds of schools (eg, schools in low socioeconomic areas, schools with significant proportions of students from non-English language backgrounds, and/or schools with significant proportions of Indigenous students). When schools are compared on the extent to which they live up to the expectations set for them in a value-added analysis, low absolute achievement can be masked – a fact acknowledged by the OECD:
The incentive to lift performance might be lowered in schools that have substantially higher contextualised value-added scores that take account of differences in socio-economic status. This might lower expectations and reduce incentives even in schools where the proportion of students with low absolute performance is worryingly high.
(OECD, 2008)

Goldstein (2001) made the following observations about value-added measures:
Their use as public accountability measures, eg, in the form of performance tables or ‘value-added league tables’ is inappropriate and would destroy their credibility and usefulness. If they were ever used to become ‘high stakes’ pieces of information … they would inevitably become distorted and no longer reflect any underlying reality of school, performance

42

Reporting and Comparing School Performances

Goldstein and Leckie (2008) went further:
The present DCSF contextual value-added tables are inappropriate for school choice, despite being promoted as such. Parents relying on league tables to select a school for their children are using a tool not fit for that purpose.

Nevertheless, we believe value-added methodologies have a role to play in research studies aimed at understanding factors that influence the performances of schools.
They may even be useful to education systems for identifying schools that perform exceptionally well or exceptionally poorly given their circumstances. But we would not recommend the use of value-added measures for school accountability purposes, for the construction of public league tables of schools, or for making fine-grained distinctions of school performance.

43

Reporting and Comparing School Performances

8. AUDIENCES AND PURPOSES FOR REPORTING
Before we turn our attention to the details of data that could be reported about schools, and how, we revisit the various audiences that require information about schools and the purposes that reporting can serve.
It is easy to assume that more information is better information, but this is not necessarily the case. In 1934, T. S. Eliot wrote in his poem The Rock: ‘Where is the knowledge we have lost in information?’ Increased information does not necessarily result in increased knowledge, unless it is provided in a form that intended recipients can make sense of. Too much information can stand in the way of knowledge if its sheer volume renders it inaccessible to its intended audience.
There are more than 9500 schools in Australia, and a single database with comprehensive data about all of them would, in reality, be inaccessible to all but a few people. Furthermore, there are issues of privacy that dictate that certain data (eg, the achievements of individual students) be available to some persons but not to others.
Data for release must be summarised, and the challenge is to identify the kinds of data summaries that best serve the legitimate needs of the various audiences.
For this reason, we first review the audiences for information about schools, and then, in Section 9, consider the kinds of data that might best serve their needs and the various formats in which nationally comparable data might be provided.
8.1

School Principals

It has been customary in all jurisdictions for central authorities to report detailed data on students’ test performances to school principals. Principals normally are expected to share these data with their staff, who use the data for a range of purposes.
Typically, individual student achievement data are used to identify students who are slipping behind, and to provide extra assistance where it is deemed necessary.
When data come from standard tests, it has been common to provide summaries at the subtest level. From these reports, a teacher may learn, for example, that her class, while doing quite well overall in Mathematics, is particularly adept at Number, but relatively weak in Algebra. This information can be used in planning and may lead to a re-allocation of time between areas within a subject, or even between subjects.
Further detailed information often is provided at the level of the test item. Australian teachers usually have access to literacy and numeracy data showing the percentage of students answering each test question correctly in their school and in the state.
Teachers have thus been able to identify specific tasks that their students have done well, and specific tasks that they have done less well. This has enabled teachers to identify areas of focus for future teaching.
The purpose of this reporting is clear: to provide school principals with knowledge to assist teachers to plan and implement more effective teaching. Reporting by central authorities to schools must always have this as its main purpose.

44

Reporting and Comparing School Performances

8.2

Employing Authorities

The school principal is ultimately responsible for providing the conditions under which teachers can teach effectively and students can learn effectively. For all schools, there is an employing authority whose responsibility is to support the school principal in ensuring that staff and resources are available to enable this to happen.
For Government and Catholic schools, there is a single organisation with responsibility for many schools (typically a Department of Education or a Catholic
Education Office – the titles vary from state to state.) For Independent schools there is typically a School Council (or similar) with responsibility for one school.
The information provided to the employing authority has a specific purpose – to enable it to be more successful in performing its role. We need to ask, therefore, what information will help to achieve this purpose.
Certainly, achievement data will be important. Employing authorities will need to know which schools have higher achievement levels and which have lower achievement levels. Where the achievement level in a school is low, authorities will want to know whether it is improving.
But information about achievement is not enough in itself. Employing authorities need to know the context in which that achievement occurs. Relevant data might include: •
•
•
•

Physical resources: Are the physical resources available in the school adequate for the programs that it needs to run?
Human resources: Are there sufficient qualified and capable staff in the school to deliver the programs that it needs to run?
Student backgrounds: Are there specific needs in the school that arise from particular characteristics of the student body (eg, high proportions of students with learning disabilities, English language deficiencies, refugee backgrounds, etc.)?
Student aspirations: What do the students generally aspire to do, post-school, and what programs are necessary for them to have the chance to achieve their aspirations? The reporting of information about individual schools to employing authorities will be useful to the extent that it enables them to make better decisions, and so to support schools more effectively.
8.3

School Communities

Every school has its own school community, which includes its students, their parents, and others who have a stake in the success of the school, including potential parents, local employers, former students and no doubt many others who have an interest in that particular school.
Information for the school community may be provided by a variety of means, including school Annual Reports, school websites, parent organisations and possibly a central website for the state and/or the sector.

45

Reporting and Comparing School Performances

This information will be used by parents to make their decisions about choice of school, about the level of support that the school needs and their willingness to provide that support, and about the adequacy of the school’s physical and human resources. For these purposes, the following kinds of information may be useful:
•
•
•
•
•

Achievement data: Are the students achieving at an adequate level, and are achievement levels in the school improving over time?
School programs: What programs are available (eg, VET, languages, music, art, sport, etc.)? What subjects are available and at what level? What extra-curricular activities and excursions are available, and at what cost?
Human resources: Are there sufficient capable and qualified staff to run the programs that the school is offering?
Physical resources: What is the quantity and quality of the physical resources available? Are the classrooms of adequate size, well heated and ventilated? Is the school small, friendly and intimate, or large and impersonal?
Fees: What fees are payable by parents? (This includes fees that may be described as voluntary, but that parents are expected to pay.) What happens if parents are unable (or unwilling) to pay?

There will be a range of information that parents and other interested parties may seek about their local schools. Some information may be obtained through visits to schools; other information may be made available by the schools or central authorities. The challenge is to identify the kinds of information that will be most useful to school communities and the most effective means of providing these.
8.4

The General Public

The general public has an interest in the performance of the Australian schooling system as a whole, including answers to questions such as:
•
•
•
•
•

How effectively are schools operating?
Are achievement levels satisfactory, and are they improving?
Are adequate resources being provided?
Are schools cost-effective?
Are my tax dollars being spent appropriately (eg, being directed to areas of greatest need)?

Reporting to the general public needs to provide information that answers these questions. A challenge is to decide on the kinds of information that might be reported publicly and the most appropriate means by which this information can be provided.
Reflections
Decisions about what to report and how to report are complex because reporting addresses numerous and complex purposes. Reporting is not simply a matter of gathering the maximum possible information and putting it into the public domain.
The task involves identifying the groups that will use and benefit from information about schools, the forms in which that information is best summarised, and the means by which it is best delivered.

46

Reporting and Comparing School Performances

There also are levels of confidentiality that must be observed. Information about the achievement of individual students is important to the school, but ethically and legally could not be made available beyond the school. Information about achievement classroom by classroom might legally be made available outside the school, but would present an ethical and practical minefield.
We believe that there will be an ongoing task to understand the information requirements of different stakeholders; to identify what kinds of information they would find helpful and how they expect to be able to use that information; and to find the most effective ways of summarising and reporting information about schools to support informed decision making.

47

Reporting and Comparing School Performances

9. PUBLIC REPORTING
In this final section of the paper we address issues relating to, and options for, the public reporting of information about schools. We consider formats for reporting and also the kinds of data that could be reported.
9.1 Reporting School Data in Tables
Initial ventures into this field used print as a vehicle for dissemination, and consequently were compelled to set out data in the form of tables. The so-called
‘League Tables’ in the United Kingdom from 1992 provided the best-known and probably most controversial example of this style of reporting.
The use of the tabular reporting format has a number of consequences. Notably, it restricts the type of data that can be presented to information that can be reduced to a single figure and presented in one column of a table.
Data reported in tables have effectively been restricted to school means (eg, average examination scores) and the percentage of students meeting specified standards (eg, A and B grades). Published data have not conveyed the different contexts in which schools work or the different levels of challenge they face. And all schools have been evaluated against the same targets, even though their students may have been on quite different educational and vocational paths.
The print format and the large number of schools in the UK meant that, for most outlets, it was only practicable to publish regional tables, and until the Internet, became widely available, complete data were difficult to access.
Data reported in this form effectively invite users to rank schools, which the press duly did – hence the immediate appellation ‘League Tables’. Rankings of schools from highest to lowest were created and published on most of the key measures, leading to opposition from teacher unions, education professions and academics. The controversy associated with this led to the discontinuance of the scheme in Wales and
Northern Ireland by 2001.
Figure A1, in Appendix A, provides an example of an English ‘League Table’, as currently published. The inclusion of Special Schools (see the last row) illustrates the difficulty of conveying information relevant to all schools in this format.
Delivery of data via the world-wide web has relieved reporting authorities of many of the restrictions imposed by the tabular print format. Print publication still occurs but all information is freely available on the authorities’ websites (eg, see the Queensland,
Victorian and Western Australian secondary school achievement reports in Figures
A2, A3 and A4, in Appendix A),
The range of data provided is quite extensive, and includes some measures of school context as well as outcomes measures, as summarised in Table 4.

48

Reporting and Comparing School Performances

Table 4. Summary of Senior Secondary School Information Tables (Victoria,
Queensland and Western Australia)
Queensland
School Context

Outcomes

School Context

Outcomes

Graduation

TEE/WACE course achievement Gender (Coeducational, Boys only, Girls only)
Breadth of Curriculum
Number of certificates of post-compulsory school education
OP-eligible with no VET qualification
OP-eligible with one or more VET qualification
Total Senior certificates awarded
Number of students completing VET competencies
Number of VET qualifications awarded
Number of students completing/continuing a school-based apprenticeship or traineeship
Percentage of OP-eligible students with OP 1-15
Percentage of students awarded Senior Certificates and awarded one or more VET qualification
Percentage of students awarded Senior Certificates with OP-eligibility or awarded a VET qualification
Percentage of QTAC applicants receiving a tertiary offer.
Victoria
Number of VCE studies at unit 3-4 level taken up by students for
2007
Number of VET certificates with 2007 enrolments
Availability of International Baccalaureate (Diploma)
Number of students enrolled in at least one level 3-4 VCE unit in
2007
Number of students enrolled in a VET certificate in 2007
Number of students enrolled in VCAL in 2007
Percentage of VCE Students applying for tertiary places
Percentage of satisfactory VCE completions in 2007
Percentage of VET units of competence completed in 2007
Percentage of VCAL units completed in 2007
Median VCE study score
Percentage of study scores of 40 and over
Western Australia
Number of full-time students eligible to graduate.
Number of full-time students eligible to graduate who graduated.
Percentage of full-time students eligible to graduate who graduated.
Number full-time students eligible to graduate who sat four or more
TEE subjects.
Percentage of full-time students eligible to graduate who sat four or more TEE subjects.
Number of full-time students eligible to graduate who had at least one scaled mark greater than 75.
Percentage of full-time students eligible to graduate who sat 4 or more
TEE/WACE course examinations and had at least one scaled mark greater than 75.
Percentage of full-time students eligible to graduate who sat four or more TEE/WACE course examinations and received average scaled marks in the Low, Mid and High thirds.

49

Reporting and Comparing School Performances

Wholly schoolassessed achievement

Number of full-time students eligible to graduate who completed three or more WSA subjects.

Structured workplace learning

Percentage of full-time students eligible to graduate who completed three or more WSA subjects.
Number of full-time students eligible to graduate who completed three or more WSA subjects and received at least one A grade.
Percentage of full-time students eligible to graduate who completed three or more WSA subjects and received at least one A grade.
Number of full-time students eligible to graduate who participated in at least one structured workplace learning (SWL) subject.
Percentage of full-time students eligible to graduate who participated in at least one SWL subject.
Number of full-time students eligible to graduate who achieved at least one A grade in at least one SWL subject.
Percentage of full-time students eligible to graduate who achieved at least one A grade in at least one SWL subject.

Figures A2, A3 and A4, in Appendix A, provide examples (the first page) of each state’s 2007 tables. Queensland’s table occupies 19 pages, Victoria’s (in a more compressed format) 11 pages, and Western Australia’s 4 pages.
All these tables, it should be noted, include a range of outcome measures designed to reflect the range of pathways for which schools prepare students in their upper secondary years. All three emphasise vocational training as well as tertiary preparation. Western Australia’s report (published as a downloadable document, includes 10 tables of school data, including listings of the top 50 schools on a range of measures derived from school graduation rates, tertiary entrance scores, schoolassessed marks and achievement of VET qualifications. Only Victoria provides an average achievement measure (the median study score); the rest of the measures are couched in terms of the percent of students achieving specific targets, such as graduation or receiving tertiary offers.
Victoria also publishes a destination survey in June of each year, known as On Track
(see Figure A5, in Appendix A,). Again, presented in a single table (over many pages), On Track provides the percentages of students from each school who apply for tertiary places, the number who receive tertiary offers, and the number who are currently enrolled in university and in TAFE/VET programs, in apprenticeships, in employment, or looking for work. Some of these data are drawn from official records, other from an annual survey conducted specifically for On Track.
The trend, in recent years, has been to publish multiple outcome measures reflecting the range of pathways that students pursue, some details of programs available at each school, and student success in achieving targets such as secondary graduation, achievement of VET qualifications, tertiary selection and employment.
But it does appear that, with the restrictions imposed by the mechanism of tabular mode of presentation, this is about as far as we can go. A single table cannot carry all of the information that these jurisdictions have been striving to present, leading to a proliferation of tables, as seen also in the United Kingdom.
50

Reporting and Comparing School Performances

A different approach has emerged in the past two years and has gained a great deal of attention. New York City began rating schools on an ABCDF grading system in
2007, and with the release of the 2008 school grades in October, ambitious claims have been made for the achievement gains that it has produced. A listing of ‘school grades’, published in the New York Times, is shown in Figure A6, in Appendix A.
The New York City table differs from the UK and Australian tables reviewed previously in two major respects:
1. it publishes ratings of schools, in addition to factual data, and
2. the ratings have explicitly-stated consequences, which can extend, in extreme circumstances, to school closures.
The school ratings, and their translation into ABCDF grades, have predictably grabbed the headlines. The ratings are derived by weighting and aggregating a range of data, as shown in Table 5. It is worth noting that the ratings depend on both school achievement data and data obtained in surveys of parents, students and teachers.
Table 5. Mechanism for generating school grades: New York City, 2008





School Environment constitutes 15% of a school's overall score. This category consists of attendance and the results of parent, student, and teacher surveys.
Student Performance constitutes 25% of a school's overall score. For elementary and middle schools, student performance is measured by students’ scores each year on the New York State tests in English Language Arts and Mathematics. For high schools, student performance is measured by diplomas and graduation rates.
Student Progress constitutes 60% of a school's overall score. For elementary and middle schools, student progress measures average student improvement from last year to this year on the New York State tests in English Language Arts and
Mathematics. For high schools, student progress is measured by credit accumulation and Regents completion and pass rates.
Source: http://schools.nyc.gov/Accountability/SchoolReports/ProgressReports/default.htm

The weightings were changed between 2007 and 2008, so the grades have, so far, constituted a shifting measure, and the fact that many schools have increased their grades in this time has numerous possible explanations.
Claims that the reports have led to significant increases in student achievement are, at this stage, premature. The system was introduced in 2007, so if it was to have had an effect it could only have occurred in 2008 measures of achievement. National
Assessment of Educational Progress (NAEP) reports for inner city schools 2002-07 show that progress in New York City from 2002 in reading has essentially mirrored that of other cities and the nation as a whole. Figure A7, in Appendix A, provides a brief summary, and more detail can be found by going to the original sources at http://nces.ed.gov/nationsreportcard/pdf/dst2007/2008455.pdf for Reading, and http://nces.ed.gov/nationsreportcard/pdf/dst2007/2008452_1.pdf for Mathematics.

51

Reporting and Comparing School Performances

In spite of the claims made in the media, we have yet to see any solid evidence that the reporting scheme has led to increased achievement. Evidence to date is insufficient to recommend a change in this direction.
9.2 Providing School Profiles
While publicity around the New York City initiative has focused largely on the school grading system, it has largely overlooked the second, perhaps, less newsworthy, aspect of the New York ‘experiment’. Each school has access to its own Report Card, which contains a range of information in addition to the school ‘grade’ (See Figure
A8, in Appendix A, for an example of a New York School Report Card).
The New York City School Report Cards have much in common with the school profiles that are already in use in many countries and several Australian states.
As noted above, the tabular format imposes limitations on the type of data that can be provided. It requires that exactly the same measures be provided for every school, even though the schools may perform very different roles (the inclusion of a Special
School in the last row of Figure A1 illustrates how inappropriate this can sometimes be). Descriptions of school environment, staffing and facilities, school philosophies and the particular pathways that schools might focus most heavily on, simply cannot be fitted into this format. For this and other reasons, there has been a move towards the provision of school profiles via a central website. Two good examples are provided by Western Australia and Tasmania.
Western Australia’s Schools Online site provides an opening screen (Figure A9, in
Appendix A), from which a user may choose any Government school in Western
Australia. Choosing a school leads the user to a School Overview page (Figure A10, in Appendix A) with a passage of descriptive information (picture and text) about the school. From the School Overview, the user goes to a School Data page (Figure A11, in Appendix A), containing summary data on attendance rates and selected measures of school outcomes. For secondary schools, this includes year 12 participation rates, and the percentage of eligible students achieving particular achievement targets – secondary graduation, scaled scores of 75 or more and ‘A’ grades in wholly schoolassessed subjects.
Tasmania’s School Improvement Report is structured similarly, although it is visually quite different. It, too, provides an opening screen (Figure A12, in Appendix A), from which a user may choose any Government school in Tasmania. Choosing a school leads the user to a School Overview page (Figure A13, in Appendix A) containing descriptive information (picture and text) about the school. From the School
Overview, the user goes to a School Data page (Figure A14, in Appendix A), containing more extensive summary data on what might be thought of as school environment measures, but are described on the website as ‘school improvement’.
For secondary schools, the data provided includes measures of:
 Achievement gains (indexed gains on literacy and numeracy tests, years 7-9)
 Student attendance
 Year 10-11 retention rates),
 Staff attendance rate

52

Reporting and Comparing School Performances






Staff satisfaction measure, from staff surveys
Student satisfaction measure, from student surveys
Parent satisfaction measure, from parent surveys
Indigenous equity (the difference between achievement outcome rates between indigenous and non- indigenous students).

Both Tasmania and Western Australia have facilitated the presentation of narrative portraits of schools via their state websites, and we consider it important that schools present this information in their own way – it is, in a sense, their window to the world.
If left to the discretion of the school, there will be great variation in the quality and range of information that schools provide. For example, in Tasmania’s School
Improvement Reports, one high school provided information as follows:
Huonville High School has 40 teaching staff and caters for 435 students from
Year 7 to Year 10. The school's address is 82 Wilmot Road HUONVILLE 7109.
Phone [phone]

A P-10 school of somewhat smaller size is much more forthcoming:
Sorell School was established on its current site in 1821 and has proudly served
Sorell and surrounding districts for all of its 187 years. We believe that it is the oldest continuously operating school in Australia. Sorell School enrols students from kindergarten to grade 10 and also runs vocational and training programs for grades 11 and 12. We have separate primary and secondary campuses and a standalone kindergarten at nearby Midway Point. Grade 6 and 7 students learn together in our Middle School which is designed to support the development of young adolescents and to smooth the transition from primary to secondary schooling. Sorell runs a number of programs that are unique to the school. For example, on our primary campus we have built a replica pioneer village that includes a working blacksmith’s shop, schoolroom and settler’s hut as well as a range of other buildings. In this village our primary students participate in role play activities as they study such historical topics as the settlement of Tasmania, the gold rush and the convict system. Our unique secondary programs include a restaurant experience for our senior citizens called Eating with Friends; horticulture which is based on our school farm and; a Men’s Shed program which is based at the old Sorell railway yards and involves our students learning practical skills from senior men in our community. Our photograph shows a session of cross-age tutoring in which our secondary students support the learning of our primary students.

Similar websites exist in many countries. Most focus heavily on achievement measures, and others are quite encyclopaedic in the information that they provide.
Five Canadian provinces use a website established by the Fraser institute to present school-by-school data on provincial test results (see Figure A15, primary, and Figure
A16, secondary, in Appendix A, for examples).
The data provided are largely statistical summaries, although some manipulation is done to provide a measure of gender gap and value-added (where suitable data are available). It also includes an overall rating out of 10, analogous to the New York
City school grades.

53

Reporting and Comparing School Performances

The explanation provided for the value-added scores and the rating is scant indeed, and one can only wonder how school staff in particular, and school communities in general, can interpret them meaningfully:




3-year value-added. These are estimates of the school’s contribution to its students’ results on the grade-6 reading and mathematics tests. Schools that have a strong positive impact on their students receive an A for this indicator. Those that receive a B or a C may have some positive impact on their students. Schools that have little positive impact receive a D.
Overall rating. The Overall rating out of 10 takes into account the nine indicators described in E through M above to answer the question, ‘In general, how is the school doing academically compared to other schools in the province?’
Source: http://www.fraserinstitute.org/reportcards/schoolperformance/howtoread/one.htm

Like the New York City grades, the Overall rating combines a number of indicators together with arbitrary weights to provide a global measure of something, intended to represent the ‘merit’ of the school’s performance. The arbitrariness of the weighting means that it will value highly the areas of strength of some schools, and value less highly the strengths of other schools. The apparent objectivity of the measure can easily conceal the subjectivity of the decisions that determine the rating awarded to any particular school.
One of the most comprehensive School Report Cards is that provided by the state of
California. For a single school, the full report can extend to 10 pages, and includes data on:
• Student enrolment
- by grade level
- by racial/ethnic subgroup
- by level of socioeconomic disadvantage
- by English learners
- by Disability category
• School facilities (repair and maintenance details)
• Teacher credentials
• Teacher mis-assignments and vacant teacher positions
• Core academic courses taught by “highly-qualified” teachers (definition provided) • Academic counsellors and other support staff
• Quality, Currency, and Availability of Textbooks and Instructional
Materials
• List of Textbooks and Instructional Materials Used in Core Subject Areas
• Expenditures per Pupil and School Site Teacher Salaries
• Types of Services Funded
• Teacher and Administrative Salaries
• Standardized Testing and Reporting (STAR)
• California Standards Test (CST): Percentage of students achieving the
Proficient or Advanced levels in English, Mathematics, Life Science and
History/Social Science
- compared to District and State
- broken down by gender, English language background, socioeconomic disadvantage and racial/ethnic background

54

Reporting and Comparing School Performances

•

•
•

•
•
•

California Achievement Tests (Norm-Referenced): Percent of students achieving at or above the 50th percentile in Reading and Mathematics
- compared to District and State
- broken down by gender, English language background, socioeconomic disadvantage and racial/ethnic background
California Physical Fitness Test Results Percent of students meeting state fitness standards
Academic Performance Index (API) - an annual measure of the academic performance and progress of schools in California.
- API Ranks—Three-Year Comparison
- API Changes by Student Group—Three-Year Comparison
Adequate Yearly Progress (AYP): Overall and by Criteria (2007)
Federal Intervention Program
School completion and postsecondary preparation (secondary schools)

For users not requiring the full details, the website provides a ‘Short Form’ of two pages, which includes a selection of the information provided in the full report, judged to be of most interest to users of the website. A sample Short Report is provided in
Figure A17 in Appendix A.
Reflection
Delivery of data to the public in tabular form is no longer necessary, and has several disadvantages: • it restricts the range of information that can be provided;
• it does not allow the presentation of ‘rich’ information of the kind that parents seek when making decisions about their children’s schooling;
• it encourages the ‘rank order’ interpretations that have been damaging in the past; and • for the whole of Australia, and even for the larger states, such tables would be extremely large and cumbersome.
The provision of information to the public in the form of school profiles is increasingly common, and has many advantages:
• it allows an almost limitless range of information to be presented;
• it facilitates the presentation of ‘rich’ information, provided by the school or by central data collection;
• it encourages users to focus on the schools that have a community of interest, eg, geographically, by sector or by religious affiliation.
• profiles for the whole of Australia, could be accessed from a single website, by making successive choices, e.g., state, region, suburb, etc..
We consider it important that schools be given some guidelines about the character of the reports that they provide in their ‘windows to the world’. While it is beyond the scope of this report to spell out exactly what these guidelines should be, we think it would be appropriate to suggest an approximate word length, and to provide a list of suggested content, such as:
 Location (including any distinguishing characteristics of the neighbourhood),
 History of the school
 Physical facilities – buildings, land, special function rooms or buildings (eg, libraries, art or drama centres), sports facilities,
55

Reporting and Comparing School Performances





School mission, values, philosophy
Special programs available that make the school distinctive
One photograph that, in the opinion of the school principal, effectively conveys what the school is like to an outsider

Some websites provide a wealth of detail that we judged to be far too detailed for consideration in this project. – for example the long reports in the California School
Accountability reports (see Figure A17) include detailed reports on school onsite inspections, all repairs recommended and those carried out. We see it as important, as far as possible, to confine school reporting to data that are readily available, and do not require high levels of bureaucratic intervention to set up.
For this reason, we are not recommending, at this point, that surveys of student, parent and teacher satisfaction, such as those conducted in several states and reported on the
Tasmanian School Improvement website, be reported nationally. The level of bureaucratic intervention required to collect such data from representative sample across Australia would be too great to contemplate.
Despite our reservations about the use of tabular reporting for public purposes, we believe tables of data would be useful for state departments of education and other employing authorities. Whatever the form of presentation, underlying all such tables is a spreadsheet, which enables sorting by any measure, easy calculation of composite scores, and identification of schools that are outliers on any measure.
The examples used in this paper have been developed by state education departments and have presented data from government sources only. Where independent schools have been included (eg, in the Canadian provinces), it appears to have been at their discretion. 9.3 What Data should be Reported?
The examples discussed in Section 9.2 demonstrate the wide range of information that school systems around the world have decided to report. In this section we consider in more detail the kinds of nationally consistent data that could be collected and reported in Australia.
For reasons outlined in Sections 6.5, 7.2 and 7.3, we believe that the comparisons most useful for schools and for public release are those with like-schools as defined using a ‘near neighbour’ methodology. As noted in Section 6.5, such a methodology does not yet exist at the national level, but needs to be developed. The term ‘likeschools’ in Tables 6 through 9 refers to comparisons based on this methodology. student outcome data: NAPLAN
NAPLAN tests provide nationally comparable student outcome data in literacy and numeracy. They also provide test results that can be compared from year to year, enabling reporting on the following timeline:
 from 2008 measures of current status
 from 2009 measures of improvement, and
 from 2010 measures of individual growth, aggregated over schools and groups of schools (regions, school types, etc.).
56

Reporting and Comparing School Performances

For public reporting, we consider it important to build on the understanding that parents and the public have built up from reading NAPLAN student reports. These reports locate each student’s achievement within a band, and in relation to Australian students in general. If school reports are to build upon this understanding, then they should enable a reader to locate the level of achievement in a school in much the same way. Tables 6, 7, 8 and 9 illustrate possible ways of reporting NAPLAN data.
Table 6. Suggested Reporting Plan for Year 3 NAPLAN Data
NAPLAN: Year 3 Percentage achieving National Standard – Current status
Reading:
School
Like Schools
Australia
Writing:
School
Like Schools
Australia
Conventions:
School
Like Schools
Australia
Numeracy:
School
Like Schools
Australia
NAPLAN: Year 3 Percentage achieving National Standard – Improvement Over Time*
Reading:
Writing:
Conventions:
Numeracy:

*

Two years Previously
Two years Previously
Two years Previously
Two years Previously

One year Previously
One year Previously
One year Previously
One year Previously

Current year
Current year
Current year
Current year

as national data become available

Table 7. Suggested Reporting Plan for Year 5 NAPLAN Data
NAPLAN: Year 5 Percentage achieving National Standard – Current status
Reading:
School
Like Schools
Australia
Writing:
School
Like Schools
Australia
Conventions:
School
Like Schools
Australia
Numeracy:
School
Like Schools
Australia
NAPLAN: Year 5 Percentage achieving National Standard – Improvement Over time*
Reading:
Writing:
Conventions:
Numeracy:

Two years Previously
Two years Previously
Two years Previously
Two years Previously

One year Previously
One year Previously
One year Previously
One year Previously

Current year
Current year
Current year
Current year

NAPLAN: Current Year 5 Students Compared to their Year 3 Results - Gain*
Reading:
Writing:
Conventions:

School:
Like Schools:
Australia:
School:
Like Schools:
Australia:
School:

Current Mean
Current Mean
Current Mean
Current Mean
Current Mean
Current Mean
Current Mean

57

Previous Mean
Previous Mean
Previous Mean
Previous Mean
Previous Mean
Previous Mean
Previous Mean

Mean Gain
Mean Gain
Mean Gain
Mean Gain
Mean Gain
Mean Gain
Mean Gain

Reporting and Comparing School Performances

Like Schools:
Australia:
School:
Like Schools:
Australia:

Numeracy:

*

Current Mean
Current Mean
Current Mean
Current Mean
Current Mean

Previous Mean
Previous Mean
Previous Mean
Previous Mean
Previous Mean

Mean Gain
Mean Gain
Mean Gain
Mean Gain
Mean Gain

as national data become available

Table 8. Suggested Reporting Plan for Year 7 NAPLAN Data
NAPLAN: Year 7 Percentage achieving National Standard –Current status
Reading:
School
Like Schools
Australia
Writing:
School
Like Schools
Australia
Conventions:
School
Like Schools
Australia
Numeracy:
School
Like Schools
Australia
NAPLAN: Year 7 Percentage achieving National Standard – Improvement Over time*
Reading:
Writing:
Conventions:
Numeracy:

Two years Previously
Two years Previously
Two years Previously
Two years Previously

One year Previously
One year Previously
One year Previously
One year Previously

Current year
Current year
Current year
Current year

NAPLAN: Current Year 7 Students Compared to their Year 5 Results - Gain *
Reading:
Writing:
Conventions:
Numeracy:

*

School:
Like Schools:
Australia:
School:
Like Schools:
Australia:
School:
Like Schools:
Australia:
School:
Like Schools:
Australia:

Current Mean
Current Mean
Current Mean
Current Mean
Current Mean
Current Mean
Current Mean
Current Mean
Current Mean
Current Mean
Current Mean
Current Mean

as national data become available

58

Previous Mean
Previous Mean
Previous Mean
Previous Mean
Previous Mean
Previous Mean
Previous Mean
Previous Mean
Previous Mean
Previous Mean
Previous Mean
Previous Mean

Mean Gain
Mean Gain
Mean Gain
Mean Gain
Mean Gain
Mean Gain
Mean Gain
Mean Gain
Mean Gain
Mean Gain
Mean Gain
Mean Gain

Reporting and Comparing School Performances

Table 9. Suggested Reporting Plan for Year 9 NAPLAN Data
NAPLAN: Year 9 Percentage achieving National Standard – Current status
Reading:
School
Like Schools
Australia
Writing:
School
Like Schools
Australia
Conventions:
School
Like Schools
Australia
Numeracy:
School
Like Schools
Australia
NAPLAN: Year 9 Percentage achieving National Standard – Improvement Over time*
Reading:
Writing:
Conventions:
Numeracy:

Two years Previously
Two years Previously
Two years Previously
Two years Previously

One year Previously
One year Previously
One year Previously
One year Previously

Current year
Current year
Current year
Current year

NAPLAN: Current Year 9 Students Compared to their Year 7 Results - Gain*
Reading:
Writing:
Conventions:
Numeracy:

*

School:
Like Schools:
Australia:
School:
Like Schools:
Australia:
School:
Like Schools:
Australia:
School:
Like Schools:
Australia:

Current Mean
Current Mean
Current Mean
Current Mean
Current Mean
Current Mean
Current Mean
Current Mean
Current Mean
Current Mean
Current Mean
Current Mean

Previous Mean
Previous Mean
Previous Mean
Previous Mean
Previous Mean
Previous Mean
Previous Mean
Previous Mean
Previous Mean
Previous Mean
Previous Mean
Previous Mean

Mean Gain
Mean Gain
Mean Gain
Mean Gain
Mean Gain
Mean Gain
Mean Gain
Mean Gain
Mean Gain
Mean Gain
Mean Gain
Mean Gain

as national data become available

The suggested tables present percentages of students meeting the National Standard
(ie, in Band 2 or above for Year 3; Band 4 or above for Year 5, Band 5 or above for
Year; Band 6 or above for Year 9).
Percentages in other bands could also be used, and it might be argued that the presentation of data related to the lowest band only (like the National Benchmarks in previous years) focuses too much attention on low achievement at the expense of high achievement. This could be addressed by the reporting of another indicator, such as the percent of students in the top 20 percent nationally. But we think it is important not to make the reports any more complex than necessary.
To maintain consistency, we suggest that the tables be accompanied by graphical representation in similar style and visual impact to the NAPLAN parent reports. This will enable parents in particular to more quickly become comfortable with the reporting format.
Figure 6 provides an illustration of the types of graphical presentations that we have in mind.

59

Reporting and Comparing School Performances

Figure 6. Sample Graphic Format Recommended for School Comparisons student outcome data: senior secondary
There is no common reporting of senior secondary school achievement across jurisdictions. As noted previously, for students seeking to proceed to tertiary education, the most valued measure is captured by the tertiary admissions agencies in each jurisdiction – namely the UAI, ENTER, TER and the Interstate Transfer Index.
Its value for students depends on the achievement of specific targets – for example, a score of 90 will gain admission to certain programs in high-prestige institutions and most in lower-prestige institutions. A score of 80 gains entrance to a narrower range of programs, and so on.
In the absence of nationally comparable measures of subject achievement in the senior secondary school, we believe that tertiary entrance ranks could be useful indicators of achievement for those students who have them. Table 10 provides an example of a possible table to report tertiary entrance data.
Table 10. Suggested Reporting Plan for Tertiary Entrance Rank Data
Tertiary Entrance Ranks: Percentage of Ranks Awarded
90 and above:
School
Like Schools
80 and above:
School
Like Schools
70 and above:
School
Like Schools
60 and above:
School
Like Schools

Australia
Australia
Australia
Australia

It would be necessary to vary the terminology from jurisdiction to jurisdiction, but the format of the table need not change. As for NAPLAN results, the table should be accompanied by charts in the format of NAPLAN student reports.
Because tertiary entrance ranks are constructed for tertiary entrance purposes, there may be privacy concerns that prevent the release of individual-level data without student permission (applying for tertiary admission in effect constitutes permission to

60

Reporting and Comparing School Performances

release the data to the institutions applied for). It may not be possible for all tertiary entrance ranks to be released to a national agency.
However, since 2002, Victoria has been able to report the percentage of students applying for tertiary entrance school by school as a result of negotiations between the
Victorian Curriculum and Assessment Authority (VCAA) and the Victorian Tertiary
Admissions Centre (VTAC). Each year a formal request goes from VCAA to VTAC requesting, not the original data, but the breakdowns by school. The resulting file is then integrated into the data generated by the VCAA, which has responsibility for publishing this information (shown in Figure A3, Appendix A).
We believe that similar arrangements could be made, where necessary, for tertiary entrance rank, tertiary applications and offers, and similar data to be reported nationally. There are no other comparable achievement data, but there are other targets that senior secondary students pursue in all jurisdictions. Looking at the tables for
Victoria and Western Australia, and the school websites provided by Western
Australia and Tasmania, it would appear that the following data would be acceptable and desirable.
Table 11. Suggested Reporting Plan for Other Senior Secondary Data
Measure

Definition

Per cent senior secondary retention The percentage of students completing Year 10 two years previously who are enrolled as Year 12 students in the current year. Per cent successfully completing senior secondary qualification The number of students achieving the award in their state that marks successful completion of secondary school (HSC, SACE,
WACE, etc.) taken as a percentage of the number whose enrolment, if completed successfully, would have gained them the award. Where there are two awards at senior secondary level (as in Victoria), the number of successful completions and the number eligible could be aggregated over the two programs, or reported separately.

Per cent tertiary applicants

The number of students applying for tertiary selection, taken as a percentage of the number whose enrolment, if completed successfully, would have made them eligible for tertiary selection.

Per cent offered tertiary places The number of students offered selection in a tertiary institution, taken as a percentage of the number who applied for tertiary selection. Per cent completion of VET studies The number of VET studies completed, taken as a percentage of all enrolments in VET studies.

61

Reporting and Comparing School Performances

Table 11 contains both outcomes (secondary graduation, tertiary offers, VET completion) and measures of school context (percent tertiary applicants, percent VET enrolment). But because of their interconnections, we think it appropriate that they be reported together. student outcome data: middle-school
As noted previously, there is no nationally-comparable measure of achievement between the Year 9 NAPLAN tests (administered early in Year 9, and therefore reflecting achievement barely beyond year 8), and the end of secondary schooling.
Even at Year 9, NAPLAN assesses only Literacy and Numeracy, which by this stage occupy a much smaller proportion of the total program in secondary schools.
We believe that, as a National Curriculum is implemented, it should be possible to introduce a set of tests to be administered late in year 10, and covering the skills and essential knowledge in core subjects that students are expected to acquire by the completion of Year 10.
Two purposes would be achieved by subject-focused tests at this stage of schooling:
 they would provide data to enable schools and their employing authorities to assess the success of programs in the crucial middle-secondary years; and
 they would provide a measure of what students bring to their senior secondary studies, and so would provide baseline measures for evaluating senior secondary programs. Late in year 10 would be an ideal time for such a testing program, since it
(approximately) marks the end of compulsory schooling, and in most schools it is the last year in which there is a common curriculum. From Year 11, nationally comparable subject assessments, if available, would apply only to students choosing those subjects.
A decision would be required about the format to be used to report Year 10 subject assessments. One possibility would be to construct achievement ‘bands’. For example, in the first year (say 2010), each band could be defined to contain ten per cent of all Year 10 students nationally, but the percentages in the bands would vary from school to school and state or territory to state or territory. With suitable equating from year to year, the Year 10 reports could not only detect improvement in a particular school, but in the nation, were it to occur.
Band
Band
Band
Band :
Band :
Band :
Band :
Band :
Band :
Band :

10:
9:
8:
7
6
5
4
3
2
1

10th decile on 2010 test score
9th decile on 2010 test score
8th decile on 2010 test score
7th decile on 2010 test score
6th decile on 2010 test score
5th decile on 2010 test score
4th decile on 2010 test score
3rd decile on 2010 test score
2nd decile on 2010 test score
1st decile on 2010 test score

62

Reporting and Comparing School Performances

student outcome data: the early years
As noted previously, there are no nationally comparable measures of early literacy skills, but a range of measures is used in the states and territories. We believe there would be value in the development, in consultation with states and territories, of measures that could be used at school commencement to assist teachers with diagnosis and planning. If data were compiled nationally, both schools and employing authorities would have a better picture of schools and regions that were facing greater-than-usual challenges and might be able to direct resources to problem areas earlier and more effectively. financial resources data
We recognise that the reporting of schools’ financial resources is likely to be controversial. We also recognise that the Commonwealth-State financial arrangements are so complex that some portions of state expenditure cannot be fully attributed to individual schools. The current system was described by Dowling (2008) as ‘unhelpfully complex and exceedingly opaque’. Dowling went on to observe:
The system encourages blame shifting between governments and high level claims that the Commonwealth under-funds government schools and counterclaims that most public funding goes to government schools anyway, rather than informed debate. The end result is that members of the education community, much less the general public, have no clear idea what individual schools actually receive from both levels of government, nor if their income is appropriate to their needs. (Dowling, 2008, p..147)

Until a simpler and more transparent system of funding is implemented, there appears to be little prospect that a comprehensive method of reporting schools’ sources of funding can be devised.
Nevertheless, we consider that there is important information that could be compiled without waiting for Australia-wide funding reform. Parents are entitled to know what costs are associated with attending different schools. Taxpayers are entitled to know
(within the present limits of possibility) what they are contributing directly to these schools. They also are entitled to information that enables them to make a judgment about the extent to which the funds they contribute are directed to areas of high need.
This necessarily involves a consideration of financial resources available to schools that are in receipt of taxpayer funds.
Information about revenue that we believe could be compiled at the present time, and reported at the school level includes:
•
•
•
•

Compulsory fees and levies (by year level)
Voluntary fees6 (average, by year level)
Commonwealth government direct grants
State/Territory government direct grants

6

Voluntary fees are those that parents are expected to pay, but are not legally enforceable, and, in most cases, are not pursued by the school if not paid. Because they often vary according to the program undertaken by students, no single figure can suffice. Probably the mot appropriate figure to provide would be the average voluntary fee, calculated by dividing the total fees levied at each year level (even if not paid) by the number of students at the year level.

63

Reporting and Comparing School Performances

•

All other income

Collection and compilation of this information for all schools in Australia would be no small task, and we recognise that aspects of this information would be controversial. Complete disclosure would indicate the total revenue received per student from State Government, from Commonwealth Government, from Fees/levies, and from all other sources (eg, fundraising). human resources data
Schools differ greatly in their capacity to attract and retain qualified staff. Full disclosure of these differences would have advantages and disadvantages. Certainly the exposure of these differences might cause some distress to the employing authorities that have been unable to meet schools’ staffing requirements, and to schools that are disadvantaged by this failure. The advantage is that full disclosure increases the pressure on employing authorities to remedy shortages where they occur. On balance, we consider that the advantages of disclosure outweigh the disadvantages. A possible format for publication is shown in Tables 7 and 8.
It would be necessary to include both tables for schools that have both Primary and
Secondary schools. The year levels in the Primary School table would vary slightly, as Year 7 students are in primary school in some states/territories and in secondary school in others.

64

Reporting and Comparing School Performances

Table 7. Suggested Reporting Format for Staffing Resources Data
(Primary Schools)
Number of staff (EFT7)

Number of students Preparatory
Year 1
Year 2
Year 3
Year 4
Year 5
Year 6
Year 7 (where present)
Composite classes
Specialist teachers
Total Teaching staff
Senior Administrators (eg,
Principal, Deputy
Principal)
Teaching support staff (e.g. teacher aides, librarians)
Administrative staff (e.g. secretaries, bursars, maintenance Total non-teaching staff
Total students

7

EFT: Equivalent full-time

65

Average class size

Maximum class size

Minimum class size

Reporting and Comparing School Performances

Table 8. Suggested Reporting Format for Staffing Resources Data
(Secondary Schools)
Number of qualified staff (EFT8)

Average class size

Maximum class size

Minimum class size

English
Mathematics
Science
History/Social Sciences
Art/Drama
Other subjects
Total Teaching staff
Senior Administrators
(e.g. Principal, Deputy
Principal)
Teaching support staff
(eg, teacher aides, librarians, lab assistants)
Administrative staff (e.g. secretaries, bursars, maintenance) Total non-teaching staff
Total students

student intake data
Student intake data that could be collected in a nationally comparable way (using the
MCEETYA Data Implementation Manual, 2008) include:
 the percentages of female and male students;
 the percentage of students who identify themselves as being of Aboriginal and/or
Torres Strait Islander origin;
 the percentage of students with language backgrounds other than English;
 the percentages of students with socio-economic backgrounds defined by the main parental occupation groupings.
For reasons of privacy, serious consideration needs to be given to the numerical limits below which data of this kind should not be reported. We are not proposing a criterion for reporting at this stage, but note it as an issue that will need to be resolved.

8

EFT: Equivalent full-time

66

Reporting and Comparing School Performances

BIBLIOGRAPHY
Angus, M. (2007). Commonwealth-state relations and the funding of Australia’s schools. In L. Connors, Making federalism work for schools: Due process, transparency, informed consent pp. 112-116. NSW Public Education Alliance.
Bosker, R. & Witziers, B. (1995). School effects: Problems, solutions and a metaanalysis. Eighth Annual International Congress for School Effectiveness and
Improvement. Leeuwarden: The Netherlands, January.
Bracey, G.W. (2004). Serious questions about the Tennessee Value-Added
Assessment System. Phi Delta Kappan, 85 (5): 716-717.
Bryk, A.S. & Raudenbush, S.W. (2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage Publications.
Bryk, A.S. & Weisbeg, H.I. (1976). Value-added analysis: A dynamic approach to the estimation of treatment effects. Journal of Educational Statistics, 1, (2),
127-155.
Campbell, D.T. & Stanley, J.C. (1963). Experimental and quasi-experimental designs for research. Chicago: Rand McNally.
Choi, K., Goldschmidt, P. & Yamashiro, K. (2006). Exploring models of school performance: From theory to practice (CSE report 673). Los Angeles: UCLA,
Center for the Study of Evaluation.
Cronbach, L. & Furby, L. (1970). How we should measure ‘change’ - or should we?
Psychological Bulletin, 74, 66-80.
Dinham, S., Ingvarson, L. & Kleinhenz, E. (2008). Teaching talent: The best teachers for Australia’s classrooms. Melbourne: Business Council of Australia.
Dowling, A. (2007). Australia’s school funding system.
Council for Educational Research.

Melbourne: Australian

Dowling, A. (2008). ‘Unhelpfully complex and exceedingly opaque’: Australia’s school funding system. Australian Journal of Education, 52, 2, 129-150.
Downes, D. & Vindurampulle, O. (2007). Value-added measures for school improvement (Paper No. 13). Melbourne: Office for Education Policy and
Innovation, Department of Education and Early Childhood Development.
Drury, D. & Doran, H. (2003). The value of value-added analysis (Policy Research
Brief). Alexandria, VA: National School Boards Association.
Fitz-Gibbon, C.T. (1997). From value-added indicators to evidence-based education: the task for the next decade (Seminar Series 65). Melbourne: Incorporated
Association of Registered Teachers of Victoria (IARTV).
Flicek, M. (2004). Judging school quality using longitudinal methods that are comprehensible to stakeholders. Annual Meeting of the American Educational
Research Association, San Diego, April.
Goldschmidt, P., Roschewski, P., Choi, K., Auty, W., Hebbler, S., Blank, R., et al.
(2005). Policymakers’ guide to growth models for school accountability: How do accountability models differ? Washington: The Council of Chief State
School Officers.

67

Reporting and Comparing School Performances

Goldstein, H. (2001). League tables and schooling. Science in Parliament, 5 (2), 4-5
Goldstein, H. & Leckie, G. (2008). School league tables: What can they really tell us?
Significance, 5(2), 67-69.
Goldstein, H. & Spiegelhalter, D. (1996). League tables and their limitations:
Statistical issues in comparisons of institutional performance. Journal of the
Royal Statistical Society, 159, 385-443.
Gorard, S. (2006). Value-added is of little value. Journal of Educational Policy, 21
(2), 235-42.
Haertel, E.H. (2006). Reliability. In R.L. Brennan (Ed.), Educational Measurement
(4th ed, pp. 65-110). Westport, CT: American Council on Education and Praeger
Publishers.
Hibpshman, T.L. (2004). A Review of value-added models. Frankfort: Kentucky
Education Professional Standards Board.
Jones, R. (2004). Geolocation questions and coding index. Canberra: Performance and Reporting Taskforce, Ministerial Council on Education, Employment,
Training and Youth Affairs
Leithwood, K., Louis, K.S., Anderson, S., & Wahlstrom, K. (2004). How leadership influences student learning. New York: The Wallace Foundation.
Masters, G.N, Lokan, J., Doig, B., Khoo, S.T., Lindsey, J., Robinson, L., et al. (1990).
Profiles of learning: The Basic Skills Testing Program in New South Wales,
1989. Melbourne: Australian Council for Educational Research.
Masters, G., Forster, M., Matters, G. & Tognolini, J. (2006). Australian Certificate of
Education: Exploring a way forward. Canberra: Department of Education,
Science and Training.
Matters, G. & Curtis, D. (2008). A Study into the assessment and reporting of employability skills of senior secondary students. Canberra: Department of
Education, Employment and Workplace Relations.
Matters, G. & Masters, G.N. (2007). Year 12 curriculum content and achievement standards. Canberra: Department of Education, Science and Training.
McKinsey & Co. (2007). How the world’s best-performing school systems come out on top. Retrieved 27 November 2008.
National Centre for Educational Achievement (2008). Tapping into the power of longitudinal data: a guide for school leaders. Austin, TX: Data Quality
Campaign (DQC) and the National Association of Secondary School Principals.
NSW Auditor-General (2008). Improving literacy and numeracy in NSW public schools: Department of Education & Training. Sydney: Audit Office of New
South Wales.
Raudenbush, S.W. (2004). What are value-added models estimating and what does this imply for statistical practice? Journal of Educational and Behavioral
Statistics, 29(1), 121-29.
Rogosa, D.R. & Willett, J.B. (1983). Demonstrating the reliability of the difference score in the measurement of change. Journal of Educational Measurement,
20(4), 335-43.

68

Reporting and Comparing School Performances

Rowley, G. (2006). Value-added measures in education and training: Report to the
Department of Education, Melbourne.
Sanders, W.L., & Horn, S. (1994). The Tennessee Value-Added Assessment System
(TVAAS): Mixed-model methodology in educational assessment. Journal of
Personnel Evaluation in Education, 8, 299-311.
Saunders, L. (1999). A brief history of educational ‘value-added’: how did we get to where we are? Journal of Personnel Evaluation in Education, 12(3),
Sirin, S. (2005). Socioeconomic status and academic achievement: A meta-analytic review of research. Review of Educational Research, 75 (3), 417-453.
Taylor, J. & Nguyen, A.N. (2006). An analysis of the value added by secondary schools in England: Is the value-added indicator of any value? Oxford Bulletin of Economics and Statistics, 68(2), pages 203-224.
Teddlie, C., Reynolds, D. & Sammons, P. (2000). The methodology and scientific properties of school effectiveness research. In C. Teddlie & D. Reynolds (Eds.)
The international handbook of school effectiveness research (pp.55-133). New
York: Falmer Press.
Tekwe, C.D., Carter, R.L., Ma, C., Algina, J., Lucas, M.E., Roth, J., et al. (2004) An empirical comparison of statistical models for value-added assessment of school performance. Journal of Educational and Behavioral Statistics, 29(1) 2004, 11 –
35.
Victorian Tertiary Admissions Centre (1998).
Australian Year 12 ENTER comparison table.
Retrieved
November
27,
2008 from http://www.vtac.edu.au/future/opconversion.html
Wainer, H. (Ed.) (2004). Journal of Educational and Behavioral Statistics: Special
Issue on Value-Added Assessment. 29(1).
Williams, R.H. & Zimmerman, D.W. (1996). Are simple gain scores obsolete?
Applied Psychological Measurement, 20(1), 59-69.
Willms, J. D. (1992). Monitoring school performance: A guide for educators,
London: Falmer Press.
Yang, M., Goldstein, H., Rath, T. & Hill, N. (1999). The use of assessment data for school improvement purposes. Oxford Review of Education, 25, 469-483.
Zimmerman, D.W. & Williams, R.H. (1982). Gain scores in research can be highly reliable. Journal of Educational Measurement, 19, 149-154.

69

Reporting and Comparing School Performances

APPENDIX. SCHOOL REPORTING – EXAMPLES FROM AUSTRALIA AND OVERSEAS

Source: http://www.dcsf.gov.uk/cgi-bin/performancetables/group_07pl?Mode=Z&Type=LA&No=837&Base=b&F=1&L=50&Year=07&Phase=1

Figure A1. Example of a British School Comparison Table 2007

70

Reporting and Comparing School Performances

Source: http://www.qsa.qld.edu.au/downloads/about/qsa_stats_yr12_outcomes_07.pdf

Figure A2. Queensland: Senior Secondary School Comparison Table 2007

71

Reporting and Comparing School Performances

Source: http://www.vcaa.vic.edu.au/vce/statistics/schoolstats/postcompcompletiondata-2007.pdf

Figure A3. Victoria: Senior Secondary School Comparison Table 2007

72

Reporting and Comparing School Performances

Source: http://www.curriculum.wa.edu.au/internet/Communications/Reports_Statistics/School_Comparison_Statistics

Figure A4. Western Australia: Senior Secondary School Participation and Achievement Data 2007

73

Reporting and Comparing School Performances

Source: http://www.eduweb.vic.gov.au/edulibrary/public/voced/ontrack/destinationdata08.pdf

Figure A5. Victoria: Senior Secondary School Destination Data 2008

74

Reporting and Comparing School Performances

Source: http://www.nytimes.com/ref/education/20071105_SCHOOLS_GRAPHIC.html

Figure A6. New York City: School Report Table 2008

75

Reporting and Comparing School Performances

Figure A7. US Inner-City NAEP Results in Reading and Mathematics 2002-2007

76

Reporting and Comparing School Performances

Source: http://schools.nyc.gov/OA/SchoolReports/2007-08/ProgressReport_HS_K499.pdf

Figure A8. New York City: Example of a School Report Card 2007
(continued on next page)

77

Reporting and Comparing School Performances

Source: http://schools.nyc.gov/OA/SchoolReports/2007-08/ProgressReport_HS_K499.pdf

Figure A8. New York City: Example of a School Report Card 2007
(continued from previous page)

78

Reporting and Comparing School Performances

Source: http://www2.eddept.wa.edu.au/schoolprofile/home.do

Figure A9. Schools Online, Western Australia: Site Overview

Source: http://www2.eddept.wa.edu.au/schoolprofile/main_page.do

Figure A10. Schools Online, Western Australia: Sample School Overview

79

Reporting and Comparing School Performances

Source: http://www2.eddept.wa.edu.au/schoolprofile/main_page.do

Figure A11. Schools Online, Western Australia: Sample School Data

80

Reporting and Comparing School Performances

Source: http://schoolimprovement.education.tas.gov.au/

Figure A12. School Improvement Report, Tasmania: Site Overview

81

Reporting and Comparing School Performances

Source: http://schoolimprovement.education.tas.gov.au/SchoolInfo.aspx?School=6506

Figure A13. School Improvement Report, Tasmania: School Overview

82

Measuring and Reporting School Performance

Source: http://schoolimprovement.education.tas.gov.au/SchoolResults.aspx?School=6506

Figure A14. School Improvement Report, Tasmania: Sample School Data
(continued on next page)

83

Measuring and Reporting School Performance

Source: http://schoolimprovement.education.tas.gov.au/SchoolResults.aspx?School=6506

Figure A14. School Improvement Report, Tasmania: Sample School Data
(continued from previous page)

84

Measuring and Reporting School Performance

Source: http://www.fraserinstitute.org/reportcards/schoolperformance/

Figure A15. School Performance Report Card, Ontario, Canada: Sample
Primary School Data

Source: http://www.fraserinstitute.org/reportcards/schoolperformance/

Figure A16. School Performance Report Card, Ontario, Canada: Sample
Secondary School Data

85

Measuring and Reporting School Performance

Source: http://studata.sandi.net/research/sarcs/2007-08/SARC320short.pdf

Figure A17. School Accountability Report, California: Sample School Data
(continued on next page)

86

Measuring and Reporting School Performance

Source: http://schoolimprovement.education.tas.gov.au/SchoolResults.aspx?School=6506

Figure A17. School Accountability Report, California: Sample School Data
(continued from previous page)

87