Ei-iE

The negative side of counting, measuring and auditing tertiary education institutions

Governments do and can legitimately (on behalf of citizens) have expectations that tertiary education institutions will “produce public value” (Moore, 2005, in Pearman, 2009: 8). However, the current measures being used in New Zealand are harmful to quality public education.

In terms of research targets, we have seen ‘results massaged’ by tertiary education institutions wanting to improve their funding and ranking positions.

In order to increase their PBRF scores, institutions have: changed people’s job categorisations moving them from teaching to administrative categories, or full teaching to supervised teaching categories; fired ‘research inactive’ staff who often carry intensive teaching, pastoral care or management workloads; and hired teams to massage the evidence portfolios upon which each individual researcher’s outputs are evaluated.

An independent audit of preparations towards the performance-based research fund round this year noted ‘variable human resource practices’ occurring at public tertiary education institutions as part of the performance funding exercise ( KPMG, 2012).

Targets

In this exercise, every New Zealand scholar is evaluated on their ‘outputs’ (on publications, peer esteem, and contribution to research environment) in order to allocate appropriate levels of government research funding to the country’s public and private tertiary providers.

Added to that, education institutions are evaluated on their ability to secure external grants and audited with regard to reaching enrolment targets (an institution is financially penalised if it goes over the agreed enrolment target by three per cent or under by three per cent).

In recent years, the government introduced targets for completions, retentions, and progression of students. In some cases the completion targets being set are as high as 80 per cent. And only a few months ago the government set out its public sector wide ‘better public services goals’ stating:

“We want our public service to be more innovative, enterprising, driven, and focused on results. We’re delivering improved frontline services, reduced costs, and better results for you and your family” ( John Key and Bill English, 2012, Press Release).

Ineducation, the better public service targets centre on boosting skills and employment – for example, the aim is for 85 per cent of 18-year-olds to have a level two qualification, or equivalent, in 2017 (currently the achievement rate at this level sits at 67 per cent).

Those responsible for policy decisions and implementation may treat measuring and auditing as objective evaluations of how well public monies are spent, but such counting and measuring is not neutral.

The official rationale for this appears benign and incontestable: to improve efficiency and transparency and to make these institutions more accountable to the taxpayer and public (and no reasonable person could seriously challenge such objectives).

Driven by accountancy

The problem, however, is that audit confuses ‘accountability’ with ‘accountancy’ so that ‘being answerable to the public’ is recast in terms of measures of productivity, ‘economic efficiency’ and delivering ‘value for money’ (Shore, 2008: 281).

We can present PBRF, student completions, and enrolment targets as just auditing and accounting measures but, in reality, they are tools to discipline behaviour by determining what counts and what does not.

What counts in New Zealand is anything where tertiary education contributes to ‘economic growth and labour market productivity’, the dominant goals found in the current government’s strategy documents (See for example, Tertiary Education Commission Statement of Intent, 2012).

The question is – do policy advisers know what they have created and imposed upon the tertiary education sector? In Foucault’s terms, policy makers“often know what they do; they frequently know why they do what they do; but what they don’t know is what what they do does”(Middleton, 2009: 193).

The output model of tertiary education policy has a substantial number of negative effects and a wide range of, hopefully, unintended consequences. Educationalists and trade unionists forewarned governments of the unintended consequences when government officials first mooted implementing performance management techniques. However, this did not deter successive governments, and their agencies, from inflicting accounting tools on tertiary education institutions to measure non-financial activity.

It is important to examine the range of perverse outcomes from setting narrowly focused economic goals for the tertiary education sector and measuring outputs.

Effects on research

The research output measurements have influenced what researchers study and get funding for. The pinnacle of academic achievement is apparently a peer-reviewed article in an ‘A’-grade journal.

This means research solely about New Zealand has reduced, because it is less relevant to those top-ranking journals, and thus harder to get published. It has resulted in researchers abandoning work with and for community organisations, because such work often does not involve publishing peer-reviewed journal articles, but rather reports for public debate.

Moreover, many academics have sidelined ‘critic and conscience’ work - by choice or because of pressure from managers - because it cannot be measured and thus does not ‘count’.

Effects on enrolment

The government introduced student achievement targets last year and we are already seeing that institutions are limiting enrolments and only admitting the ‘right’ students – those who will definitely complete courses.

When enrolments are based on preconceived notions of who will succeed, indigenous students and ‘second-chance learners’ are the most likely groups who will miss out. We entrench already existing patterns of educational, social, and economic disadvantage. Added to this, staff are being pressured to pass students by whatever means necessary.

In some cases, institutions are doing the right things - setting up peer-mentoring schemes, pastoral care positions, and additional tutorial support. In other cases, staff  are picking up students and driving them to classes to improve completions rates.

We see managers changing final grades of students without consultation in order to improve completions and progression rates for their department. We see institutions cutting courses with persistently low completion rates without asking why there are low completion rates or if those rates are justified by the difficulty of the materials taught.

And we see institutions insisting staff let students re-sit assessments repeatedly to ensure they ‘pass’.

Such outcomes from the managerial auditing and accountability measures of institutions are too high a price to pay. They trample over legislated academic freedom and institutional autonomy, and over good pedagogy.

Effects on teaching

The imposition of a perverse management culture and instrumental linking of tertiary education to the economy in New Zealand is changing the very nature of teaching and learning. Over the last three decades, the New Zealand tertiary education sector has been driven much more to meet national, or more correctly, government objectives (See McLaughlin, 2003: 25-28; Zepke, no date: 3).

Economic benefit has become the predominantly desired outcome (Zepke, no date: 5) and the immeasurable outcomes of tertiary education have fallen to one side (See an example of this in work of Bhaskaran et al, 2007: 4).

Reclaiming responsible autonomy and academic freedom

It is crucial to push back against the rampant measuring and counting which is disciplining the daily life of education staff and creating perverse outcomes in the tertiary education sector. To do this, members of the New Zealand Tertiary Education Union (NZTEU) are creating an alternative vision for tertiary education.

We believe that a twenty-first century tertiary education sector must deliver knowledge to ensure broad social, scientific, human, and economic progress. We know that the benefits from attaining tertiary qualifications are much broader than purely individual monetary benefits.

And, in order to ensure quality public tertiary education in New Zealand, we need a funding and policy regime which achieves a balance between research, teaching, community service, and credentialing (providing degrees).

Teaching and learning considerations must be weighted as being more important, or at least as equally important, as economic considerations when deciding on who can learn and what they can study.

We also must have a system which values the voices of staff, students, and the community. And management of the sector must enable autonomy, diversity, and creativity in teaching and research.

But creating the alternative strategy is only part of the process. We are going to have to fight for our strategic vision and this is not a simple task. The real fight back would be a version of ‘civil disobedience’, a mass rejection of auditing and accounting tools to measure that which is non-economic.

In New Zealand, it would be very difficult to organise such ‘disobedience’ as our institutional leaders and many of our members now support the external validation model imposed by New Public Management and managerialism.

This means we must first work to convince all of our members that external validation in the form of a ‘score’, a ‘ranking’, or a ‘measurement’ is not who they are. That, in fact, the type of counting and measuring imposed by governments is de-professionalising academia.

Union campaign

The NZTEU is tackling this through its ‘ Speak Up for Education’ campaign, with meetings around the country with teachers, researchers, students, and community leaders to discuss the harm being caused by the current political and policy direction in tertiary education.

But our job does not stop with members. The majority of New Zealand’s tertiary education provision is publicly funded and the public has a right to know how their taxpayers’ funds are being spent. However, measuring and counting the outputs of education does not guarantee quality or sound use of funding.

So we must convince the public that they do not need output statistics to show them how well their education system is doing. Quality can be better evaluated and defended in a range of other ways.

In particular, the academic profession itself can provide quality assurance, as it has done in the past, through rigorous and transparent peer-reviewing processes and collegial governance. The public needs to realise that tertiary education has a wide range of purposes (some of which cannot be measured in a traditional accounting sense).

We also must let the public know that the professionals who work in tertiary institutions care deeply about quality teaching, learning, and research. To do this, our members are taking to the streets, the classrooms, the media, to the blogosphere, and anywhere we can in order to promote the importance of tertiary education to our society.

Finally, we need to turn policy leaders away from their current trajectory of increased counting and measuring, and tighter controls on the actions of teachers and researchers. We need to convince all political parties that economic measures are really too narrow a focus for tertiary education and other things can be counted.

Governments around the world are making statements, such as the one above, extolling the virtues of performance management in the education sector. And teachers, researchers, students, and communities across the globe are questioning the results. So what can be learnt from New Zealand’s plunge into the world of goal-setting, performance management, accountability, and auditing of the education sector?

Strategic steering and managment in New Zealand tertiary education

New Zealand is an interesting policy case study due to the speed and depth at which new governance approaches can be introduced to public sector bodies, including universities and further education providers.

New Zealand is a small nation (four million people), with highly centralised political decision-making and policy implementation. It has a unicameral political system (one house of parliament), no federalism, and no activist judiciary. This means power is concentrated in our 20-person executive.

There is a single public service, so decision-making centres around small policy communities which can be - and currently are - tightly directed by the political party in government.

Tertiary education profile

In tertiary education, two central agencies are responsible for policy and funding decisions for the sector. They are the Tertiary Education Commission (TEC) and the Ministry of Education. And there are also two agencies responsible for quality assurance: the New Zealand Qualifications Authority and Universities New Zealand (the vice-chancellors of our eight universities).

The size of the policy community and unicameral political system results in strong coherence between political decision-making and policy implementation. In the case of tertiary education, this coherence is achieved via the government’s Tertiary Education Strategy, which identifies priorities for the sector through the Statement of Tertiary Education Priorities (STEP).

Individual institutions then outline how they will address these priorities through their investment plans (negotiated with government representatives), which must reflect their institutional profile (for more details, see OECD 2006).

The outcome of these processes and these institutional arrangements is “an obvious alignment of the tertiary education sector with the country’s objectives” (OECD, 2006: 40).

This strategic steering has been imposed on the tertiary education sector as part of three decades of continuous and unrelenting change in the way tertiary education is managed, funded, and delivered.

The changes begin with the adoption of neo-liberalism, or new right policies, which governments introduced rapidly and deeply across New Zealand state and society in the 1980s and 1990s.

For New Zealand’s tertiary education sector, this meant the creation in legislation in 1989 of a single ‘sector’ encompassing all post-compulsory education, and the placement of for-profit and public education institutions on a ‘level playing field’.

1980s mass-market approach

We moved in the 1980s from an elite higher education system, where a small number of students received near full funding to complete tertiary qualifications, to mass market and the introduction of fees and student loans. One effect of the move to a mass market model was the significant increase in enrolments in higher education.

In 1990, 20.5 per cent of 18- to 24-year-olds studied at tertiary education institutions; by 1998, this had risen to 31.9 per cent ( Ministry of Education, 1998: 23).

By 2000, the mass-market model had become a problem for successive governments. There were rising costs because more people were studying and a proliferation of courses and providers, as tertiary education institutions competed in the new ‘open market’ for students and the government funding attached to each student.

Time lag in benefits

Current modes of measuring and counting educational outcomes are highly problematic, if for no other reason than timing. Many of the things we think are important outputs of tertiary education, such as creating active citizens and a more socially just world, will take 10, 20, or even 30 years to come to fruition.

What New Zealand governments want, in contrast, is evidence that their policy levers are leading to never-ending annual improvements in educational outcomes, or at least improvements that fit with the three-yearly election cycle.

So it will not be easy, but it must be done. Performance management, output measurements, and external auditing processes will be the death of quality public education in New Zealand, the death of open access to education, and the death of universities being critical spaces in our democracy – it is a fight we must take on.

Government response

The government’s response was to introduce ‘strategic steering’ of tertiary education. A newly formed government agency, the Tertiary Education Advisory Commission (TEAC), recommended more active engagement by government in the tertiary education system, including policies such as capping student numbers, targeting funding, and funding institutions based on differentiation and the creation of strategic investment plans for each institution.

TEAC was responding to the perceived lack of direction in the sector, the result of which was seen as inefficient use of funding (OECD, 2006: 135).

This concern with the inefficient use of resources led to the introduction of New Public Management techniques to ‘measure’ whether or not tertiary institutions were spending public monies well. “The traditional professional culture of open intellectual enquiry and debate has been replaced with an institutional stress on performativity, strategic planning, performance indicators, quality assurance measures and academic audits (Olssen and Peters in Shore, 2010: 16).

The new ‘auditing’ and ‘accountability’ measures ranged from those focused on research outputs to evaluating student outcomes. A performance-based research fund(PBRF) was introduced in the mid-1990s.