image
Business Education January 2002
How to put together a 2,100-piece data jigsaw
By Della Bradshaw
Published: January 17 2002 17:30GMT | Last Updated: January 18 2002 17:30GMT
image

It takes 2,100 pieces of data to make up the chart that comprises MBA2002, the Financial Times ranking of full-time, international MBA programmes. But behind the scenes at least a million statistics are used to calculate the final table.

The biggest problem with compiling such a complex ranking is to ensure that the data is accurate and beyond reproach. With this aim, and in line with instructions given to participating business schools in 2000, the FT introduced a system of data auditing for the first time with MBA2002.

The scheme had two aims: to ensure that the data supplied by business schools to the rankings is accurate, and to help the FT in defining the questions asked, in order to avoid future misinterpretation.

Above all, the aim of the project is to improve the quality of the FT rankings, not to isolate schools who might try to cheat.

Ten schools were asked - and agreed - to participate in this pilot audit. Over the next five years all schools will be required to undertake the audit, and any school which refuses will be omitted from the ranking.

Although the idea of a business school data audit is new for newspapers, it is not necessarily new to all business schools or to auditors.

Queen's University in Kingston, Ontario, has long employed auditors KPMG to audit various data it supplies to business school rankings. Because of its experience, therefore, it was decided to use KPMG in Kingston to perform specified audit procedures at the pilot 10 schools for the FT.

A thornier issue was how to select the 10 schools. A random selection may have laid those schools open to suspicions that they had exaggerated their data. It was therefore decided to protect the pioneering 10 by selecting them from a close geographical area. Because Queen's, as veterans of auditing, was chosen as the first school for the audit, the other schools were local to it.

The geographical approach had two further advantages: it made the audits easier to complete and meant that schools throughout the top 100 table were included.

The 10 schools were the University of Chicago and the Kellogg school at Northwestern University from the top 10 of the ranking, the University of Michigan and the Ivey school at the University of Western Ontario from the top 25, three further US schools - the Broad school at the State University of Michigan, the Simon School at the University of Rochester and the Weatherhead school at Case Western Reserve - and three further Canadian schools based in the Toronto area - Queen's, the University of Toronto and the Schulich School at York University.

The audit requires participating schools to submit data on their students, faculty, overseas course content and PhD programmes - some 30 per cent of the data used in the rankings. Each audit took a day on site to complete and the FT bore the total financial cost.

Responses to the move were extremely positive from both the 10 pioneering schools and many other schools who participate in the ranking. As Deszo Horvath, dean of the Schulich School, says: "We at the Schulich School of Business welcomed the introduction of the audit as a regular part of the annual FT survey. I believe the use of auditing will be good for all of the schools involved in the annual ranking."

Other schools said the audit helped them decide how better to collect the data in future. Some non-participating schools lobbied to be included in this year's audit while others suggested they might pay for their own audit to be carried out with an approved auditor on an annual basis.

There was one overriding conclusion from the exercise, says Michelle Podhy, Assurance Partner at KPMG in Kingston. "What was clear was that there was no intentional abuse. The schools do want to get it right."

Where small errors did occur they were as likely to be a case of under-reporting as a case of over-reporting, and would have been detrimental to the school had the data been used in the ranking.

Patrick Gaudet, Assurance Manager at KPMG in Kingston, who completed the specified audit procedures on behalf of KPMG, came up with some general advice for schools participating in the rankings in future. The first was to review their submissions after they had completed the online questionnaire - small errors were often caused by mis-keying the information on the FT database.

He also expressed concerns about the procedures for data collection. In some cases, this was particularly professional, in other cases the faculty or staff had no idea why they were being asked to supply the data or what it was to be used for. "Some schools did not have formal procedures around the collection and retention of data," he says. In particular, some schools had not retained the data once it had been submitted.

Mr Gaudet says some schools expressed concerns about whether the data they supplied to different surveys and rankings was consistent. In some cases, this related to the different definitions organisations put on the different categories.

Although auditing is new to business school rankings, business schools themselves, particularly in the US, have long expressed their concerns about the consistency of data supplied to the marketplace, either through rankings or through general publicity material.

Because of this, the General Management Admission Council (GMAC), the body which administers the GMAT test, the entry examination for all top business schools, has spent the past two and half years in focus groups with its members developing a set of standard reporting criteria on issues surrounding MBA programmes. These include the GMAT scores, and the participants' ethnicity and citizenship.

Some trial audits have already been carried out and official audits will be completed this year. The FT is working with GMAC to see if there is enough commonality of data for joint audits to be carried out in future.

Of the 10 schools that participated in the pilot rankings, six moved up the table and four fell.