Berkeley study: major university rankings may be biased

Global university rankings may be distorted by the business activities of the ranker, including selling consulting, analytics and other services to universities, according to a new study from the University of California (UC), Berkeley’s Center for Studies in Higher Education.

The results suggest that Russian universities frequently using services provided by QS – a major ranking company – improved their positions in the QS World University Rankings over time, regardless of improvements in their institutional quality.

According to UC Berkeley, the study provides the first-ever empirical evidence of how business relationships between rankers and universities affect rankings.

The study by senior researcher Igor Chirikov assesses the impact of conflicts of interest, when rankers not only evaluate universities but also provide them with commercial services.

University rankers claim that conflicts of interest do not affect the rankings process and that internal procedures are in place to prevent consulting or advertising services from influencing ranking outcomes, the study says.

However, the study questions that assertion after matching data on the positions of 28 Russian universities in the QS World University Rankings from 2016 to 2021 with information on contracts these universities had for services from QS.

It compares the fluctuations in QS rankings with data obtained from the Times Higher Education (THE) university rankings and data recorded by national statistics. Changes in rankings were compared between universities that frequently contracted with QS for services, and those that never or seldom contracted for QS services.

The study says the results show that universities with frequent QS-related contracts experienced much greater upward mobility in both overall rankings and in faculty-student ratio scores over five years in the QS World University Rankings.

Positions of Russian universities that had frequent QS-related contracts increased on 0.75 standard deviations (approximately 140 positions) more than they would have increased without such contracts.

In a similar way, QS faculty-student ratio scores of Russian universities that had frequent QS-related contracts increased on 0.9 standard deviations more than they would have increased without frequent QS-related contracts. Taken together, these findings suggest that conflicts of interest may produce significant distortions in global university rankings.

“This is the first empirical evidence to date that rankers’ conflicts of interest may negatively impact outcomes in university rankings,” says Chirikov, the author of the study.

“The results are novel for higher education, but they are consistent with the numerous studies from the other sectors of economy suggesting that conflicts of interest lead to biased evaluations.

“Most likely, rankers that depend on universities for resources are vulnerable to an unconscious self-serving bias also reported by the studies of the conflict of interest among auditors. Self-serving bias may lead to a more favourable consideration of the data coming from universities that are also frequent clients as compared to other types of institutions,” Chirikov said.

QS World University Rankings was chosen as the focus over other rankings because QS offers a wider array of services than other rankers, including a fee-based rating system Q Stars, but also because it had more contracts with Russian universities than other rankers and because it had been “frequently called out by media outlets and higher education experts for having not reported conflicts of interest”.

In the study Chirikov argues that the structure of QS activity suggests a greater dependence upon universities than other rankers such as THE and US News & World Report, which earn revenue from substantial subscription-based content.

He says QS Stars is one of QS’s most controversial products due to its high potential for creating conflict of interest. This involves an assessment of the quality of the university measured against eight categories such as teaching, research and internationalisation and the award of 0-5+ stars, a result which is displayed at the university profile on the QS website and in the ranking tables.

Universities are required to pay for the initial audit as well as an annual fee that allows them to use QS Stars graphics in their promotional material, the study says.

Rankings team ‘functionally independent’

When asked to comment on the study, QS spokesperson Tim Edwards told University World News that QS is confident in the robustness of its approach, which is “underpinned by well-established governance and quality assurance processes, which ensure that our rankings operations and team are functionally independent”.

He said QS’s transparency extends to ensuring that in-depth detail of its methodologies and data definitions for all of its ranking exercises is made publicly available, and the company welcomes feedback and scrutiny.

“We are mission-driven with integrity at the core of both our values and of our business model,” he added.

“Our role as a trusted partner to the sector for over 30 years – currently serving more than 1,300 universities in more than 50 countries – is testament to the strength of our integrity in all our activities which serve the sector.”

QS describes itself on its website as “the world’s most popular source of comparative data about university performance”, with 149 million views of its website in 2019.

Credibility based on perceived impartiality

The study says the credibility of the rankings is based on a belief that rankers provide impartial information for prospective students, university administrators and policy-makers. It says it is implied that all universities are evaluated equitably or at least according to a uniform set of criteria.

“However, rankers face conflicts of interest when, in addition to objectively evaluating universities’ performance, they offer these universities fee-based analytical, consulting and advertising services.

“The conflicts of interest potentially interfere with the objectivity of measures used in rankings and may provide some universities advantages that are not related to their institutional quality,” it says.

The study warns that biased measures could misinform prospective students, universities, governments and funders about global standings of universities and countries. It advises not to use global rankings where conflicts of interest may be generated by the ranker’s business activities.