|Karen Horn| *
There is virtually nothing that cannot be ranked. From the performance of football clubs to vintage wines, from bestselling books to competitiveness and the degree of economic freedom in different countries – it is easy to put together an ordinal scale, so that everyone can see who is on the top and who is at the bottom. And most people love rankings. They provide orientation and make life easier. In football, for example, rankings help us to see immediately who is best. Success is simply defined as winning a match, and that is perfectly reflected in the score. That’s straightforward. In the book market, however, things are not so evident. Bestseller lists don’t help us much. True, we don’t have to read all available books to find out which is on the top, we just look at the list. But “bestselling” doesn’t necessarily mean “best” in terms of literary quality or whatever else one might be hoping for; the “wisdom of crowds” can rarely reach that far. Rankings are only as good as their criteria, and they merely measure what is measurable. This is the reason why, when it comes to academic quality, rankings can be completely beside the point. Worse, they might even set perverse incentives.
Let’s look at the case of economics. Rankings abound – in a vast variety, ranging from the international list of top authors registered with the RePEc (Research Papers in Economics) platform (https://ideas.repec.org/top/top.person.all.html) to the Guardian’s “league table in economics” (http://www.theguardian.com/education/ng-interactive/2015/may/25/university-guide-2016-league-table-for-economics), focusing on universities. Many of them are published by newspapers. The presence of such a large number of rankings, each with a different philosophy, already indicates how difficult it is to measure academic excellence in economics. The first problem comes with picking the relevant criteria, a choice often narrowed by limitations in practical measurement. What should a ranking focus on? Should it focus on impact, which it would be almost impossible to isolate? Or on influence, which is to a large extent in the eye of the beholder and extremely hard to capture?
Most rankings look at academic excellence, claiming to increase comparability and foster a spirit of competition. Excellence is measured by some more or less objective indicator. But here comes the second problem. It is a problem of focus and scope: where exactly is excellence important? Is it in research alone? What about teaching? What about giving economic policy advice, engaging actively in the transfer of knowledge between academia, the public, the media and politics? Is there an ideal combination of these different activities, which would justify applying standard weights to them? Why not allow for individual specialisation? If someone is very productive publishing theoretical papers in specialised economic journals but hardly ever appears in the media, like Ernst Fehr at Zurich University, one of the international figure-heads of behavioural economics, that should certainly not diminish his being acknowledged as an excellent economist.
There is only one logical consequence: One must wave good-bye to the idea of an overall ranking, breaking it up into a set of special indexes to which the interested public may refer according to its specific need. General rankings compare the incomparable. One can’t have the cake and eat it. Either complexity is maximally reduced at the cost of producing meaningless or even misleading rankings, or specialisation is taken into account and complexity reintroduced through the back door.
Once the criteria are defined, however, the next problem comes up with operationalisation and measurement. Good luck with measuring excellence in teaching: objective methods would refer to the number of online postings or email backings-and-forthings between the teacher and his students or any other similarly arbitrary item. Subjective student evaluations however are notoriously biased. Bad marks for the teacher may sometimes be the result of a very demanding course, which however, from a pedagogical point of view, is a rather good thing. Measuring influence is not much easier. In some cases, a data base has been generated by asking a panel of experts in public administration or a random public about names in economic science that they have heard of. This is as uninteresting as it is circular. The economists who appear regularly on television and thus influence public opinion are known well enough. What remains is excellence in research, which however must be looked at in its own right, not as a proxy for the rest. Research excellence can indeed help to improve someone’s performance as a good teacher, but some talent is needed as well. There is no reliable correlation.
Fortunately or unfortunately, however, economics isn’t physics. Economists don’t deal with natural laws that only need to be deciphered, but with an ever changing social reality and human perception. Economic findings therefore cannot so easily be classified as “correct” or “incorrect”. The methods used may be apt or not; the specification of a model may be fitting or not; the data may be complete or incomplete; the implications drawn may be convincing or unconvincing. In many cases, only time – and repeated testing by the peers – will tell. Nevertheless, most rankings try to grasp research excellence by counting the number of publications within a given time – as if quantity meant quality! Focussing on quantity just reinforces the pressure to publish: “Publish or perish”, as a phrase from the Thirties goes. If there is – quite plausibly – such a thing as diminishing returns in economic science, though, like almost everywhere else, quantity at some stage must drive down quality.
Some rankings aim to remedy this problem by applying different weights to the journals in which the papers are published. For example, it is considered more prestigious to have a paper in the American Economic Review (AER) than in Kyklos (ever heard of?). The academic community distinguishes A, B, and C journals. With the corresponding weights attached, a paper in the AER then counts more than one in Kyklos. This looks alright at first sight – but only at first sight. That the AER is considered an A journal doesn’t mean that all papers in the AER are also A papers. The distribution of quality within a journal is usually very uneven. Would it make more sense to look at citations, i.e. at ex post instead of ex ante indicators, as some economists claim? Absolutely: if a paper is truly excellent, it will find recognition and spur further research, which means that many people will refer to it explicitly. And yet, there remains a caveat which has to do with the implicit incentive for citation cartels. “I quote you and you quote me” is indeed already a typical deal in academia. Considered in full light, citation-based rankings may say more about the network of a researcher than about the quality of his work.
The practical impact of rankings is as yet unclear. One would hope that the spirit of competition that rankings allegedly seek to foster won’t simply increase superficial short-termism. True scientific innovation and depth may likely take a few years of profound reflection and experience, without much written output, and universities should be wise enough to make this possible for their faculty. And one would also hope – and trust – that universities investigate into the capabilities of their potential staff in a much broader and at the same time more detailed way than rankings do. There remains a worry that rankings may create a public reputation which may be difficult to ignore even within academia. But perhaps the most problematic effect is that they may foster an academic mind-set already biased in favour of publishing instead of teaching – to the detriment of students.
-------------
*This is a guest column by Dr. Karen Horn -- one of the leading voices sound economics throughout Europe. Dr. Horn for many years was the economics editor at Frankfurter Allgemeine Zeitung and was the Director of the Berlin office of Institut der deutschen Wirtschaft; she currently teaches history of economic thought at Humboldt University in Berlin and is the editor of Perspektiven der Wirtschartspolitik, a German scholarly journal published by the Verein für Socialpolitik. Here is a fantastic talk she gave at the European Students for Liberty Conference in 2014 and her wonderful introduction and overview of Prof. Israel Kirzner's contributions when Kirzner was honored in Germany this past year.
Comments