When we discuss the standards of Italian universities, the talk is about criteria used in other countries. One of these is to look at the number of times the works of a lecturer or candidate for a post have been cited in the academic press. Some institutions provide detailed figures, and this kind of check seems at first sight a good idea. But like all quantitative measures, it has its limits. It’s rather like the idea that has also been proposed, and sometimes applied, of establishing the efficiency of a university on the basis of the number of graduates. A university that churns out a lot of graduates gives the appearance of being efficient, but it’s easy to see the limitation of such statistics. You might have a very poor university that attracts many students by giving free credits and not being too strict about the quality of its theses, so a numbers criterion would then be of negative value. What about a university that has exacting standards and prefers to produce fewer, better graduates? A more reliable criterion, though also open to criticism, would be to compare the number of graduates with the number of students who enroll at the start of the course of study. A university that has only a hundred enrollees but produces fifty graduates would seem more efficient and exacting than another that has ten thousand enrollees and two thousand graduates.
So criteria that are merely quantitative have their shortcomings. Let’s go back to the question of checking the number of citations. First let me say that this criterion may apply more to hard-science publications (mathematics, physics, medicine, etc.) than to those in the soft sciences, like the so-called social sciences. For example: I publish a book in which I show that Jesus was the true founder of Freemasonry—note that for a substantial sum, to be donated to charity, I could also provide an appropriate, up-to-date bibliography, though it would contain works that haven’t been taken particularly seriously. If, however, I manage to find some apparently reliable supporting documentation, it would cause pandemonium in the field of historical and religious studies, and hundreds of essays would appear citing my work. Let’s also assume that most of these essays cite my book in order to dispute it. Is there any quantitative control that discriminates between positive and negative citations?
What then can be said about a solid and well-argued book that has nevertheless stirred controversy and criticism, such as Eric Hobsbawm’s book on the short twentieth century, and what criteria could be used to remove all citations by those who discuss it critically? And then, would we refuse Darwin a teaching post simply by demonstrating that over fifty percent of those who cited him, and who still cite him, did so and still do so in order to say he was wrong?
If the criterion is purely quantitative, we would have to accept that among the authors most cited over recent decades are Michael Baigent, Richard Leigh, and Henry Lincoln, who have written a book on the Holy Grail that became a bestseller. They have written a pile of rubbish, but they have been, and will be, frequently cited. If the criterion was quantity alone, a university that offered them a post in the history of religions ought to leap to the top of the charts.
Such doubts in relation to the soft sciences ought to be raised in many cases for the hard sciences. Stanley Pons and colleagues shook the world of science a few years ago with a much-criticized and probably false theory on cold fusion. They have been cited endlessly, almost always to refute them. If the criterion is only quantitative, we have to take them into serious consideration. Some people might argue that in such cases the quantitative criterion should apply only to journals of serious scientific value. But leaving aside that this itself would be a qualitative criterion, what do we do if these serious journals refute what the academics have said? Qualitative criteria would have to be introduced once again. I’d like, however, to see how much criticism Einstein received when he announced his general theory of relativity, and there again, let’s take one of the most debated questions, whether what is known as the Big Bang actually happened. We know that eminent scholars have conflicting views. If a new theory appears that rejects the Big Bang, do we have to cross out all negative citations by those who still support this idea?
I say these things not because I have a ready-made solution, but to highlight how difficult it is to establish criteria of excellence on quantitative bases and how dangerous it is to introduce qualitative elements, which in the end were those used by Stalinist official culture to expel from the scientific community those who didn’t subscribe to the principles of dialectical materialism or didn’t take seriously the theories of Lysenko. Nor do I want to claim that no criteria exist. I wish only to point out how difficult it is to formulate them, and what a sensitive subject it is.
2003