Possible Difficulties in Evaluating University PerformanceBased on Publications Due to Power Law Distributions : Evidence from Sweden

Abstract: Measuring the research performance of a university is important to the universities themselves, governments, and students alike. Among other metrics, the number of publications is easy to obtain, and due to the large number of publications each university produces during one year, it suggests to be one accurate metric. However, the number of publications depends largely on the size of the institution, suggesting, if not addressed, that larger universities are better. Thus, one might intuitively try to normalize by size and use publications per researcher instead. A better institution would allow individual researchers to have more publications each year. However, publications, like many other things, might follow a power-law distribution, where most researchers have few, and only a few researchers have very many publications. These power-law distributions violate the assumptions the central limit the orem makes, for example, having a well-defined mean or variance. Specifically, one can not normalize or use averages from power-law distributed data, making the comparison of university publications impossible if they indeed follow a power-law distribution. While it has been shown that some scientific domains or universities show this power-law distribution, it is not known if Swedish universities also show this phenomenon. Thus, here we collect publication data for Swedish universities and determine whether or not, they are power-law distributed. Interestingly, if they are, one might use the slope of the power-law distribution as a proxy to determine research output. If the slope is steep, it suggests that the ratio between highly published authors and those with few publications is small. Where as a flatter slope suggests that a university has more highly published authors than a university with a steeper slope. Thus, the second objective here is to assess if the slope of the distribution can be determined or to which extent this is possible. This study will show that eight of the fifteen Swedish universities considered follow a power-law distribution (Kolmogorov-Smirnov statistic<0.05), while the remaining seven do not. The key determinant is the total number of publications. The difficulty here is that often the total number of publications is so small that one can not reject a power-law distribution, and it is also impossible to determine the slope of the distribution with any accuracy in those cases. While this study suggests that in principle, the slopes of the power-law distributions can be used as a comparative metric, it also showed that for half of Sweden’s universities, the data is insufficient for this type of analysis.