 |
 |
 |
 |
 |
 |
© 2004 European Consortium for Political Research |
 |
|
|
| simon hix |
||
| Rankings of departments are key information tools for central administrations of universities, public and private funders of research, and students and faculty. For example, to find out how European universities perform compared to institutions in the United States, the European Economics Association commissioned research into the global ranking of economics departments (esp. Coupé, 2003). Applying a similar method in political science to the dominant method used in the natural sciences and economics – looking at the number and impact of articles published in the main journals in the discipline in a given period – produces quite robust results for our discipline (Hix, 2004). One of the advantages of this method is that it enables comparisons to be made across Europe, between Europe and the United States, and within European countries. The next section explains and justifies the method. Section two then presents the results of an analysis of the content of sixty-three political science journals between 1993 and 2002 and the performance of European universities in the resulting rankings. THE METHODExisting ranking methods The most common method for ranking departments in the natural sciences is to look at research outputs in scientific journals in a particular discipline. This method has started to be applied in the social sciences, especially in economics (e.g. Dusansky and Vernon, 1998; Coupé, 2003). The main advantage of this method is that it is supposedly ‘objective’, since the person compiling the rankings simply ‘adds up the numbers’. Yet, in political science the most common method for ranking departments has been through subjective ‘peer evaluation’: where senior academics are asked to evaluate the quality of other departments. This method is used by the US National Research Council and the U.S. News and World Report and in the British Research Assessment Exercise. No ranking method is perfectly ‘objective’. For example, there is an element of subjectivity in rankings based on publications in journals, since what gets published depends on the subjective judgements of journal editors and article reviewers. However, editors and reviewers are experts on the subjects of the papers they publish or review, whereas the small sample of academics used in these peer evaluation methods cannot possibly be experts in all areas of research produced by the institutions they rank. Not surprisingly, rankings based on peer evaluations tend to be several years behind, and the overall reputation of the university has an effect on the respondents’ expectations of the performance of a political science department – something known as the ‘halo effect’ (Lowry and Silver, 1996; Jackman and Siverson, 1996; Katz and Eagles, 1996). There are two further disadvantages of the peer evaluation method compared to the journal publication method. First, peer evaluations are highly costly and time-consuming to organise, whereas looking at publications in journals is easy to mechanise and update. Second, peer evaluations are inevitably nationally specific, as different criteria and measures are used in different countries. In contrast, rankings based on journal publications, if constructed carefully, can allow comparisons across countries and between Europe and the United States. Nevertheless, existing applications of the journal publications method in political science have focussed on producing a ranking of departments in the United States, and have hence only looked at outputs in US-based journals (e.g. Miller, Tein and Peebler, 1996; Teske, 1996; Ballard and Mitchell, 1998; McCormick and Rice, 2001). For a truly global ranking to be produced, a much larger sample of journals needs to be analysed. An alternative approach To produce a reasonably objective and more global ranking, one should look at the quantity and impact of articles published in all the main journals in political science in a given period. Before adding up the numbers, however, several key decisions need to be made. First, what time period should be studied? Looking at one year’s worth of articles would be a too small sample in each journal. Looking at a ten-year period would not allow more subtle changes to be measured. Hence, I decided to look at publications in ‘rolling’ five-year periods between 1993 and 2002: in other words, 1993-1997, 1994-1998 and so on, up to 1998-2002. Second, what are the ‘main’ journals in political science? The 2002 edition of the Social Science Citation Index (SSCI) contains 143 journals that it lists as political science, international relations or public administration. But, several prominent international journals are not listed in the SSCI, such as the journals of the French, Italian and Dutch political science associations, and several major European-based sub-field journals, such as the Journal of Public Policy, European Union Politics, Nations and Nationalism, and the Journal of Legislative Studies. Also, many journals in the SSCI are really journals in other fields, such as law, economics, sociology, history, social policy, philosophy, or management. And, many journals in the SSCI list have only a marginal impact on the discipline. So, the ‘main’ journals can be divined as follows.
First, as many journals as possible were added to the SSCI list.
Then, a journal is excluded if either (a) it is not a political
science journal (it is not edited by a political scientist or does
not have a majority of political scientists on its editorial board),
and (b) it has only a marginal impact (there were fewer than 100
citations to articles published in the journal from the articles
in the other 8,000+ journals in the SSCI in 2002). Third, how should an article be counted? All main articles and research notes were included, and all editorial comments, book reviews, and short notes were excluded. There were 18,000 such publications in the 63 journals between 1993 and 2002. Each article was then counted as follows: an article by a single author with a single institutional affiliation, or by two or more authors from a single institution, scored 1.000 for the institution; an article by two authors from two different institutions, or by a single author with two institutional affiliations, counted as .500 for each institution; an article by three authors or three institutions counted as .333 for each institution; and so on. Observations where an institutional affiliation could not be derived from the editorial information were excluded. This left a total of approximately 24,000 single observations for analysis. Fourth, how should the impact of an article be measured? Two different
articles in the same journal may have vastly different impacts on
the field. Conversely, some articles may be fashionable or cited
simply because they are written by a well-known scholar. But, if
one assumes that a journal applies a common standard in the review
process, it is reasonable to assume that all articles in a particular
journal are of approximately equal quality. Hence, to measure the
overall impact of a journal, we can look at the average number of
citations to a journal in a given period. ANNUAL_CITESjy=ß1JOURNALy–ß2YEARjy–ß3YEAR2jy+Ejy where j (journal) = 1,…63, y (year) = 1,…10, and JOURNAL is a vector of sixty-three dummy variables, one for each journal. Estimating this model using OLS regression produces the following results (t-statistics in parentheses): ß2 = 17.944 (2.65), ß3 = .709 (.590), and sixty-three constants, ranging from a high of 882.49 citations per year for the American Political Science Review (APSR) to a low of 133.49 for the Rivista Italiana di Scienza Politica (RISP).[2] In other words, on average, a paper in the APSR had about seven times the impact of a paper in the RISP. These constants can then be used as proxies for the relative impact of each journal.[3] Using this method of measuring the impact of journals, there were thirteen Europe-based or joint Europe/US-based journals in the top thirty most cited journals in political science between 1993 and 2002: Journal of European Public Policy (9th), Journal of Common Market Studies (12th), British Journal of Political Science (13th), Journal of Peace Research (14th), Europe-Asia Studies (18th), European Union Politics (19th), West European Politics (20th), Political Studies (22nd), European Journal of Political Research (24th), Public Administration (25th), Party Politics (26th), European Journal of International Relations (27th), and Electoral Studies (29th). Fifth, how should the ranking be constructed from these observations and impact scores? People are interested in different things. For example, a prospective graduate student looking for a department with broad research interests may be interested in the total output of a department, whereas the central administration of a university may be interested in the average productivity of a department. To placate a broad set of interests, five separate rankings were calculated for each consecutive five-year period between 1993 and 2002:
The information on the size of a department was gathered from two sources: for the British universities, I used the number of full-time staff submitted in the Politics and International Relations section of the 2001 Research Assessment Exercise; and, for all other universities (including those British universities who made a submission for this section in 2001), I collected the number of full-time staff with a rank of Full, Associate or Assistant Professor (or equivalent) listed on the departments’ website in November to December 2003.[4] In other words, this includes only the number of staff in a political science department plus related institutes, or the number of political scientists in a broader department of social science. This has its limitations. First, it assumes that the number of political scientists in a particular institution remains constant, which is clearly not the case. Second, it only counts staff in political science departments, whereas we have counted research outputs by anyone from a particular institution, regardless of where they are based. RESULTSTable 1 lists the top 100 political science institutions in Europe on the basis of their output in the main political science journals in the five years between 1998 and 2002.[5] Table 2 shows the ‘rolling’ top 50 departments in the six five-year periods between 1993 and 2002. The European University Institute (EUI) came top in 1998-2002, and the EUI and Essex swapped places at the top for most periods, with the London School of Economics sneaking in above Essex in the most recent period. But, the position of the EUI is primarily a function of the small size of its faculty. For example, in 1998-2002 the EUI only came 17th and 30th globally in terms of the volume and impact of its publications, respectively. But, the EUI came 2nd and 3rd when the volume and impact scores were divided by the seventeen political scientists I could find listed on the EUI’s website in the Social and Political Science Department, the Robert Schuman Centre, and the various other programmes in Fiesole. However, simply counting the number of full-time political scientists at the EUI may not be a fair comparison with other universities, as there is a very large community of doctoral and post-doctoral researchers at the EUI who publish in journals and who list their institutional affiliation as the EUI. Hence, perhaps a fairer measure of the size of a faculty would be the total size of a research community – which would include full-time Full, Associate and Assistant Professors (or equivalent) as well as doctoral and post-doctoral researchers. But, this would be difficult to calculate for a large number of institutions. One obvious criticism is that these rankings are biased towards English-speaking countries, since universities in the United Kingdom account for nine of the top ten places, thirteen of the top twenty, and thirty-five of the top fifty. But, applying a very similar method in economics produces almost as skewed results: with six British universities in the top ten, eleven in the top twenty, and nineteen in the top fifty (Coupé, 2003). In other words, the dominance of English-language universities may simply be a reflection of the dominant position of English as the global language in all social sciences. This leads to an inevitable bias towards British universities, at least in the top twenty in both political science and economics. But, lower down the rankings, British universities are less prominent in economics than political science. However, this difference is probably explained by the fact that economics has a more established common methodology and vocabulary across Europe than political science, and is hence closer to being a truly European or global discipline than political science. But, any alleged British-bias in the European-wide rankings is not relevant when looking at the performance of all European universities on a global scale. The overall performance of European universities in the global rankings gradually improved between 1993-1997 and 1998-2002. Four European political science departments made it into the top twenty in both these periods. But, in the global rankings, the number of European departments grew from ten to thirteen in the top fifty, twenty-five to thirty-one in the top 100, and seventy-seven to eighty-five in the top 200. Also, a possible language bias in the pan-European rankings is irrelevant when using the global rankings to generate separate national rankings in Europe. Table 3 shows the rolling rankings for each country or group of countries that had at least five or ten institutions placed in the global top 500 political science institutions in every five-year period between 1993 and 2002. These rankings may not reflect the status or performance of a department within a particular country. However, they do reveal the relative global impact of departments within each country. Anyone with a cursory knowledge of the state of the discipline in these countries would recognise most of the institutions on these lists and predict the order on the lists relatively accurately. Interestingly, though, the results reveal a high level of stability over time in most countries. One institution held the top spot in every period in Austria/Switzerland (Geneva), France (IEP Paris), Ireland (Trinity), Italy (EUI), Netherlands (Leiden) and Scandinavia (Oslo), and one institution held the top spot in five out of the six periods in Germany (Mannheim) and the UK (Essex). Nevertheless, in most countries there was considerable movement below these top places, with some clear upward and downward trends over time. Two interesting examples of improvements in their respective rankings are the Central European University in Budapest and Pompeu Fabra in Spain. Again, anyone with some knowledge of these institutions would not be surprised by these results. CONCLUSIONMeasuring the research output of institutions in the main journals in the field in a given period is a less subjective method of ranking departments in political science than the most widely used current method, of canvassing the views of senior academics. This method produces a series of global, European and national rankings that seem intuitively correct in many respects and compare reasonably well with the equivalent rankings in economics. One possible criticism is that there may be an English-language bias in the global and European rankings. But, English is the international language for the publication of research in political science, as in the other social sciences and natural sciences. Hence, because of the ease of reading, publishing in, and teaching from these international journals, academics in the United Kingdom and Ireland are more likely to be integrated into the global discipline than academics in the rest of Europe. As a result, a ranking of departments using research published in the main international journals in any field may not accurately reflect the quality of departments outside the English-speaking world. More non-English language journals could be included in the analysis. However, as discussed, only three non-English language journals seem to have a significant impact on the global discipline. Hence, it may be hard to make a case for including some non-English journals while omitting others, or even for including non-English journals with low citations while excluding some English-language journals with higher citations. Nevertheless, the possible existence of an English-language bias does not preclude using this method to assess the relative performance of European universities compared to universities elsewhere in the world (and mainly in the United States). It also does not preclude using the rankings to evaluate the relative performance of departments within each European country or group of countries. A second problem is that book publications are more important in political science than in economics or the natural sciences. However, if one assumes that good departments produce lots of articles and books, and that less good departments produce fewer articles and books, then there should be a high correlation between a ranking of departments based on journal publications and a ranking based on book publications. Nonetheless, this hypothesis can only be checked if a similar ranking could be established using book publications, and the results of the two rankings are compared and perhaps integrated. Despite these shortcomings, two other advantages of the method developed and applied here are that it would be simple to mechanise, and so easy to update, and it would be easy to add other journals or books to the dataset. Hence, if the European political science community, perhaps via a committee of the European Consortium for Political Research, could agree a set of English and non-English language journals or book publishers that are the main vehicles for research output in political science globally or in Europe, it would not be too difficult to establish a mechanised system for updating the dataset and calculating new rankings every year. Criteria could also be agreed for what constitutes a full-time faculty member in political science, and each institution that wanted to be included in the rankings could be asked to provide up-to-date information about the size of its political science community. TABLESTable 1: The Top 100 Political Science Institutions in Europe, 1998 - 2002
Table 2: Rolling European Top 50, 1993 - 2002
note: '=' means that an institution is in the same position as the institution listed immediately before it. Table 3: Rolling National/Regional Rankings, 1993 - 2002
note: these are the top 10 institutions in each country-region that were placed in the global top 500 political science institutions in each period. Where a country had less than four institutions in the global top 500 in most periods it is grouped together with another country. Where a country/region had more than 10 institutions in the global top 500, only the top 10 are reported. Where a country had less than 10 in a period, only the top 4 or 5 are reported. Blank cells indicate that a country/region had less than 10/5 institutions in the global top 500 in that period. referencesBallard, M. J. and N. J. Mitchell (1998), ‘The Good, the Better, and the Best in Political Science’, PS: Political Science and Politics, 31:4, 826-828. Coupé, T. (2003), ‘Revealed Preferences: Worldwide Rankings of Economists and Economics Departments, 1969-2000’, Journal of the European Economic Association, 1:4, http://student.ulb.ac.be/~tcoupe/ranking.html. Dusansky, R. and C. J. Vernon (1998), ‘Rankings of U.S. Economics Departments’, Journal of Economic Perspectives, 12:1, 157-170. Hix, S. (2004), ‘A Global Ranking of Political Science Departments’, mimeo, http://personal.lse.ac.uk/hix. Jackman, R. W. and R. M. Siverson (1996), ‘Rating the Rating: An Analysis of the National Research Council’s Appraisal of Political Science Ph.D. Programs’, PS: Political Science and Politics, 29:2, 155-160. Katz, R. and M. Eagles (1996), ‘Ranking Political Science Departments: A View from the Lower Half’, PS: Political Science and Politics, 29:2, 149-154. Lowry, R. C. and B. D. Silver (1996), ‘A Rising Tide Lifts All Boats: Political Science Department Reputation and Reputation of the University’, PS: Political Science and Politics, 29:2, 161-167. McCormick, J. M. and T. W. Rice (2001), ‘Graduate Training and Research Productivity in the 1990s: A Look at Who Publishes’, PS: Political Science and Politics, 34:3, 675-680. Miller, A. H., C. Tien, and A. A. Peebler (1996), ‘Department Rankings: An Alternative Approach’, PS: Political Science and Politics, 29:4, 704-717. Teske, P. (1996), ‘Rankings of Political Science Departments Based on Publications in the APSR, JOP, and AJPS, 1986-1995’, State University of New York, Stony Brook, manuscript. notes
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
