Adam Finch

Introduction

One of the challenges facing organisations manufacturing or using a product is measuring its quality and comparing it with that of competitors in a standardised way; although it is different from normal commercial products in many respects, the same is true of research. In most cases, the primary initial output of research takes the form of journal articles, conference proceedings and books; these publications then become the focus of evaluation. Authors may need to prove the importance of their work when applying for a new post; institutions might wish to demonstrate and publicise their strengths; funding agencies seek to support the strongest research and evaluate the impact of the funding; and Editors can build reputations by improving their journal. Indeed, for research, the importance of indicators of quality is twofold; not only are authors producing research, they are incorporating the research of others into their work and so need a way to identify crucial islands of work in the sea of information. As it is a priority for their key customers and clients, publishers too are increasingly focused on the best way to demonstrate the usefulness, and therefore the value, of the research they publish.

Fortunately, there are measurable elements associated with publications that have been used to analyse their quality. The best and longest established example is counting citations (Gross and Gross, 1927). When one publication includes another in its reference list, this is a citation. As research tends to build incrementally, each new conclusion based on and extending the established body of knowledge, these citations are taken as recognition of value in the cited work. The more a work is cited by other publications, the more valuable it is.

With the online revolution, usage too became a viable option. When the online version of a work is accessed, this can be counted like the hits measured for any page on the web. Less significant research may attract chance usage based on the relevance of the title or an author involved, but more significant work is likely to be more popular and accessed a great many times. It has been argued that usage data are in some ways more useful than citation data as they provide a more current picture of research importance (Bollen et al., 2009).

Analysis of the content of a book or journal can also be a useful third approach. If articles or chapters dealing with a specific topic or from a certain author or institution are becoming more common, that may indicate a rise in the importance or acceptance of an idea or researcher. The study of the measurable elements of publications – citations, usage and content – is referred to as bibliometrics, although the term is most commonly used in publishing to refer to citation analysis.

There have been moves to use other data to evaluate research but these are not yet as widely accepted. Counting patents, for example, has been proposed, but as different countries and regions have different patent laws and databases, establishing counts and coverage is difficult. Features tracking references to research in social media and blogs have been introduced by some publishers (PLoS One, 2009) but this approach is too recent to have become mainstream and remains more open to manipulation than other methods. This said, citations and usage are not without their problems either.

Quality, impact and popularity

It is tempting to use terms such as quality, impact and popularity more or less interchangeably; however, there are subtle but important differences. Citations, for example, can only demonstrate the impact of a work on a subject area, not the rather more subjective property of quality; a citation can be made to a work that is poorly written or presented, or recognising value in only part of the research presented. Moreover, citations may be made in contention with the cited work. Usually, it could be argued, this ultimately helps to establish consensus on a certain topic and so is still a valuable contribution to knowledge. In all cases, however, impact rather than quality is being measured.

Likewise, usage shows popularity rather than quality. A certain proportion of all usage will be based on the title, author name or the source in which the work appears, and just because a reader looked at an article does not mean they found it useful or important. Thus one might take impact and popularity as measuring elements of quality but not the whole.

Both citations and usage also have limitations and drawbacks as the foundations of metrics. Citations are inherently retrospective; it may take an author several years to read a work, incorporate it into their research, write an article addressing the topic and have this article published. Furthermore, the databases that index these citations tend to cover more journals in the US and Europe than elsewhere (Harzing and van der Wal, 2008), and coverage of the literature across subject areas varies wildly. The average number of citations in each subject area also differs, being more numerous in medicine, for example, than in mathematics. Self-citation, where authors or journals preferentially cite their own work to inflate their performance, may also be an issue (Van Raan, 2008). Many citations contain bibliographic errors and some are made without the cited work being read (Simkin and Roychowdhury, 2003). Some authors even question the underlying assumption that citations can be used to fairly measure impact (Todd and Ladle, 2008).

Usage data are more problematic still. Unlike citations, no central or neutral organisation counts usage for articles that are published. Although the COUNTER project has provided an immensely useful standard for what is counted when a user accesses an article (Shepherd, 2002), different publishers vary in their treatment of usage aggregated through third-party hosts such as Ovid. Usage counts can also be affected by changes in the indexing policies of Google and other search engines, making fair comparison over time impossible. Usage looks only at the electronic version of an article, ignoring print usage. It is also easier to manipulate than citations because all that is required is a click of a button rather than the publication of a citing article. Last but not least, publishers are also reluctant to yield their usage data, as they could provide business intelligence to competitors. Although work on a Usage Factor is underway (Usage Factors, 2010), no journal usage metrics have yet been produced for a significant proportion of titles across publishers.

In any case, neither usage nor citations provide us with a complete picture of research impact. The former is an indicator of where useful research was sought, while the latter only shows the subset of instances where a researcher’s use of the research was finally published. The situation was perhaps best elucidated by Carl Bergstrom: ‘Usage data tell us where the net was cast; citation data tell us where the fish were caught’ (Butler, 2009). Both approaches are utilised far more frequently with journal articles than with books. At the time of writing, only selected book series and collections of conference proceedings are included in either of the main citation indices and usage counts are most commonly provided by publishers for their journal products.

It is quite possible to undertake useful citation and usage analysis of book titles where the data exists, but for the aforementioned reasons, the commonly used metrics in publishing apply to journals and count citations rather than usage. It is these metrics on which this chapter will focus; but before metrics can be calculated, publications and citations must be recorded in a citation index.

Citation indices

There are currently three major, international citation indices available, each with different coverage of published research.

Journal impact metrics

With the increasing importance of citations in evaluating the impact of research, it was natural that metrics would be devised to measure the citation performance of different journal titles. A great number of metrics of varying degrees of sophistication have now been proposed, with their merits and demerits debated extensively; however, only relatively few have been calculated for a significant proportion of journals and made available over a number of consecutive years. These are the most useful from a publishing perspective as they can be used both for comparative analysis and publicity.

Backlash against citation metrics

As the application of journal citation metrics has grown, so too has opposition to the practice. It is held by some that, far from providing a useful guide, the chasing of high impact has begun to damage research itself.

In some cases, objections are made to specific attributes of the metrics, which make them unsuitable for measuring science. For example, journals publishing a large number of non-citable items, or a small number of very long citable items, may have an unrepresentatively high numerator and a low denominator in metrics that look at average citations (Campbell, 2008). Journals in fields with lower average numbers of citations, such as mathematics or social sciences, can be unfairly discriminated against if funding decisions are based on one of the majority of metrics that do not account for variations in subject area.

Some criticism has been levelled at editors and publishers; for example, it has been suggested that focusing on big-name authors to attract citation creates a barrier to up-and-coming researchers (Tsikliras, 2008). Another criticism argues that clinical articles and case studies, which tend to receive fewer citations (Oh and Lim, 2009), might be avoided despite their importance to a journal’s community.

Other issues have been raised with the application of the metrics. Chinese science authors have been instructed to publish articles only in journals appearing in the Science Citation Index, and which therefore receive an Impact Factor (Al-Awqati, 2007). Authors from Imperial College London have had the Impact Factor of the journals in which they publish incorporated into a Publication Score (Colquhoun, 2007). In Brazil, Impact Factors are used to evaluate graduate programmes (de Albuquerque, 2010). There are similar reports of the Impact Factor being used in Italy, Japan and Spain in a similar fashion (Cameron, 2005).

This is particularly problematic; the impact of a journal cannot be used to generalise about the impact of the authors publishing in it because the distribution of citation strength within a journal is highly skewed. In some samples, the most cited 15 per cent of the articles account for 50 per cent of the citations, and the most cited 50 per cent of the articles account for 90 per cent of the citations (Seglen, 1997); in some cases, the skew is even greater (Campbell, 2008).

These concerns have been part of the driving force behind innovations in article-level metrics. Some publishers have begun giving the usage and citation counts for individual articles, allowing authors to report these rather than focusing on journals. Although usage data are incomparable between publishers, owing to variations in what is counted and how, the citation data supplied by CrossRef allow meaningful comparison of citation impact at an article level.

Strategic journal development

From a publisher’s perspective, it is crucial to maximise a journal’s impact, whichever metric is used to measure it. Authors will often consider Impact Factors when deciding where to publish their work and citation performance may well be one of the elements in a librarian’s decision over whether to subscribe to a publication. When working with society-owned titles, retention of contracts may depend heavily on maintaining a strong citation performance. Publishers are therefore compelled to maximise the impact of their publications as much as authors and institutions.

Journal Editors may, of course, have other priorities for a title than maximising its Impact Factor, such as service to the subject community. Moreover, as already noted, some of the tactics used to maximise impact can end up unduly influencing the format or focus of research. Publishers should be sensitive to these considerations and seek to increase impact through citation analysis only when their efforts do not debase the knowledge they seek to make available.

In the future, the metrics that weight citations by the impact of the citing journal may become more popular, in which case it will be important both to attract a high volume of citations and that these citations come from high-impact journals. Currently, however, the Impact Factor is the most widely recognised and prioritised measure, so the key aim of publisher citation analysis is to attract more citations per article for the journal.

Before this is possible, there are some technical constraints to overcome. As previously noted, WoS and Scopus both provide two sets of data, one with bibliographic information and the other with citation counts by year. It may therefore be necessary to join the two sets of data using a database or a program like Microsoft Excel. Address data are also usually combined into one field, so it may be worthwhile separating out the different lines to enable study of institutions, countries and world regions. Some publishers have created tools to automate these processes, which saves a great deal of manual work.

When studying the performance of a journal and working to improve it, it is often best to download several years of articles and their associated citation data, so that trends over time can be established. It is also advisable to acquire records for similar or competitor titles, so that strengths, weaknesses, opportunities and threats can be identified and results can be viewed in proper context.

There are numerous ways to break down and study article data. While a top article list may allow someone familiar with the subject area to spot patterns in what is best cited, more transparent approaches are often effective. Aggregating the articles by author can allow identification of the top names in a field. Articles or reviews could then be commissioned from these individuals or they could be recruited to a journal Editorial Board, to attract papers from their networks. The same can be done for author affiliation, identifying institutions with strong publishing records in certain areas. It should be noted, however, that there will usually be only a few articles for each author or institution, and care should be taken to avoid reaching strong conclusions based on a small sample size.

If country and region data can be extracted from the bibliographic records, these can often provide useful guidance. While all research from a certain country in a field will not be of a similar level of citation strength, analysis by country can allow the identification of areas where publication frequency is growing or where citation strength is trending upwards. Extending a journal’s scope or Editorial Board to draw in more papers from other countries can increase the chances that a high-impact article will be submitted to the journal.

More simply, journal strengths and opportunities can be identified through a breakdown of the document types published. It may become apparent that a journal’s articles are well cited but its proceedings papers are not; or that a great many citations are attracted by editorials and relatively few by reviews, in comparison with competitor titles. The decision could then be taken whether to decrease the numbers of weak document type articles published, or to try to improve them by looking at what makes a certain document type successful for other journals. This approach can also work when comparing special issues with normal issues; because each citable item contributes to the denominator of an Impact Factor, it is important to ensure that special issues, if published, are attracting as many citations per article as normal issues.

One of the more useful but commensurately difficult analyses often requested is a study of which topics within a field are more or less cited. Although some subject areas, such as chemistry, have an existing taxonomy of subjects with the tagging of indexed articles to allow easier analysis, this is rare. It is tempting simply to look up the citations for a single keyword, but the citation performance of a single keyword is unlikely to be representative of a topic; it is more reliable to aggregate keywords describing topics into clusters and then study the average citation performance of these clusters. Without degree-level experience of the journal topic, however, the assistance of a journal’s Editorial Board may be required in establishing topic keyword lists.

There are some simpler tricks to improving Impact Factor performance that do not involve debasing the research presented. Citations count towards metrics like the Impact Factor on a calendar year basis, so that a 2012 article would contribute to the 2013 and 2014 Impact Factors regardless of publication month. However, publishing an issue in January 2011 rather than December 2012 will mean its articles have 36 rather than 24 months before the end of this ‘citation window’. These extra 12 months can increase average citation levels significantly. For this reason, publishers often load the early issues of a year with more papers, or fill them with articles written by well-respected authors who are more likely to be cited. However, the most honest, the most effective and the least damaging way to increase an Impact Factor is to identify and publish better research with a higher potential for citation.

WoS additionally allows users to search for and download the citations themselves, along with the data of the articles that made them. This can allow analysis of which authors, institutions and countries cite a particular journal most frequently, which can feed into marketing and sales efforts. More usefully still, it is possible to use these data to check, predict and simulate Impact Factors ahead of their formal release. In the rare case that citations have been missed from the JCR calculations, it is possible to alert Thomson Reuters to the issue and request correction in the September re-release of the JCR. Impact Factor simulation for non-indexed titles also provides a useful guide when applying for coverage in WoS, as a basic level of citation activity is required for admission.

The ‘early view’ effect

Most major publishers now make copies of their articles available online before they are incorporated into an issue and given page numbers. Certain subjects also have preprint online repositories where the text of an article accepted for publication (but prior to formatting by a publisher) can be hosted; one example of this is the arXiv.org repository for Physics, Mathematics and Computing. Making a version of an article available online ahead of print publication in this way may increase the citations received by those papers before a certain deadline; they are online earlier, more likely to be read and therefore more likely to be cited. The citations are, in essence, brought forward and made sooner (Moed, 2007); this is the early view effect.

Citations from these papers are only counted by WoS when the final paginated version is indexed, but citations to these papers made before that date can theoretically be recorded and added to the article record when it appears in an index. For such a citation to be counted on WoS, it must contain the DOI for the cited work.

However, even if the CrossRef DOI is not given and the citation is consequently not attached to an article’s entry on WoS, the citation may still be counted as part of an Impact Factor calculation. This is because the only information sought when identifying citations for the Impact Factor are the cited journal title and the year of the cited article. No attempt is made to verify that the target article has been published in the given year. This means that the effect of making an article available online early is, ideally, to extend the window of citation for that article by the duration of the lag between the early view version becoming available and the paginated version being published.

For example, citations to an article published in December 2011 will be counted towards the 2012 and 2013 Impact Factors. That means a citing article must be written and published within 24 months of the target article becoming available in December or the citation will not contribute to an Impact Factor. In many subject areas, this is too short a window and citations are ‘lost’. However, if the target article was made available for early view in December 2010, the ‘window’ during which citations will count is extended to 36 months, theoretically increasing the likelihood that the target article will be cited.

There are arguments against this theory. It relies on taking an article in isolation; if the target article was available 12 months earlier, there may have been other articles available then that citing authors would have chosen in preference to it. Citations exist as part of a network of knowledge and altering one element may change the whole. It is also possible that citations to the article will be given as ‘In Press’ or that authors will continue to cite the version under the early view date (2010). Moreover, it is very difficult to test the early view effect, because it is impossible to know how many citations an article would have received if it had not been available for early view. Work on arXiv has found both an early view effect and a strong quality bias in what was submitted to the repository (Moed, 2007). Only through a controlled trial, with articles randomly made available online early or not, would it be possible to test scientifically whether early view increases citation counts; we can, however, be reasonably confident that it does not lower them.

Open Access (OA) and citations

One hotly debated issue in citation analysis is whether making articles freely available increases the numbers of citations made to them. The theory goes that if an article is available to any reader online, the pool of potential citing authors will be larger and that a proportion of these additional citations will be indexed, benefiting authors and the journals in which they publish. Indeed, early studies (Antelman, 2004; Hajjem et al., 2005) showed a correlation between articles being freely available and higher levels of citation; the authors of some of these studies held that there was a causal link between an article being free and receiving more citations (Harnad and Brody, 2004).

Subsequent examinations of the subject found methodological problems with the early studies, such as imbalances in the size of OA and non-OA samples and the use of inconsistent windows for citation counting (Craig et al., 2007). Moreover, higher levels of citation could have been caused by other factors, such as the aforementioned early view effect or selection bias – that authors would only make their best work freely available online or that better authors tended to self-archive (Moed, 2007). Another study, by Davis (2008b), sought to eliminate selection bias by randomly selecting which articles were made available via OA and then checking citation counts over time to see if there was a difference between the two groups; this study found that there was no OA advantage after one year.

Further research by proponents of OA citation advantage compared articles that were made available online by choice of the author with those that were mandatorily made free, for example as part of requirements by a funding agency, but did not randomly select which articles were made OA (Gargouri et al., 2010). Criticism had also been levelled at the Davis study that one year was too short a time for OA advantage to become apparent (Harnad, 2008).

Davis has recently updated his results to show that there is still no OA advantage three years after publication (Davis, 2010), but this has not been accepted by the proponents of OA, whose results suggest there is an OA advantage for both mandated and self-selected OA articles (Harnad, 2010). The two conclusions seem contradictory, although there is a difference between institution- or funding body-mandated OA and random selection; it could be argued still that articles published as a result of funding body grants are more likely to contain significant and citable conclusions. While neutral parties seem to be satisfied with Davis’ approach, it has not been suggested that his results can be generalized to the entire journal population. It would take a wide range of large samples for one to be confident that any result could be generalised to all journals. However, the approach of randomly assigning OA status gives a view of the data that is the most free from confounding variables so far. While OA citation advantage cannot be considered to have been comprehensively disproved, it has now been called seriously into doubt.

Author metrics

Although publishers are primarily concerned with metrics relating to journals, many key stakeholders are also authors. As journal metrics have grown in importance over recent years, so too have author metrics. The first metric to gain significant currency was the H-Index, proposed by Jorge Hirsch (Hirsch, 2007). An author has an H-Index of 3 if, from all the articles they have published, there are 3 that have at least 3 citations each; they have an H-Index of 5 if there are 5 articles with at least 5 citations each; 10 if there are 10 articles with at least 10 citations each, and so on (Figure 10.1). The H-Index is therefore a measurement of both quantity of publications and their quality.

H-indices have not been calculated and released in the same way that Impact Factors are because they may change every time a citation is indexed. This is, however, something of a weakness, as there is no definitive version of an author’s H-Index. The value will vary depending on whether it is calculated from WoS, Scopus or Google Scholar data. Even the automatic calculations on WoS and Scopus may give inaccurate results if based on incomplete unique author article sets. But there are problems with the metric beyond calculation (Bornmann and Daniel, 2007). It is, like the Impact Factor, strongly affected by differences in citation frequency between different subject areas. It is also insensitive to time, in that it can never fall; long-dead researchers may well have higher H-Indices than their living counterparts. Researchers who co-author in large groups tend to receive more citations and so have higher H-Indices, a pattern leading to increasing levels of co-authorship (Ioannidis, 2008). The metric can also be manipulated by publishing a large number of weaker articles that cite each other and would fail to properly rank those authors who publish a very few but critically important articles.

A number of variations on the basic H-Index have been proposed (Bornmann et al., 2008). Alternatives include the AR-Index (Jin et al., 2007), which takes into account the age of the citations, or the Individual H-Index (Batista et al., 2006), which aims to eliminate the effect of co-authorship. None of these deals satisfactorily with gulfs in citation levels between different subject areas that skew as a result of which index is used.

The future of research performance metrics

The next several years will probably see the emergence of some new metrics and the consolidation of some existing ones. At a journal level, the Impact Factor, misused as it sometimes is, will most likely remain the most accepted metric. It took decades for the Impact Factor to reach this position, so we should not expect any of the competitor metrics to replace it imminently, particularly given the complexity of their calculation. It is more likely that we will see the emergence of new approaches to measuring research performance at a journal level. As previously mentioned, the Usage Factor is still in development, and although it may be as prone to manipulation as the Impact Factor, if not more so, it will add a new dimension to the tools we have available. Time will tell whether it is accepted.

Work continues on developing new approaches to research performance evaluation. Grant information could theoretically be used to evaluate research, but the funding a project receives and its eventual usefulness are disconnected. The RAND corporation recently reported on novel efforts to evaluate research using systematically collected ‘atoms’ of data (Wooding, 2011) but these would seem to be too subject-specific to apply to the entire gamut of journal fields. Given the difficulties encountered in fairly comparing journals from different subject areas, it is even conceivable that field-specific metrics will evolve, but there has been little movement in this direction so far.

The continued development of metrics at other levels is a certainty. The major publishers are investing heavily in usage, citation and social media metrics at an article level, and the sophistication of these is only likely to increase. With the increased adoption of unique researcher identification, the use of author-level metrics should become easier and more reliable; all that remains is to find a strong metric to which author data can be applied (Finch, 2010).

Governments are also making the measurement of research performance a priority at an institutional and subject level. In the US, the National Institutes of Health, National Science Foundation and White House Office of Science and Technology policy has launched STAR METRICS, the first phase of which has looked at job creation resultant from research funding. The second phase plans to look at publications and citations but also social and environmental outcomes, patent grants and company start-ups based on the application of research innovations. In Australia, the ERA (Excellence in Research for Australia) initiative continues to develop its bibliometric indicators as one element of research performance evaluation.

The path and destination of the road ahead are quite uncertain; if only one thing is for sure, it is that publishers’ bibliometricians will need to know about far more than just the Impact Factor.

References