Auto-correlation of journal impact factor for consensus research reporting statements: a cohort study

Introduction

The journal impact factor (JIF) was introduced as a method to compare journals regardless of the number of articles they publish, to help select journals for the Science Citation Index (SCI), recognizing that small but important journals would not be chosen based solely on absolute publication or citation counts (Brodman, 1960).

Today, JIFs are widely used to rank and evaluate journals, as a proxy for the relative importance of a journal within its field; journals with higher JIFs are in general deemed to be more important than those with lower ones (Brembs, Button & Munafò, 2013; Eyre-Walker & Stoletzki, 2013). This application of the JIF is also widely extended to the evaluation of individual scientists, with the JIF of the journals they have published in used as a precondition for grants or faculty promotion and tenure (Archambault & Larivière, 2009; Fuyono & Cyranoski, 2006).

There have been many criticisms levelled at the use of JIF to evaluate a journal’s importance. It has been demonstrated that JIFs, and citation analysis in general, are affected by field-dependent factors (Archambault & Larivière, 2009; Bornmann & Daniel, 2008), which may invalidate comparisons not only across disciplines but even within different fields of research of one discipline (Anauati, Galiani & Gálvez, 2014). Furthermore, the JIF is based on the arithmetic mean number of citations per paper, yet it is obvious that citation distribution is not normal, but follows a power-law distribution—20% of papers in a journal can account for 80% of the total citations—making it a statistically inappropriate measure (Archambault & Larivière, 2009; Pulverer, 2015; Adler, Ewing & Taylor, 2009). This means that a small number of highly-cited articles contribute disproportionately to a journal’s JIF, with most articles within a journal receiving fewer citations than would be predicted from the JIF (Anonymous, 2005).

These problems are exacerbated when the use of JIFs is extended to evaluate not only the journals, but papers therein (Eyre-Walker & Stoletzki, 2013; Pulverer, 2015). A number of studies have shown that articles published in journals with high JIFs are cited more frequently than those published in lower-JIF journals (Callaham, Wears & Weber, 2002; Seglen, 1994; Nieminen et al., 2006; Filion & Pless, 2008; Etter & Stapleton, 2009), with the implication that research articles published in high-JIF journals have a greater impact on the scientific literature and are perceived to be of better quality than those published in less prominent journals.

However, the number of citations a paper receives has been found to be an extremely error-prone measure of scientific merit, with poor correlation observed between article quality as rated by external assessors and the number of citations the article receives (Eyre-Walker & Stoletzki, 2013). Furthermore, a number of external factors have been shown to influence the citation rate of articles (Bornmann & Daniel, 2008), including the publication licence (open access vs. subscription) (Piwowar & Vision, 2013), number of authors, contributing institutions (Figg et al., 2006; Stringer, Sales-Pardo & Nunes Amaral, 2010) and the number of article accesses (Watson, 2009; Schloegl & Gorraiz, 2010; Paiva, Lima & Paiva, 2012).

Due to these influences, evaluating whether the JIF is correlated with citations counts requires identical articles published simultaneously across multiple journals. Larivière & Gringras (2010) compared the number of citations to 4,532 pairs of duplicate publications (based on identical titles, first author and length of reference list), showing that the citation impact of duplicates published in the journals with a higher JIF was almost double that of those published in lower-JIF journals. This provided evidence for preferential attachment—also known as the Matthew effect (Merton, 1968)—demonstrating a cumulative advantage for high-JIF journals, whereby ‘the rich get richer and the poor get poorer.’ However, this comparison was limited to pairs of publications, which prevented a more nuanced evaluation of the link between the journal JIF and number of citations the articles received.

Unlike research papers, reporting guidelines are often published simultaneously to encourage wider adoption and dissemination, as they are not scientific publications, but rather recommendations for authors. Perneger (2010) investigated the presence of journal-bias using the reporting guidelines QUOROM, CONSORT, STARD and STROBE, finding a strong correlation between the JIF and the number of citations to each publication.

The advantage of investigating the Matthew effect for journals using reporting guidelines is that they are published in multiple journals and control for many of the factors that influence citation counts. However, Perneger’s study did not take into account the number of article accesses, which has also been shown to be correlated with the number of citations (Watson, 2009; Schloegl & Gorraiz, 2010; Paiva, Lima & Paiva, 2012). Furthermore, since the publication of Perneger’s study, the number of reporting guidelines has increased dramatically providing a larger base for comparison.

The purpose of this study was to investigate the correlation between JIF and the number of citations to consensus research reporting statements published simultaneously in multiple journals, and the correlations between the total article accesses according to the COUNTER code of practice (Pesch, 2015) and number of citations. The hypothesis was that there will be an observable Matthew effect for consensus reporting statements published simultaneously across multiple journals, with articles in high-JIF journals being more highly-cited than those in lower-JIF journals, independent of the number of article accesses that statement received.

Methods

Eligible articles were consensus statements on research reporting listed on the EQUATOR Network website (http://www.equator-network.org/) that were published simultaneously in identical formats in three or more journals, each with different citations details. Republished articles—those which cited another version of the guideline, stated that they were republished within the article or not published simultaneously (or in the next available issue)—were excluded. Research reporting statements that directly cited another version of the reporting guideline as definitive were excluded, as were those published after 31 December 2013, because the citation data were not yet fully available at the time of data extraction.

The JIFs were obtained from the Journal Citation Reports (JCR), with a one-year offset (e.g., using the 2013 JIF value for articles in 2014), as this would have been the JIF attributed to the journal during the citation accumulation period. Journals that were not indexed in the JCR were included in this analysis for completeness as they would also accumulate citations and article accesses, which would impact on those received by the other journals, but were assigned an JIF of zero, to represent their position at the bottom of the relevant JCR category.

Total citations counts were obtained for all these articles using the abstract and citation database Scopus (Elsevier B.V., http://www.scopus.com), on the same day (22/07/2015); this allowed coverage of journals without an JIF, but restricted the citation counts to those accumulated within the scientific literature.

Total article access counts according to the COUNTER code of practice were obtained between 20 July 2015 and 24 July 2015 from the publishers or the journal website, where available. In many cases, these data were not available either due to errors, system migrations or the publishers’ not collecting the data according to the COUNTER standards.

Results

Overall, 280 reporting guidelines were evaluated for inclusion in this analysis; of these, 13 were associated with three or more publications. Four guidelines were excluded: one as it was published too recently for there to be robust citation data (TRIPOD), and three as the articles were republications, citing a single definitive version (QUOROM, ARRIVE and SQUIRE). Therefore, nine reporting guidelines were included in this analysis, representing 85 articles published across 58 journals in biomedicine. The journals included in this analysis are included in the Appendix.

Table 1 shows basic information about the nine reporting guidelines included. The median number of published articles per reporting guideline was 10 (range 5–15), published between 2003 and 2013. Articles that were not published simultaneously (defined as same publication date for continuous publication journals, or next available issue) were excluded from the analysis. The range of the journal 2013 JIFs was 0–39.4.

The number of citations to individual articles ranged between 0 and 2,941. The median number of citations varied depending on the guideline in question, and ranged between 5 and 962. The number of individual article accesses ranged between 137 and 49,961. The median number of articles accesses according to reporting guideline varied between 951 and 10,696.

Spearman’s correlation analysis showed a median correlation of 0.66 between the JIFs of the source journal and the number of citations (95% CI [0.45–0.90]; Table 2), with six of nine analysed reporting guidelines showing statistical significance (p < 0.05). The four guidelines that did not reach statistical significance demonstrated a weaker correlation between JIF and the number of citations (0.45 for STREGA, 0.52 for GRIP, 0.61 for CARE and 0.33 for CHEERS). Analysis of logarithm-transformed values using Pearson’s correlation coefficient revealed a median correlation of 0.83 between JIF and the number of citations (95%CI [0.43–0.81]; Table 2), with all but two reaching statistical significance (0.43 for STREGA and 0.81 for REFLECT).

These results were supported by linear regression analysis (Table 2), with doubling the JIF associated with a 3-fold increase in citations (1:1.2 logarithmic units, range 0.5–3.0; Table 2 and Fig. 1).

A median correlation of 0.71 was observed between article accesses and citations, using Spearman’s correlation coefficient (95% CI [0.5–0.8]; Table 2) and 0.6 using Pearson’s coefficient for the logarithm-transformed values (95% CI [0.19–0.78]; Table 2), for all reporting guidelines except GRIPS, which showed weak-to-no correlation. Univariate linear regression showed that increasing the article accesses by a factor of 10 was associated with a 3.2-fold increase in citations counts (1:0.5 logarithmic units, range 0.3–0.6), with trend of article accesses having greater impact on citation counts for older reporting statements (Table 2).

I also conducted multivariate analysis at the reporting guideline level, considering both JIF and total article accesses as predictors for citations (Table 2). When adjusted for the individual reporting guidelines, increasing the JIF by a factor of 10 predicted a median 6.3-fold increase in citation counts (1:0.8 logarithmic units, 95% CI [−0.4–5.2]), while article accesses needed to increase by a factor of 50 to achieve the same increase in citation counts (1:0.1 logarithmic units, 95% CI [−0.9–1.4]). This model explained a median 26% of the variance in citations (median adjusted r² = 0.26, range 0.18–1.0), but only reach statistical significance for STROBE (p = 0.0001).

Discussion

This analysis extends the work by Perneger (2010), to show that both the JIF of the journal in which an article is published, and the number of accesses that article receives are strongly correlated with the number of citations to that article. Identical articles published simultaneously in multiple journals were used for this analysis, which controlled for variations in the characteristics of the articles themselves (for example, scientific content, number of authors and institutions); therefore, any variation in the citations count for the article reflect either the journal they are published in or the number of article accesses.

The results demonstrated a strong correlation between the number of citations and the JIF and between the number of citations and the number of accesses for most of the reporting guidelines included, suggesting the presence of a Matthew effect. The regression slope for citations and article accesses demonstrated a weaker influence of article accesses on the total number of citations. This was also supported by the multivariate regression analysis, in which the coefficient for article accesses was lower than for JIF for every consensus reporting statement analysed, except PRISMA and STREGA. This suggests that JIF has a proportionally greater effect on the number of citations to an article.

The calculated correlation coefficients for STARD and STROBE were higher than those calculated by Perneger (0.66 vs 0.65 for STARD, and 0.93 vs 0.81 for STROBE, respectively) (2010). Similarly, the influence of article accesses on the number of citations generally increased with the time since publication. This could be explained by early citation counts influencing later and total article citations, as was previously demonstrated by Adams (2005). However, article accesses and citations have a different obsolescence pattern (Schloegl & Gorraiz, 2010), with accesses rapidly accumulating on publication, while citations take longer to build. Due to these different obsolescence patterns, it is essential, to consider a time-window of several years, which could explain the increasing regression coefficients with the age of the publication.

A number of factors have been previously shown to influence the citation rate of articles (Bornmann & Daniel, 2008), including the publication licence (open access vs. subscription) (Piwowar & Vision, 2013), number of authors, contributing institutions (Figg et al., 2006; Stringer, Sales-Pardo & Nunes Amaral, 2010) and the number of article accesses (Watson, 2009; Schloegl & Gorraiz, 2010; Paiva, Lima & Paiva, 2012); however, by using consensus reporting statements, this study was able to control for many of these factors. Lozano, Larivière & Gingras (2012) have further suggested a weakening relationship between the JIF and article citations since 1990.

The results of the current study suggest that, rather than the JIF serving as a proxy for scientific value of the journal or article, it is instead auto-correlated, with articles being cited simply because they are published in a high-JIF journal. This may help to explain the stability of journal rankings within the JCR.

Conclusion

In this study, the impact factor of the journal in which a consensus reporting statement was published was shown to correlate with the number of citations that statement will gather over time. Similarly, the number of article accesses also influenced the number of citations, although to a lesser extent than the impact factor. These findings suggest that citation counts are not purely a reflection of scientific merit, and the journal impact factor is, in fact, auto-correlated.

Supplemental Information