ASCB Logo

Impact Factor Insurrection Catches Fire; Over 6,000 Signatures and Counting


The San Francisco Declaration on Research Assessment

The “journal impact factor” rebellion is spreading. In the two weeks since it first went online, DORA—the San Francisco Declaration on Research Assessment that calls on scientists and scientific organizations around the world to minimize use of the journal impact factor (JIF) in evaluating research and researchers—has seen the number of individual signers jump from 155 to 6,083 while the number of scientific organizations signing on has gone from 78 to 231.

These numbers were accurate at midday, May 30, but signatures continue to pour in to the DORA site http://am.ascb.org/dora/. The 6,000th DORA signatory was Maritza Montero at Universitaet Regensburg in Germany. The 200th organization to sign was the Institute of Physiology at Guandong Medical College in China.

“These numbers—over 6,000 individuals and 200 or more groups—are arbitrary markers but extremely encouraging. I was expecting to reach 5,000 only in my wildest dreams,” said Stefano Bertuzzi, Executive Director of the ASCB, which hosts the DORA declaration. “They demonstrate that the DORA message is resonating powerfully across the worldwide scientific community. The ‘impact factor’ obsession is warping the way science is practiced and now thousands of scientists and scholars are joining the original signers of DORA to say that the misuse of this flawed, inappropriate metric has to stop.” Bertuzzi urged all who are connected with scientific and scholarly research to read and sign DORA.

DORA grew out of an ad hoc coalition of researchers, journal editors, and others concerned about the JIF and other citation algorithms who met last December at the ASCB Annual Meeting in San Francisco. The San Francisco group agreed that the JIF, which ranks scholarly journals by the average number of citations their articles attract in a set period, was warping the way that research in science and other scholarly disciplines was being conducted, reported, and funded.

A plea to the world scientific and scholarly world to sign DORA went public on May 17 with coalition signatures from 155 individuals and 78 organizations. Among the groups to sign DORA in recent days are research institutions or scholarly societies in Brazil, Venezuela, Qatar, Singapore, Switzerland, Belgium, Chile, China, Indonesia, and Nigeria. Scientific and scholarly fields now extend to agriculture, soil science, dramatics, athletics, psychology, veterinary medicine, philosophy, informatics, social work, taxonomy, and the history of science.

There are a number of citation ranking systems today, but the oldest and most influential is the so-called “two-year JIF” devised by Eugene Garfield’s Institute for Scientific Information (ISI) in the 1950s as a subscription-buying tool for academic and medical librarians. The JIF, which appears in Journal Citation Reports, is now compiled by the Thomson Reuters (ISI) Web of Knowledge.

The DORA insurgency, as it has become known, points to known defects in the JIF, distortions that skew results within journals, that gloss over differences between fields, and that lump primary research articles in with much more easily cited review articles. Further, the JIF can be “gamed” by editors and authors, while the data used to compute the JIF “are neither transparent nor openly available to the public,” according to DORA.

In a response to DORA, Thomson Reuters said, "No one metric can fully capture the complex contributions scholars make to their disciplines, and many forms of scholarly achievement should be considered. The journal impact factor is singled-out in the declaration not for how it is calculated, but for how it is used."

The statement concluded, "Thomson Reuters continues to encourage publishers, researchers, and funders to consider the correct use of the many metrics available, including the journal impact factor and data from the Web of Science, when performing research assessments."

—John Fleischman

Created on Thursday, May 30, 2013
Modified on Friday, May 31, 2013

Share
 

NIH Funding Forecast: Just Awful or Getting Much Worse?


Uncertain funding forecast: NIH leaders tell Senate appropriators that fiscal weather is bad but fear worse. Photo by John Fleischman

A standing-room-only crowd at a Senate Appropriations Subcommittee May 15 heard the U.S. National Institutes of Health (NIH) top brass bemoan the stagnation in the last decade of federal funding for biomedical research and plead to be spared further cuts in the fiscal year 2014 (FY14) budget.

NIH Director Francis Collins told the committee members that the NIH budget’s purchasing power had declined by 22% in the last 10 years. Flanked by NIH Institute Directors Anthony Fauci, Gary Gibbons, Richard Hodes, Story Landis, and Harold Varmus, Collins recounted for the senators a recent conversation with a former postdoc from the Collins lab who told him that she was deeply anxious about prospects for any career at the bench in biomedical research.

Subcommittee chair and longtime NIH champion Tom Harkin (D-IA) expressed his frustration at the declining NIH budget but declared that he would not “savage” other programs in order to provide the NIH with the funding it needs. “I will not get engaged in pitting NIH against other worthwhile endeavors in this appropriations bill.”

The current temporary sequestration cuts are only making things worse, Harkin said. “I can promise you, if sequestration stays in effect next year, there’s no chance that we will get close to the president’s request for NIH, let alone back to fiscal year 2012 levels. It just won’t happen.”

Senators Jerry Moran (R-KS) and Richard Shelby (R-AL), both strong advocates for the NIH, also voiced their strong desire to increase funding for the NIH. Shelby even advocated another doubling of the NIH budget, similar to the great research funding expansion in the 1990s. Frustration with NIH’s funding dilemma led Barbara Mikulski (D-MD), who chairs the entire Senate Appropriations Committee, to promise action on ending the sequester and writing a FY14  budget that provides NIH with the funds it needs. “I’m going to work my earrings off to make that happen,” Mikulski vowed.

As usual, subcommittee members ranged all over NIH territory with their questions, seeking information on everything from the recently announced BRAIN initiative to the role NIH research might play in responding to the spike in suicides by Iraq and Afghanistan War veterans.

The entire hearing is at: http://www.c-spanvideo.org/program/HealthFis.

— Kevin M. Wilson

Created on Wednesday, May 22, 2013

Share
 

Scientific Insurgents Say Journal Impact Factors Distort Science


The journal impact factor insurrection, which began last December in San Francisco, is spreading through scientific institutions and organizations around the world.

ASCB Urges All Researchers to Sign the DORA Pledge

An ad hoc coalition of unlikely insurgents—scientists, journal editors and publishers, scholarly societies, and research funders across many scientific disciplines—today posted an international declaration calling on the world scientific community to eliminate the role of the journal impact factor (JIF) in evaluating research for funding, hiring, promotion, or institutional effectiveness.

The San Francisco Declaration on Research Assessment, or DORA, was framed by a group of journal editors, publishers, and others convened by the American Society for Cell Biology (ASCB) last December in San Francisco, during the Society’s Annual Meeting. The San Francisco group agreed that the JIF, which ranks scholarly journals by the average number of citations their articles attract in a set period, has become an obsession in world science. Impact factors warp the way that research is conducted, reported, and funded. Over five months of discussion, the San Francisco declaration group moved from an “insurrection,” in the words of one publisher, against the use of the prominent two-year JIF to a wider reconsideration of scientific assessment. The DORA statement posted today makes 18 recommendations for change in the scientific culture at all levels—individual scientists, publishers, institutions, funding agencies, and the bibliometric services themselves—to reduce the dominant role of the JIF in evaluating research and researchers and instead to focus on the content of primary research papers, regardless of publication venue. The DORA coalition calls on all individuals and organizations engaged in scientific research to sign the San Francisco declaration.

The release of the declaration was timed to coincide with editorials in scientific journals around the world, including an endorsement of DORA by Bruce Alberts, Editor-in-Chief of Science Magazine in the journal’s May 17 issue. ASCB helped assemble a coalition of 78 organizations from all around the world including Howard Hughes Medical Institute, (HHMI), Wellcome Trust, EMBO, American Association for the Advancement of Science (AAAS), and Proceedings of the National Academy of Science (PNAS) along with 151 individuals as original signers of the DORA statement. Editors signing DORA represent Journal of Cell Biology (JCB), Traffic, Genetics, eLife, Journal of Cell Science, Aging Cell, Molecular Biology of the Cell (MBoC), BioArchitecture, The EMBO Journal, Journal of Cell Science, Journal of Surfactants & Detergents, Cell Structure & Functions (Japan), Lipids, Genes, Journal of the Electrochemical Society, and Development.
 
Mark Patterson, Executive Director of eLife, points to the diverse list of DORA signers as evidence that concern about impact factors is not an Us vs. Them issue in scientific publishing. “You’ve got the very old like Science and the very new like PeerJ or eLife. So you’ve got the old and the new, the non-profit and the for-profit, the open access and the subscription model. It’s really a mix,” says Patterson. “The issue cuts right across and not just publishers of all shapes but all the different constituencies involved in research assessment, all the way from individual researchers to the institutions that they work in or are funded by to the journals that they publish in.”

There are a number of citation ranking systems today, but the oldest and most influential is the so-called “two-year JIF” devised by Eugene Garfield in the early 1950s and originally published by his Institute for Scientific Information (ISI) as a subscription buying tool for academic and medical librarians. The JIF, which appears once a year in Journal Citation Reports as part of the Thomson Reuters (ISI) Web of Knowledge, is the average number of citations received in a year per paper published in the journal during the two preceding years. The earliest that a new journal can have a JIF is the end of its third full year of publication.

Even though the JIF is only a measure of a journal’s average citation frequency, it has become a powerful proxy for scientific value and is being widely misused to assess individual scientists and research institutions, say the DORA framers. The JIF has become even more powerful in China, India, and other nations emerging as global research powers. “It’s maddening,” says David Drubin, Editor-in-Chief of ASCB’s journal MBoC. “This is a metric that really drives a lot of traffic. You really see it most clearly when you travel to foreign countries and I especially see it with my foreign postdocs. They only want to publish in journals with high impact factors.”

The San Francisco declaration cites studies that outline known defects in the JIF, distortions that skew results within journals, that gloss over differences between fields, and that lump primary research articles in with much more easily cited review articles. Further, the JIF can be “gamed” by editors and authors, while the data used to compute the JIF “are neither transparent nor openly available to the public,” according to DORA.

Since the JIF is based on the mean of the citations to papers in a given journal, rather than the median, a handful of highly cited papers can drive the overall JIF, says Bernd Pulverer, Chief Editor of the EMBO Journal. “My favorite example is the first paper on the sequencing of the human genome. This paper, which has been cited just under 10,000 times to date, single handedly increased Nature’s JIF for a couple of years.”

“The Journal Impact Factor (JIF) was developed to help librarians make subscription decisions, but it’s become a proxy for the quality of research,” says Stefano Bertuzzi, ASCB Executive Director, one of more than 70 institutional leaders to sign the declaration on behalf of their organizations. “Researchers are now judged by where they publish not by what they publish. This is no longer a question of selling subscriptions. The ‘high-impact’ obsession is warping our scientific judgment, damaging careers, and wasting time and valuable work.”

The SF declaration urges all stakeholders to focus on the content of papers, rather than the JIF of the journal in which it was published, says Bertuzzi, “The connection is flawed and the importance of the finding as reflected by the light of a high JIF number is often completely misleading, because it is always only a very small number of papers published in a journal that receive most of citations, so it is flawed to measure the impact of a single article by this metric. Great papers appear in journals with low JIFs and vice versa.”

One of the four editors of Traffic who signed DORA, Michael Marks acknowledges that the group realized that the scientific world has been using impact factors inappropriately. “Initially our gut reaction was to blame the JIF itself but it’s not the JIF’s fault,” says Marks. “It’s our use of the JIF that’s the problem.”

DORA’s 18 recommendations call for sweeping changes in scientific assessment, says Drubin. They will hopefully lead to “a change in the culture where people will choose the journals that they publish in not on the prestige but on the fit. Is the format correct? Is the audience correct? Does the editorial board have the appropriate expertise?” A difficult change, Drubin concedes, but vital to the integrity of scientific self-assessment, which is the engine by which modern science advances. “For me, it was just a matter of when enough is enough,” says Drubin of his role in the great JIF insurrection.

Read the complete “San Francisco Declaration on Research Assessment” at www.ascb.org/SFdeclaration.html where you can also find a list of the original signers and a link to a form for adding your signature to DORA. You can also download the DORA logo and add it to your home page or your slides for your next talk.  

Created on Thursday, May 16, 2013

Share
 

The Stem Cell Renewal Theory: The Other Big Paper of 1953


Charles Philippe Leblond -- His stem cell renewal theory was shocking in 1953. Photo courtesy John Bergeron, McGill University

The year 1953 is generally considered the year zero for molecular cell biology with the publication of Watson and Crick’s celebrated Nature paper on the structure of DNA. But there was another big paper in 1953 by Yves Clermont and Charles Leblond of McGill University that appeared in the American Journal of Anatomy. It contained a discovery nearly on the order with that of DNA. Working in the rat testes, Clermont and Leblond described a population of undifferentiated cells that divided to produce spermatozoa but also to maintain their own undifferentiated ranks. The researchers called them “stem cells” and their hypothesis the “stem cell renewal theory.” Unlike DNA, Leblond’s theory of “plentipotent” stem cells met a storm of criticism but in the fullness of time, “stem cells” turned out to be real and the hypothesis correct. Few researchers cite 60-year-old papers but if stem cells scientists were still crediting Clermont Y & Leblond CP, Renewal of spermatogonia in the rat, its 60-year impact factor would be off the charts.

In Montreal, McGill’s Department of Anatomy & Cell Biology has put up a page to mark the 60th anniversary of the stem cell renewal theory paper. Leblond had a very long and extraordinary professional life (he died at 97 in 2007). We can only refer the curious to Wikipedia for a quick summary. Leblond was also an early and influential member of the ASCB. He joined in 1963, served on Council from 1969 to 1971, and received the E.B. Wilson Medal, the ASCB’s highest scientific award, in 1982. 

—John Fleischman

Created on Tuesday, May 14, 2013

Share
 

Post Yourself on The ASCB Post

Send us links to and comments on hot papers, cool science, great talks, interesting snippets, honest opinions, new uploads, great downloads, and your life in science.

Email: PostNews@ascb.org
View our Website Posting Rules