The State of Peer Review. Solution? - U.S. Politics Online: A Political Discussion Forum

Kijana · #1 (**permalink**) 05-20-2008

I got this out of the latest Fisheries journal in the editorial section. I think it's a interesting read, and highlights a problem that the scientific community is having getting good reviewers with the increasing amount of journals popping up. The solution he proposes is very interesting. See what you think.

Quote:

Editorial: Fisheries Forum

Transparency and the Peer-Review Process

Kevin Amos

Amos is an AFS/Fish Health Section Certified Fish Pathologist and has been an AFS member since 1973. He lives in Surprise, Arizona, and can be contacted at doradoamos2@comcast.net.

Pages 197-198 Fisheries • vol 33 no 4 • april 2008 • American Fisheries Society

Kafee: I want the truth.

Col. Jessep: You can’t handle the truth.

These memorable lines spoken by Jack Nicholson and Tom Cruise in the movie A Few Good Men sruck a chord recently as I was reading an article in Science magazine on parasites and their impact on wild salmon. Some of the information in the article did not jibe with my understanding of epiemiology, parasitology, and the population dynamics of wild Pacific salmon. I began to wonder if I was out of touch with cutting-edge fisheries technology. I also asked myself who might have conducted the peer review on this article. Do they know something I don’t or perhaps they are only protecting me because for some reason I’m unable to handle the truth? Perhaps all these things are true. But rather than sit back and whimper about injustices and misperceptions in the world of fisheries science, I thought I might try to shed some light on the topic from firsthand experience and propose an approach to peer review that might benefit us all.

Newspapers, radio, and the evening news on TV—it is difficult to obtain unbiased information. It is evident in the current presidential campaign that erstwhile TV anchors have already “cast their vote” and find it difficult to restrain their enthusiasm over a particular candidate. But this type of “enthusiasm” and biased reporting is not supposed to take place on the playing field of science. Isn’t science

the last remaining fortress free from political influence? Is it not in science that we find truth? (perhaps it is in vino veritas!) Well think again! From firsthand experience I can report that the peer review process not only is opaque but it often does not weed out articles that should never have been published. A few years ago I was asked to review an article submitted to our own North American Journal of Fisheries Management. At the end of the day I rejected the article and suggested the methodology and results did not support the conclusions. Further, I suggested that this type of research would be more appropriately submitted to the Journal of Aquatic Animal Health. After reading the comments by the other two reviewers I came to believe that they did not have expertise in fish pathology and consequently could not reach the same conclusions. Never mind! I was told by AFS editorial staff that the other two reviewers accepted the article and I was outvoted by a score of 2 to 1. Whenever I see that article cited as a scholarly reference I cringe. Maybe you’ve had a similar experience? Perhaps it is something that scientists don’t talk about, lest they might some day be on the receiving end of criticism?

What can we do about the peer review process? I don’t presume to think that the prestigious weekly magazine Science would consider my proposal but perhaps the journals of my organization, the American Fisheries Society, might consider another approach—an approach that brings transparency and rigor to the peer review process. Once upon a time scholarly journals would provide space at the end of an article for experts or the peer reviewers to weigh in and express their thoughts on the preceding journal article. I enjoyed reading these discourses because they provided unique insight into the issue, often challenging or supporting the methods and the results of the study. Also, some journals today such as Diseases of Aquatic Oganisms (DAO) provide a list of not only editors but also a list of topical experts who review articles. Anyone who submits an article to DAO knows which experts will peer review their submission. Isn’t it time AFS re-examined how we peer review the articles destined for our journals?

I suggest for AFS journals that we provide a simple two step process to insure the integrity and transparency of what we publish:

1. All individuals who agree to peer review an article also agree to have their names listed at the end of the article when it is published. As an alternative, all the AFS journals will publish a list of topical experts who will in fact be the ones that review journal submissions, such as is done by DAO.

2. All peer reviewers will be provided a brief opportunity to make a comment on the article. At the point of publication, edits suggested by the peer reviewers will presumably already have been incorporated into the article. This “new” opportunity I’m describing will allow comments on relevance, importance, and innovation provided in the article or perhaps shed some new insight on similar research previously published (or unpublished).

I know our publishers (AFS) might bemoan the fact that I’m suggesting a few pages more in each journal with no revenues for page charges. I also suspect that some people will be reluctant to be reviewers as they might fear the wrath of the prospective authors. I say phooey to each of these criticisms. Most journals today are published electronically. I suspect in a few years the hard copy will be the rare exception—so page charges should not be an issue. Besides, electronic journals are more eco-friendly. As to the fearful reviewers, if you don’t have the courage to stand behind science and publicly express your views and be critical of the work of others, I suspect that perhaps you truly are not the expert you represent yourself to be and should not be providing a peer review anyway.

I challenge the AFS leadership and our members to consider my proposal and take steps to put truth and transparency back in the science we publish. Let’s do our best to put science back where it belongs—in the light of day, and make scientists, peer reviewers, and editorial boards accountable for what is published! look forward to seeing your feedback in a future issue of Fisheries.

Otter · #2 (**permalink**) 05-20-2008

Quote:

Originally Posted by Kijana

I got this out of the latest Fisheries journal in the editorial section. I think it's a interesting read, and highlights a problem that the scientific community is having getting good reviewers with the increasing amount of journals popping up. The solution he proposes is very interesting. See what you think.

It is interesting. I do wonder whether making rewies non-anonymous might make especially younger scientists who are asked to review an article tend to be a little more lenient than they would otherwise be, for fear of reprisal form someone more entrenched in the field. Reviewer comments at the end of an article could be printed anonymously, I suppose. They could even be included in the electronic version only (with a notice in the print version that these are available online), as is currently done with supplimentary materials.

...yay for fish pathology!

Si modo · #3 (**permalink**) 05-20-2008

Fear of reprisal is a real fear. I think the option of remaining anonymous with one's comments, whether new or entrenched, is the way to go until the reprisal aspect disappears.

I also like the idea of seeing who has reviewed what, for those who want to be acknowledged. Often folks request reviewers who will give a favorable rating or request that another be excluded because of competition (not that they are guaranteed to get their requests). Let the reader decide who is being partial in their comments based on this.

Otter · #4 (**permalink**) 05-20-2008

You know, I wonder whether other journals will follow the format of Nature news, and have a section for comments at the end of the article. THose can be left failry anonymously, and by anyone, and occasionally interesting discussions follow. Right now, it's just the news summary articles, not actual research articles that have this feature but there's no reason why it couldn't be done with research papers. Just like a worldwide journal club, you know?

CDavidNeely · #5 (**permalink**) 3 Weeks Ago

Quote:

Originally Posted by Kijana

I got this out of the latest Fisheries journal in the editorial section. I think it's a interesting read, and highlights a problem that the scientific community is having getting good reviewers with the increasing amount of journals popping up. The solution he proposes is very interesting. See what you think.

Quote:

I know our publishers (AFS) might bemoan the fact that I’m suggesting a few pages more in each journal with no revenues for page charges.

I think you would have a hard time convincing reviewers to put their names on their reviews because of two reasons.

I don't see any changes being made in the system until we remove the contamination of money considerations in science. There are so few research establishments outside of the corporate world that science is becoming increasingly unreliable as a method for understanding the universe. Scientist have become fearful of extending their names because it might "endanger" their funding.
The emotional content placed into being right and the desire to be famous are another reason why you would have trouble getting people to sign their reviews. If you read scientific history you can see how science has always been plagued by this.

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode