The world's scientific and social network for malaria professionals
9297 malaria professionals are enjoying the free benefits of MalariaWorld today

What should we do: Peer review or not?

January 16, 2014 - 13:36 -- Bart G.J. Knols

The science world is undergoing rapid changes, and so does the field of scientific publishing. The Lancet recently featured five articles on the current value and reduction of waste in biomedical research. An article in the Economist from a few months before titled 'How science goes wrong' is another eye-opener. Clearly, much is changing in the science world, and this includes us scientists working on malaria.

Here we are asking for your views regarding an issue we are discussing for the MalariaWorld Journal, entering its 5th volume this year: Should we continue with peer review, yes or no, or should we perhaps make it optional?

Peer review

The role of peer- review is obvious. Experts not directly engaged with the research you are intending to publish check the merits of the manuscript, the way data were generated, analysed, and interpreted. At least, that's how it is supposed to be. And how it used to work just fine. But today, with the number of publications skyrocketing, experts become over-stretched and can no longer spend many hours a week reviewing other people's work. The result is that finding reviewers has become a nightmare for editors, or, alternatively, you receive 'quick and dirty' reviews that don't really improve the value of the manuscript.

At the MalariaWorld Journal we witness this process first hand. It sometimes takes weeks to find suitable reviewers. They then need three reminders before they do it, sometimes taking months to receive feedback, which sometimes is of disappointing quality.

So what to do?

There are already quite a few journals out there that do no longer offer peer review but rather let the scientific community gauge the value of the work. Mind you, there is still editorial support to make sure that manuscripts are complete, use good language, have all the references, etc. But they are no longer sent to reviewers. Instead, once posted online, the scientific community can list comments with articles to express their views and opinions. We wonder, would this be a good model for the MalariaWorld Journal?

What is missing in this reasoning is that sometimes peer review results in a dramatic improvement of the quality of the end product, which could be lost if peer review no longer takes place...

Taking this into account, we could offer both options:

1) You submit your manuscript to us and request for peer review. This is the traditional route that may give you important feedback to improve the manuscript before publication. It may of course be rejected - that's your gamble. And for sure, you will have to be patient, as the review process may take months.

2) You submit your manuscript to us and indicate that you wish to proceed with publication without peer review. We will indicate this on the paper once it is published. If you are convinced that your manuscript is of high quality, and you are happy to proceed with internal editiorial review by us only, the we proceed to the copy-editing stage fast and your work will appear online much quicker (weeks). 

Before we make a decision, we would very much appreciate your views...thank you for posting comments below this blog.

 

Comments

Submitted by Anonymized User (not verified) on

Despite drawbacks, there is no substitute for peer review. We can work to improve the peer review process though. There is a little to gain in being swamped with unreviwed articles that are fundamentally flawed.

Submitted by Nayna Patel on

This sounds like an excellent idea! It could lead to genuine progress in the publishing of scientific work. However, there may be some changes in mass thinking required – this may take some time to evolve! The main hurdles/perceptions:
• The commonly held belief that a paper which has not been ‘traditionally’ peer reviewed is somehow inferior.
• The fear that such a paper may not count as a legitimate ‘citation’ or a ‘publication’ in the truest sense of the word for the author/s of the paper or those that need to cite the work. Can a publication that has not been traditionally peer reviewed still be counted as a legitimate publication for the author?
A query: Will you be allowing questions at the end of the paper/publication?
Whilst it may be true that ‘traditional’ peer review may improve a scientific paper, the World Wide Web is one of the best critiques for any finding, product or theory, given half the chance. If you need to know the truth about anything- test it out on the World Wide Web. If the scientific community and indeed, the public at large, are allowed to comment on a paper and ask questions and if the author is given the right to reply, this can be far more rigorous than 2 traditional peer reviewers – no matter how good they are. The author can then have the option of changing the text (maybe within a certain time frame?) or leaving the comments, queries and answers so that others can gain a better understanding of their work. This is not possible with traditional publishing and with time, authors may actually prefer this route, once it gains in acceptance. Even if an author fails to provide answers to queries, the questions will still be food for thought for others. An author should be able to be in control of which queries he is happy to answer or allow at the end of their paper. Another outcome (if questions at the end of papers are allowed) could be that serious science can become accessible to scientists for whom the topic is not their field and indeed to the general public, simply by the fact of being able to ask questions about the findings of the research. The function of science is to solve problems and to advance knowledge and if it means that being allowed to ask questions makes someone’s work better understandable to everyone else, then this is true progress and a good way to go!
There are many commendable studies that don’t reach the light of day because someone somewhere decided that it did not meet ‘their criteria’ for publication and yet the work carried out may truly be of benefit if it was known. What is suggested should bring far more studies and findings out of the lab and into the forefront of knowledge – where it belongs. Many good pieces of investigation struggle to be known because of the many hurdles that need to be crossed by the traditional publishing route. What is suggested should take away some of the barriers which have left many scientific studies and findings in the dark and still in the lab.
Another query: Some journals allow their papers to go on to ‘pre-print’ servers prior to publication, as this does not infringe on their copyright – the copyright is retained by the author whilst the paper is on the web. This seems like a waste of time - to try and get published traditionally when the paper is already on the web. Who will retain copyright when a paper is published on Malaria World?

Submitted by Pierre Lutgen on

I applaud to what Nayna Patel is saying
I would be glad if Malariaworld offered the possibility to proceed with publication without peer review. Evidently opening the possibilty to ask questions or to make comments.
I know too many African research teams who see their paper rejected because it offers other options than dogmatic or lucrative Western medicine

Submitted by Ricardo Ataide on

Times are definitely changing, but publishing without peer review seems a bit extreme. I mean, definitely I believe that the system has to change and I would certainly include in the process things like post-publishing reviewing and a 'reviewer database', sort of speak, where we could identify a reviewer's code and thus know who was reviewing what, but without actually revealing the reviewer's true identity. But publishing without reviewing, at this point in time, seems contra-productive. Can you really imagine what it would be like if a lot of scientists that are under a lot of pressure were allowed to publish 'at will' ? We cannot assume that the editors of the journals would be able to review everything before publishing or even have the necessary expertise to do. And to trust that the internet community will provide that criticism and reviewing reveals a lack of awareness at the problems that even a platform as specific and well thought out like Malariaworld has of engaging the attention of only a limited number of scientists on commenting and participating in matters that relate to their work, passion and sometimes entire devotion: malaria.
So, the time may come when peer reviewing before acceptance of a manuscript will disappear, but not right now.

Ricardo Ataíde

Submitted by Irene Teis on

Peer review often makes me think about what was common in my small catholic country in my catholic youth. Some experts at the bishop’s office decided what might be published or read.

I found a document written by Pr.Dr. Cornelia Oertle Bürkl of the Berner Fachhochschule, unfornately in German. I translated a few excerpts

»Falsifications, Freedom and Objectivity in Research »

The 21st century is probably going to enter history as the century of falsifications Even the elitist ETH of Zürich had to digest in 2009 a research, peer reviewed paper, which was based on fake chemical data. The reasons for these deviations are multiple: not only the « publish or perish » pressure, but mainly the fact that it Germany to-day 70% of research is sponsored by industry. This is to walk on a sharp ridge: how to maintain freedom and integrity when the sponsor expects results which are in his interest. Matthew and Luke said 2000 years ago: „No one can serve two masters“. And Max Weber 100 years ago: » Facts are more important than interests ».

There is also the blog entry of Bart Knols on MalariaWorld of 4 June 2013

List of Predatory Open Access Publishers

“As long as there is much money to be made in academic publishing we will continue to see an increase of these misleading claims that look good but are all geared towards one thing: money”. Who is in charge of peer review in these journals where you have to pay large amounts for the publication of your paper?

And there is also the famous book of Federico Di Trocchio

“Le Bugie della Scienza” (The big Swindle in Science)

Again the translation of a few excerpts: “ The history of Galileo is well known, but it is less known that the Nazis rejected Einstein’s relativity theory and replaced it by a “ Deutsche Physik”, that Stalin promoted the biology of Lyssenko to a creed, It has become vital for the manager of an academic research laboratory to-day to do a lot of PR-work to find sponsors, be it from industry or his own government. He knows that a few key words in a research project can open the doors: climate change, sustainability, future generations, toxic health effects, even for innocuous molecules. The origin of this corrupted research system started in the US. It may have led to some major breakthroughs in science, but it also led to obedience. Creativity is stifled.

Peer reviewers are recruited from this school: they too often are mercenaries.” They leave little chance to young research teams from the South to see their papers published in glorious Journals from the North.

Why do many people prefer their paper to be peer reviewed?

The new work is submitted to what amounts to a committee of people who are (at least presumed to be) eminent in the field. The trouble is that the reviewers, being eminent in the field, are by definition wedded to whatever the conventional wisdom may be on that subject. Anything new or startling, especially if it contradicts the current “consensus”, is virtually certain to be rejected. Peer review only partially fulfills the requirement for questions and critics from the rest of the scientific community. Scientists abdicate their responsibility to examine and attempt to disprove discoveries in favor of turning over to gatekeepers who ensure that nobody rocks the boat. It’s particularly pernicious when funding is involved. A bureaucrat charged with deciding who gets the money for experiments is unlikely to be deeply knowledgeable about the field, and must therefore play it safe by going with what the reviewers say is reasonable. It means, ultimately, that the pace of scientific discovery is slowed to a crawl.

So it is, today, with “climate science”. There does exist a consensus — that the Earth is warming, that the result of the warming will be disastrous, and that human activities are responsible for it — and contrary views or assertions are savagely attacked as “denialist“. While “researchers” who echo the Conventional Wisdom are allowed to conceal or even destroy data that might contradict their thesis. In peer reviewed papers!

Let’s hope that what happened to Climate Science will not happen to Malaria Science.

Submitted by Ricardo Ataide on

Hi Irene,

It might be a bit of an overstretch to compare the peer-reviewing of facts with the blind defense of ideologies in Nazi Germany...

As I said, i do agree that changes need to be made to the system as it is. We already have to spend hours reading papers and new ideas knowing that they have been through some kind of sieve (good or bad, that is another question). Can you imagine the nightmare it would be to try and read every single paper that someone decides to write and publish so that we can make up our minds to decide if it is good or not? And if you do do that, the obvious result would be the disappearance of journals altogether and the creation of huge online repositories of information. Not to say that I don't see the merits of such a future, but maybe that is a system that could run in parallel with peer-reviewing?

Again, peer-reviewing as we know it today must certainly end, but it is not Nazi Germany nor is it 16th century Catholic Church. Pubpeer, for instance, is a good example of how an online platform can help with the post-publishing review of papers. Maybe another good change to the system could be the publication, online only and together with the paper, of the reviewers comments, which could be important if the reviewers (like I suggested before) could be identified by a code. For instance you could see that reviewer 12H2Y always has concerns for a certain thing or that reviewer 3HrTY6 seems to always reject a certain kind of idea. It would even be useful to publish the comments from reviewers and journals that have previously rejected the paper. That means, though, that the entire peer-reviewing process would have to be a centralized and shared process between ALL journals. Not impossible to accomplish I think.

So, just to summarize my ideas (maybe more to myself than anything else):
1- we could create a central peer-review agency where every reviewer could register
2- The reviewers could get a usercode and id
3- That usercode/id would be visible on the published version of the paper
4- we could be able to search a certain usercode/id and see how many papers they had reviewed, on what subjects and all the associated reviewing process
5- Papers could be published with the entire reviewing process attached as SI, including the reviews from previously rejected versions of the papers.
6- And besides this, we can always have post-publishing comments...

we would be left with the quetsion: who would administer such an endeavour? Anyway, What do you think?

Ricardo Ataíde

Andrew Lover's picture
Submitted by Andrew Lover on

ArXiv has been doing this quite awhile, but it's more of a pre-print idea. I suspect many authors will be wary of submitting research for web-pub without peer-review, esp in this era of cut-throat funding. That said, the pace of peer-review is unsustainable: PLOS One published 31,500 papers in 2013. At some point it is going to be be simply impossible to find reviewers, but then who will have time to vet/comment/critique non-peer reviewed papers?

I think a mixed-model journal is likely to confuse everyone even if it is clearly stated on each article (although PNAS has dual tracks as well).

Jeff Juel's picture
Submitted by Jeff Juel on

Richard Smith is one of the most prominent critics of peer review. He said: We have little evidence on the effectiveness of peer review, but we have considerable evidence of its defects. In addition to being poor at detecting gross defects and almost useless for detecting fraud, it is slow, expensive, profligate of academic time, highly subjective, something of a lottery, prone to bias and easily abused.

I found the above in chapter 15 in a fascinating book that shines a light on some deeply flawed peer-reviewed scientific literature in the field of paleoclimatology. The book's title is The Hockey Stick Illusion by A.W. Montford.

Peer review is, at best, like a box of chocolates,

Jeff Juel, PE
www.jueltide.com

Submitted by Pierre Lutgen on

Albert Einstein published more than 300 journal articles between 1901 and 1955. Only a single paper of Einstein’s was ever subject to peer review. It was rejected, not by the Germans but by the journal Physical Review in 1936.

50 years ago peer review was still uncommon. It became easier with the commercial introduction of photocopiers in 1959. Peer review is at best an imperfect filter for validity and quality, but has a procrastinating effect on scientific discoveries. It survives by social inertia.

With internet the young generation of academics has a fast access to scientific results and breakthroughs. This audience is much tougher than any reviewer from the old generation.

We congratulate MalariaWorld for opening the door to the publication of scientific results without peer review.

Jeff Juel's picture
Submitted by Jeff Juel on

Pierre,

The Einstein story is compelling. (Peer review is frequently pointless.)

I agree with you. The internet and the next generation will revolutionize science and communication; Hopefully the wheat will quickly, efficiently, and reliably be separated from the chaff.

I have a minuscule knowledge of Latin, but I do know history and I know of Julius Caesar's famous proclamation regarding his conquest of Briton. "Veni Vidi Vici" = "I came, I saw, I conquered".

What is "Veni Vidi Evici" and how does it relate to peer review? I have feeling that this has a clever translation / meaning and relevance. Please elaborate.

Cheers to Malaria World for their revolutionary ideas for advancing science! I hope that this sanity propagates beyond the realm of the science related to combating Malaria.

Jeff Juel, PE
www.jueltide.com

Submitted by Ricardo Ataide on

Indeed, the Einstein anecdote is interesting, but in no way is it a valid point for the discussion. Many other things were done in science that made science advance fast and without being held back and I cannot see anyone wanting their return!

One thing I don't seem to grasp is the notion you have of peer-review being such an awful thing. It seems to me that you see peer-reviewing only as a form of censorship, which i will admit might happen but which is definitely not its intended goal. Journal editors must be good enough to detect when opinions and non-fact based ideas are being used to discredit a paper over the evidence of facts. When a reviewer uses his/her knowledge to review a manuscript it should do so to verify the validity of the conclusions, the use of the statistics, the accuracy of the methods, the ethics of the work involved and ultimately engage in a discussion with the authors that will enable them to elevate their work to a higher level. It may actually be the only time you thoroughly discuss your paper with anyone!

Being reviewed by your peers is not exclusive to science. In all walks of life you get reviewed by your peers. When you submit a job application for a University for instance, or for a position in your institute, or if you are a medical doctor, a certified engineer or a lawyer, your peers are there to review your work. Also, imagine a world where you are sentenced without being trialed by a jury. Isn't that a form of pre-publishing peer review(of a sentence, in this case)? I mean, it is a dramatic one I know, but would you like to be thrown in jail first and have your case published online so that the internet community could judge you? That is assuming of course that they would actually take the time to read what you did and critically analyse it and discuss it. I can tell you right now that if you take participation of MW members in discussing matters pertaining to their day-to-day work-lives as an example, you would spend many a sunset behind bars.

In any case, if pre-publication peer-reviewing is such a degrading thing there are already numerous ways in which you can avoid it and publish your work online (blogs, webpages, facebook, newsletters... you name it). You can also find a publisher and write a book with your ideas. That has always been a good way to avoid peer-reviewing and putting non-conventional ideas out there. A scientific article in a peer-reviewed journal, though, is distinct from all that in the unique sense that it was subjected to a quality check by those who understand more about the facts involved(and I repeat facts because you seem to consider peer-reviewing an 'idea-killing system').

The moment peer-review stops existing, the quality of the work published will drop enormously. Journals will eventually disappear (since there is no assurance of quality, there is neither the point to pay a subscription nor of paying to have your work published since you can do it anywhere, anytime). This will of course result in the disappearance of a very important measure of quality, not only for us researchers (although we have the tools to ultimately judge that quality), but for polity decision-makers, financing bodies and ultimately the general public.
As I said before, the system in place right now is not perfect. It is something like democracy. The worst system of all, excluding every other system. So what we need is to think about peer-reviewing 2.0 and not its elimination.

The MW journal has decided on giving authors the option of not having their work peer-reviewed and maybe that will be the future, but why anyone is afraid to have their work judged i cannot understand. To me, on a personal note, I always look forward to the engaging discussions and the to and fro that results from having my paper discussed with someone who understands very well what it is all about. Many good ideas for work and/or collaborations have resulted from it. But maybe I have been very lucky so far...

Ricardo Ataíde

Submitted by Olivier Briet (not verified) on

For the MWJ decision-making poll record, I agree with the views expressed by Ricardo in his comment "Peer reviewing is not the Big Bad Wolf". Also, the article at the end of the link posted by Andrew convinced me that the open-peer review model is not the way forward.

Submitted by Barbara Oleksyn on

I totally agree with Ricardo!
The peer-reviewing is very helpful for any author, because it may not only point at the errors in the author's reasoning (or even in his or her English), but also suggest some corrections and also conclusions which the author has not discovered himself(herself).The exchange of letters with the reviewer and discussion with him (her) may help the author to draw conclusions which he(she) has not found out.
Even, if the author happens to be more competent than the reviewer,they both, with the help of the editor, may achieve an agreement and bring the paper to a higher level and make it more valuable for the readers.
It seems to me that to abandon the peer-reviewing would be much worse (and even dangerous in some cases) than to continue this procedure. Let's do not throw the baby out with the bath water!
With kind regards,
Barbara Oleksyn,
Faculty of Chemistry, Jagiellonian University, Krakow, Poland

Submitted by Ahmad Mahdavi (not verified) on

Yes, but there must be more considering the importance of malaria, I mean there is no harm to continue peer review beside other more important activities.

William Jobin's picture
Submitted by William Jobin on

I am no authority on this subject, but have occasionally published decent articles in decent journals, and have been doing it for about 50 years. It is true I am occasionally frustrated by a rejection of an article for superficial reasons - like not enough of a literature search at the beginning. But my overall experience has been positive. I have been pointed toward additional references, or forced to develop a more airtight argument. All in all, what one would want Peer Review to do.

Bill Jobin

William Jobin Director of Blue Nile Associates

Andrew Lover's picture
Submitted by Andrew Lover on

Some insights here from a blog that examines the F1000 (Faculty of 1000) open peer-review model; make of it what you will.

http://scholarlykitchen.sspnet.org/2013/03/27/how-rigorous-is-the-post-publication-review-process-at-f1000-research/

Submitted by Ricardo Ataide on

Great finding Andrew. Thanks for sharing!

Ricardo Ataíde

Submitted by JD Charlwood (not verified) on

One thing that I find strange these days is that when submitting a paper one is asked to provide the names and addresses of potential reviewers - so, so much for 'blind' reviewing by ones peers!
I agree that it is through the review system that one gets the chance to improve a manuscript and defend ones conclusions and discover new possibly relevant references and as mentioned it is about the only time that anyone ever comments on (or even reads) our work.
Perhaps reviewers should be paid?

Ingeborg van Schayk's picture
Submitted by Ingeborg van Schayk on

Hi Derek, interesting point re. paying reviewers. Any suggestions as to whom should be paying the reviewers? If it wasn't the author, nor the reader.

Submitted by Olivier Briet (not verified) on

Dear Derek,

The request for suggested reviewers is indeed a bit strange. As an author you can choose your friends or foes (to try to convince them of your argument). Of course it is still totally up to the editor to choose any reviewer in the end.

About reviewer payment: In this thread: http://www.malariaworld.org/blog/paying-authors-open-access-publishing-open-access-30
we started a discussion on paying reviewers. Maybe worth continuing? If peer reviewers are both author-suggested and paid, we will get a competition among favourable reviewers. The end of objective peer-review.

Peter James's picture
Submitted by Peter James on

The idea of a manuscript being published with going through proper scientific scrutiny does not sound or even augur well for the scientific community let alone malaria world. The advantages of a work being peer-reviewed far outweighs it demerits. proper peer-review process not only lend credence to the work done but also ensures that the research was done in a scholarly and scientific manner and that whatever findings of such study are acceptable and reliable. This can be relied upon to inform policies decision and interventions as the case may be. No matter how painstaking it may look, it is still worthy and there should not be an alternative to it.

P James