Hi Ross,
Unfortunately, I absolutely agree with you.
I also think that the reviewing
process is breaking down, exactly for those reasons
which you have describe. We face a brutal
inflation of new and new and new journals. Some of them publish
anything. Unfortunaly, also the high IF journals often contribute. They
wish sexy clear stories, and please short. No complex discussions,
please. Everything must be perfect and in perfect agreement.
There are some limitations..... What? Limitations? Then "publish
in a specialised journal". In know cases of say mixed exp/theory
papers published in say Nature where I know the simulations
most likely have been completely corrupted. Ask the authors for
trajectories, you will not get them. I met a student at a conference
struggling for almost two years with a method I know would never work for
that. I told him, this is not going to work and suggested an
alternative approach. He was depressed. Year
later, they published their paper, the original method, and everything was
perfect. Hiding errors, perhaps falsifying data, or may be restraining
almost everything (of course without mentioning in the text).
We already started to apply partial self-censoring when publishing
unpopular results, since I am not brave enough and I also need
to survive somehow. Typical argument - publishing this result
"would be undermining our field".
Well, my quess, hiding problems is undermining our field on a long term
perspective, and is underminig credibility of whole
science in the society. Since I can compare several different
fields, I would tell in MD simulations we have quite a large
problem, because there are no quality standards, for
a variety of reasons.
I mean, I see absolutely no problem if imperfect results
are getting published, this is science, but there should
always be some analysis of the accuracy/applicability issues.
(It is much easisier to publish unpopular results in QM,
for a variety of reasons, including the fact that there are
benchmark databases and things can be reproduced or not
reproduced).
best wishes, Jiri
-------------------------------------------------------
Jiri Sponer
Head of Department of Structure and Dynamics of Nucleic Acids
Institute of Biophysics
Academy of Sciences of the Czech Republic
Kralovopolska 135
CZ-61265 Brno
Czech Republic
e-mail: sponer.ncbr.muni.cz
fax: 420 5412 12179
phone: 420 5415 17133
http://www.ibp.cz/
http://www.ibp.cz/en/departments/structure-and-dynamics-of-nucleic-acids/
-----------------------------------------------------------
On Sat, 20 Jun 2015, Ross Walker wrote:
> Date: Sat, 20 Jun 2015 15:54:36 -0700
> From: Ross Walker <ross.rosswalker.co.uk>
> Reply-To: AMBER Mailing List <amber.ambermd.org>
> To: AMBER Mailing List <amber.ambermd.org>
> Subject: Re: [AMBER] accuracy of FEW free energy calculation
>
> Hi Jiri,
>
> I agree these days a lot of people are scared to say things controversial - it's a shame and in my opinion science is suffering because of it. A lot of the issue with the quality of papers though I believe comes because these days the peer review process is more or less at breaking point, if not already broken. The number of journals and papers being published keeps going up, the pressure to publish due to endless 'metrics' introduced by top heavy university administrations and the sheer workload of the average professor is such that most peer review these days is cursory at best. I am guilty of this (having only 15 to 20 minutes free to review a paper say which precludes me trying to test any of the results / dig deeper into what they have done) as is everyone else - the net result is that these days peer review really just serves to kick out the badly written / obviously wrong / just plain dull papers and instead the fact something is published says nothing about it's reliability whatsoever. Perhaps we need the yelp / tripadvisor equivalent for science. Search the web and see what people are saying about what was published (I am not convinced citation count helps here since a citation for being wrong is still a +1), test the bits that are relevant to you and then make up your own mind based on experience.
>
> And of course the caveat emptor always applies. If it sounds too good to be true, it probably is.
>
> All the best
> Ross
>
>> On Jun 20, 2015, at 10:38 AM, Jiri Sponer <sponer.ncbr.muni.cz> wrote:
>>
>> I think it has been quite interesting discussion and I in fact I would
>> appreciate to see more such say "controversial" discussions.
>> In general I see that year by year papers are getting on average more and
>> more "inflated". Publication of negative and unpopular results is definitely NOT rewarded in contemporary science,
>> I would even tell it is risky.
>>
>> I need to emphasize that I do not want to judge that specific problem due to lack of experience.
>>
>> Jiri
>>
>>
>>
>>
>> On Sat, 20 Jun 2015, Jason Swails wrote:
>>
>>> Date: Sat, 20 Jun 2015 12:29:10 -0400
>>> From: Jason Swails <jason.swails.gmail.com>
>>> Reply-To: AMBER Mailing List <amber.ambermd.org>
>>> To: AMBER Mailing List <amber.ambermd.org>
>>> Subject: Re: [AMBER] accuracy of FEW free energy calculation
>>> On Sat, Jun 20, 2015 at 1:08 AM, Albert <mailmd2011.gmail.com> wrote:
>>>
>>>> Hi Ross:
>>>>
>>>> thanks a lot for such honest comments.
>>>>
>>>> I just knew it, they were using many cheating and tricks in that JACS paper
>>>> and made such beautiful story to the outside. As far as I know, Schrodinger
>>>> is going to sell this product to many big pharmaceutical company with
>>>> bloody price that you cannot imagine. However, after asthmatically testing,
>>>> most of them dump it which I could expect in advance.
>>>>
>>>> For science part, the Schrodinger FEP Mapper solvate the protein/ligand
>>>> complex in a 5A TIP3P box and then produce 5ns MD in all for ligand. It
>>>> assigned OPLS-2.1, which claimed to be the most accurate FF for small
>>>> molecules, for the system. The job is primarily running on GPU. I don't
>>>> know whether this will lead to the converge of the calculation or not for
>>>> Alchemical FEP. Meanwhile, in this mapper stuff, if one provide
>>>> experimental date in the input file, they will use some mathematical
>>>> function to "MAKE" you final results correlate the data one provided at the
>>>> beginning. Probably that's one of the key reason why the plot in that JACS
>>>> paper looks so perfect. If this is the case, then the plot is totally a
>>>> fake and unreliable, it is a totally commercial strategy.
>>>>
>>>
>>> I think this is a little pessimistic. If you are having trouble
>>> correlating your results in the same way that Schrodinger did in their
>>> paper, you should contact them for help. They wouldn't stay in business if
>>> their methods were based on cheating and could only achieve good results on
>>> hand-tuned retrospective studies. I haven't read the paper yet, and it's
>>> entirely possible that the method doesn't work as well as the paper would
>>> suggest, but I know and respect a lot of the scientists that work at
>>> Schrodinger. Your accusations are at odds with my experience with them.
>>> So I would suggest trying to reconcile why your simulations are not
>>> yielding similar results to theirs rather than simply chalking it up to
>>> scientific dishonesty and corporate deceit.
>>>
>>> Another comment I'll make is that papers are subject to peer review which,
>>> while not a perfect system, has been quite effective at pushing forward
>>> good science and filtering out the bad (at least eventually). By contrast,
>>> none of our emails in which we answer questions here are subject to peer
>>> review, so I would suggest treating our responses with at least as much
>>> (and usually more) skepticism than, for instance, JACS papers. This isn't
>>> to say you should blindly accept anything people publish in JACS, but you
>>> shouldn't consider what we say here to be any more reliable (treat them as
>>> tips to help give you a fuller picture and/or get you pointed in the right
>>> direction).
>>>
>>> Good luck,
>>> Jason
>>>
>>> --
>>> Jason M. Swails
>>> BioMaPS,
>>> Rutgers University
>>> Postdoctoral Researcher
>>> _______________________________________________
>>> AMBER mailing list
>>> AMBER.ambermd.org
>>> http://lists.ambermd.org/mailman/listinfo/amber
>>>
>> _______________________________________________
>> AMBER mailing list
>> AMBER.ambermd.org
>> http://lists.ambermd.org/mailman/listinfo/amber
>
>
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
>
>
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Sun Jun 21 2015 - 02:30:03 PDT
