On Wed, Aug 5, 2015 at 3:00 PM, Joseph Baker <bakerj.tcnj.edu> wrote:
> Hi Ian,
>
> Thanks for the reply. This appears to happen across several GPU types here,
> and the machines have been rebooted recently (this also happened before the
> reboot). I have never seen this for any of my larger systems, just these
> fairly tiny dipeptide+water box cases. Also, a colleague of mine has seen
> this behavior on NVidia Tesla K80s. Running systems again with a different
> seed sometimes gets them all the way through to the end without an NaN
> error, and sometimes it does not. Looking a little more closely, the NaN's
> appear to be showing up for a handful of water molecules in the simulation
> (verified by writing out several frames from the nc file using cpptraj as
> rst7 and looking at the coordinates). I am writing to binary nc file, so
> too large coordinates shouldn't be the problem from what I understand.
>
The TIPnP water model does not have any van der Waals terms on the
hydrogens -- it's expected that the oxygen radius is big enough to shield
the hydrogens from a catastrophic collapse.
But it may happen that occasionally (very rarely) water molecules get close
together, and the electrostatic and van der Waals forces become large for a
couple interactions (but with different signs). Since pmemd.cuda
accumulates forces in fixed precision (using an unsigned long long int),
it's possible that there's an overflow leading to a NaN (particularly if
the density is high at that step).
Are you using the Monte Carlo barostat? It may be that a proposed volume
change is particularly unfavorable (and should be summarily rejected), but
it's sending the simulation to NaNdyland as an unfortunate side effect...
It would also be good to use the validation suite that Ross Walker has
posted on the mailing list before to make sure the GPUs you're using are
still good.
Hope this helps,
Jason
--
Jason M. Swails
BioMaPS,
Rutgers University
Postdoctoral Researcher
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Wed Aug 05 2015 - 17:30:03 PDT
