Dear Amber Users,
Several weeks ago I was advised to change the variable gridhi in order
to handle SMD simulations in very long solvent boxes exceeding 512A.
After setting it to 1024, we were able to proceed with pmemd.MPI runs.
Thank you again for the great advice.
However, an SMD simulation for a large system of 529,377 atoms in a solvent
box 720x108x80A in size, run with Amber 14's pmemd.cuda on nvidia Tesla K40m
GPUs with 12GB of "cuda device global memory", (pmemd.cuda version 14.0.1,
06/20/2014) runs into trouble and is halted with the following error message:
------------------------------------------------------------------------------
ERROR: Calculation halted. Periodic box dimensions have changed too
much from their initial values [...]
The GPU code does not automatically reorganize grid cells and thus you
will need to restart the calculation from the previous restart file.
-------------------------------------------------------------------------------
The output file reports correct recognition of the large solvent box's dimensions
(Ewald parameters):
Box X = 686.858 Box Y = 104.032 Box Z = 76.054
Alpha = 90.000 Beta = 90.000 Gamma = 90.000
NFFT1 = 720 NFFT2 = 108 NFFT3 = 80
After a fairly normal-looking (odd pressure) initialization from a restart file
(NSTEP=0, below), the output immediately becomes corrupt, with "NaN"s and
out-of-range values for the majority of parameters (see NSTEP=500).
NSTEP = 0 TIME(PS) = 0.000 TEMP(K) = 149.44 PRESS = -4859.6
Etot = -2417155.3570 EKtot = 157688.4371 EPtot = -2574843.7941
BOND = 179.3862 ANGLE = 619.0596 DIHED = 2320.2887
1-4 NB = 890.4678 1-4 EEL = -11987.1845 VDWAALS = 467609.5171
EELEC = -3034475.3288 EHBOND = 0.0000 RESTRAINT = 0.0000
EKCMT = 52456.5699 VIRIAL = 622666.7810 VOLUME = 5434440.8225
Density = 0.9815
------------------------------------------------------------------------------
NMR restraints: Bond = 0.000 Angle = 0.000 Torsion = 0.000
===============================================================================
NSTEP = 500 TIME(PS) = 1.000 TEMP(K) = NaN PRESS = 3110.9
Etot = NaN EKtot = NaN EPtot = **************
BOND = ************** ANGLE = 521973.0963 DIHED = 0.0000
1-4 NB = ************** 1-4 EEL = ************** VDWAALS = **************
EELEC = ************** EHBOND = 0.0000 RESTRAINT = 0.0000
EKCMT = 12582912.0000 VIRIAL = 3800205.9385 VOLUME = 130758951.6874
Density = 0.0408
------------------------------------------------------------------------------
NMR restraints: Bond = 0.000 Angle = 0.000 Torsion = 0.000
===============================================================================
A much smaller system (53,826 atoms in a 200x64x60A solvent box) runs on GPUs
without any problems. Amber pmemd.MPI binary can also handle this job (although
at a snail's pace on 32 core nodes that I can use). It appears that either the
large SMD simulation is running of of physical memory (but it runs for quite
a while without crashing before it is halted), or that it exceeds some internal
allocation limits and uses unallocated memory until it fails to make sense of
its own results (hence the NaNs ?).
If the latter is the case, what parameters (and in which source files) could
we reset and recompile the code in order to be able to deal with systems of
this size? Any advice and comments would be very welcome.
Best regards, Voytek Kasprzak
Wojciech (Voytek) Kasprzak (Contractor)
Analyst Programmer,
Basic Science Program,
Leidos Biomedical Research, Inc.
Frederick National Laboratory for Cancer Research (FNLCR)
Frederick, MD 21702
(301) 846 5537
http://binkley2.ncifcrf.gov/users/kasprzak
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Fri Apr 22 2016 - 06:00:07 PDT
