Re: [AMBER] NaN error in .rst files

From: Marek Maly <marek.maly.ujep.cz>
Date: Wed, 26 Jan 2011 17:23:01 +0100

Hello All,

it seems that the error is reproducible as indicated
Peker sooner - see data below (the verification test was done on the same
machine as the original one, with the same random seed) (although that
restarted simulation with ig=-1
helped here to continue simulation from the critical point, but of course,
this is not the systematic solution and the given error might occure also
in
future of this run ...).

So now I will try to verify this error

#1 - on another machine (again with GTX 470)

#2 - on TESLA C2050, unfortunately here I have no bugfix 12 applied, but
box is not small ( 106.2227312 106.2227312 106.2227312 109.4712190
109.4712190 109.4712190 ) so
there shouldn.t be problems with 10 A cutoff or am I wrong ?. Anyway I can
later recompile with the
bugfix 12 as well.

#3
on CPUs which can also help to find out the source of the error.

When I have some results I will post them here.

Best wishes,

      Marek



############### ORIGINAL RUN ##########################
############### ORIGINAL RUN ##########################
############### ORIGINAL RUN ##########################
############### ORIGINAL RUN ##########################
############### ORIGINAL RUN ##########################



           -------------------------------------------------------
           Amber 11 SANDER 2010
           -------------------------------------------------------

| PMEMD implementation of SANDER, Release 11

| Run on 01/24/2011 at 00:12:00

   [-O]verwriting output

File Assignments:
| MDIN: equil_G4malTRI_10BIL.in
| MDOUT: equil8_G4malTRI_10BIL.out
| INPCRD: equil7_G4malTRI_10BIL.rst
| PARM: G4malTRI_10BIL.prmtop
| RESTRT: equil8_G4malTRI_10BIL.rst
| REFC: refc
| MDVEL: mdvel
| MDEN: mden
| MDCRD: equil8_G4malTRI_10BIL.mdcrd
| MDINFO: mdinfo


  Here is the input file:

heat ras-raf
  &cntrl
   imin=0,irest=1,ntx=5,
   nstlim=250000,dt=0.002,
   ntc=2,ntf=2,
   cut=10.0, ntb=2, ntp=1, taup=1.0,
   ntpr=5000, ntwx=5000,
   ntt=3, gamma_ln=2.0, ig=-1,
     temp0=298,
  /




|--------------------- INFORMATION ----------------------
| GPU (CUDA) Version of PMEMD in use: NVIDIA GPU IN USE.
| Version 2.1
|
| 12/20/2010
|
|
| Implementation by:
| Ross C. Walker (SDSC)
| Scott Le Grand (nVIDIA)
| Duncan Poole (nVIDIA)
|
| CAUTION: The CUDA code is currently experimental.
| You use it at your own risk. Be sure to
| check ALL results carefully.
|
| Precision model in use:
| [SPDP] - Hybrid Single/Double Precision (Default).
|
|--------------------------------------------------------

|------------------- GPU DEVICE INFO --------------------
|
| CUDA Capable Devices Detected: 1
| CUDA Device ID in use: 0
| CUDA Device Name: GeForce GTX 470
| CUDA Device Global Mem Size: 1279 MB
| CUDA Device Num Multiprocessors: 14
| CUDA Device Core Freq: 1.22 GHz
|
|--------------------------------------------------------


| Conditional Compilation Defines Used:
| DIRFRC_COMTRANS
| DIRFRC_EFS
| DIRFRC_NOVEC
| PUBFFT
| FFTLOADBAL_2PROC
| BINTRAJ
| CUDA

| Largest sphere to fit in unit cell has radius = 43.365

| New format PARM file being parsed.
| Version = 1.000 Date = 01/21/11 Time = 18:45:59

| Note: 1-4 EEL scale factors were NOT found in the topology file.
| Using default value of 1.2.

| Note: 1-4 VDW scale factors were NOT found in the topology file.
| Using default value of 2.0.
| Duplicated 0 dihedrals

| Duplicated 0 dihedrals

--------------------------------------------------------------------------------
    1. RESOURCE USE:
--------------------------------------------------------------------------------

  getting new box info from bottom of inpcrd

  NATOM = 92246 NTYPES = 16 NBONH = 89280 MBONA = 2945
  NTHETH = 6302 MTHETA = 4240 NPHIH = 11120 MPHIA = 8029
  NHPARM = 0 NPARM = 0 NNB = 147846 NRES = 29261
  NBONA = 2945 NTHETA = 4240 NPHIA = 8029 NUMBND = 33
  NUMANG = 67 NPTRA = 38 NATYP = 25 NPHB = 1
  IFBOX = 2 NMXRS = 77 IFCAP = 0 NEXTRA = 0
  NCOPY = 0

| Coordinate Index Table dimensions: 17 17 17
| Direct force subcell size = 6.2484 6.2484 6.2484

      BOX TYPE: TRUNCATED OCTAHEDRON

--------------------------------------------------------------------------------
    2. CONTROL DATA FOR THE RUN
--------------------------------------------------------------------------------



General flags:
      imin = 0, nmropt = 0

Nature and format of input:
      ntx = 5, irest = 1, ntrx = 1

Nature and format of output:
      ntxo = 1, ntpr = 5000, ntrx = 1, ntwr
= 500
      iwrap = 0, ntwx = 5000, ntwv = 0, ntwe
= 0
      ioutfm = 0, ntwprt = 0, idecomp = 0,
rbornstat= 0

Potential function:
      ntf = 2, ntb = 2, igb = 0, nsnb
= 25
      ipol = 0, gbsa = 0, iesp = 0
      dielc = 1.00000, cut = 10.00000, intdiel = 1.00000

Frozen or restrained atoms:
      ibelly = 0, ntr = 0

Molecular dynamics:
      nstlim = 250000, nscm = 1000, nrespa = 1
      t = 0.00000, dt = 0.00200, vlimit = -1.00000

Langevin dynamics temperature regulation:
      ig = 676075
      temp0 = 298.00000, tempi = 0.00000, gamma_ln= 2.00000

Pressure regulation:
      ntp = 1
      pres0 = 1.00000, comp = 44.60000, taup = 1.00000

SHAKE:
      ntc = 2, jfastw = 0
      tol = 0.00001

| Intermolecular bonds treatment:
| no_intermolecular_bonds = 1

| Energy averages sample interval:
| ene_avg_sampling = 5000

Ewald parameters:
      verbose = 0, ew_type = 0, nbflag = 1, use_pme
= 1
      vdwmeth = 1, eedmeth = 1, netfrc = 1
      Box X = 106.223 Box Y = 106.223 Box Z = 106.223
      Alpha = 109.471 Beta = 109.471 Gamma = 109.471
      NFFT1 = 108 NFFT2 = 108 NFFT3 = 108
      Cutoff= 10.000 Tol =0.100E-04
      Ewald Coefficient = 0.27511
      Interpolation order = 4

--------------------------------------------------------------------------------
    3. ATOMIC COORDINATES AND VELOCITIES
--------------------------------------------------------------------------------


  begin time read from input coords = 3700.000 ps


  Number of triangulated 3-point waters found: 28820

      Sum of charges from parm topology file = -0.00266701
      Forcing neutrality...

| Dynamic Memory, Types Used:
| Reals 2846082
| Integers 2613755

| Nonbonded Pairs Initial Allocation: 27902108

| GPU memory information:
| KB of GPU memory in use: 876823
| KB of CPU memory in use: 77099

--------------------------------------------------------------------------------
    4. RESULTS
--------------------------------------------------------------------------------

  ---------------------------------------------------
  APPROXIMATING switch and d/dx switch using CUBIC SPLINE INTERPOLATION
  using 5000.0 points per unit in tabled values
  TESTING RELATIVE ERROR over r ranging from 0.0 to cutoff
| CHECK switch(x): max rel err = 0.2738E-14 at 2.422500
| CHECK d/dx switch(x): max rel err = 0.8314E-11 at 2.736960
  ---------------------------------------------------
|---------------------------------------------------
| APPROXIMATING direct energy using CUBIC SPLINE INTERPOLATION
| with 50.0 points per unit in tabled values
| Relative Error Limit not exceeded for r .gt. 2.33
| APPROXIMATING direct force using CUBIC SPLINE INTERPOLATION
| with 50.0 points per unit in tabled values
| Relative Error Limit not exceeded for r .gt. 2.80
|---------------------------------------------------

  NSTEP = 5000 TIME(PS) = 3710.000 TEMP(K) = 298.54 PRESS =
19.7
  Etot = -208138.5945 EKtot = 55606.0703 EPtot =
-263744.6648
  BOND = 12770.3337 ANGLE = 5327.0090 DIHED =
1796.1712
  1-4 NB = 1220.6297 1-4 EEL = 32614.9838 VDWAALS =
44968.0029
  EELEC = -362441.7951 EHBOND = 0.0000 RESTRAINT =
0.0000
  EKCMT = 25890.2734 VIRIAL = 25498.7611 VOLUME =
920970.7237
                                                     Density =
1.0185
  ------------------------------------------------------------------------------


  NSTEP = 10000 TIME(PS) = 3720.000 TEMP(K) = 296.22 PRESS =
73.8
  Etot = -208072.6867 EKtot = 55172.7422 EPtot =
-263245.4289
  BOND = 12797.2804 ANGLE = 5280.3633 DIHED =
1793.7009
  1-4 NB = 1236.9252 1-4 EEL = 32530.0890 VDWAALS =
45147.6190
  EELEC = -362031.4066 EHBOND = 0.0000 RESTRAINT =
0.0000
  EKCMT = 25827.2976 VIRIAL = 24357.5606 VOLUME =
921836.9062
                                                     Density =
1.0176
  ------------------------------------------------------------------------------


  NSTEP = 15000 TIME(PS) = 3730.000 TEMP(K) = 299.32 PRESS =
133.8
  Etot = -207140.0864 EKtot = 55750.0508 EPtot =
-262890.1372
  BOND = 12813.3463 ANGLE = 5316.1231 DIHED =
1777.2270
  1-4 NB = 1214.8564 1-4 EEL = 32504.7227 VDWAALS =
45134.9078
  EELEC = -361651.3206 EHBOND = 0.0000 RESTRAINT =
0.0000
  EKCMT = 26061.3868 VIRIAL = 23397.2429 VOLUME =
922304.1794
                                                     Density =
1.0171
  ------------------------------------------------------------------------------


  NSTEP = 20000 TIME(PS) = 3740.000 TEMP(K) = NaN PRESS
= NaN
  Etot = NaN EKtot = NaN EPtot
= NaN
  BOND = 0.0000 ANGLE = 795708.3759 DIHED =
0.0000
  1-4 NB = 0.0000 1-4 EEL = 0.0000 VDWAALS =
-1408.8764
  EELEC = NaN EHBOND = 0.0000 RESTRAINT =
0.0000
  EKCMT = 0.0000 VIRIAL = NaN VOLUME
= NaN
                                                     Density
= NaN
  ------------------------------------------------------------------------------


  NSTEP = 25000 TIME(PS) = 3750.000 TEMP(K) = NaN PRESS
= NaN
  Etot = NaN EKtot = NaN EPtot
= NaN
  BOND = 0.0000 ANGLE = 795708.3759 DIHED =
0.0000
  1-4 NB = 0.0000 1-4 EEL = 0.0000 VDWAALS =
-1408.8764
  EELEC = NaN EHBOND = 0.0000 RESTRAINT =
0.0000
  EKCMT = 0.0000 VIRIAL = NaN VOLUME
= NaN
                                                     Density
= NaN
  ------------------------------------------------------------------------------




############### VERIFICATION RUN ######################
############### VERIFICATION RUN ######################
############### VERIFICATION RUN ######################
############### VERIFICATION RUN ######################
############### VERIFICATION RUN ######################



           -------------------------------------------------------
           Amber 11 SANDER 2010
           -------------------------------------------------------

| PMEMD implementation of SANDER, Release 11

| Run on 01/26/2011 at 15:38:33

   [-O]verwriting output

File Assignments:
| MDIN: equil_G4malTRI_10BIL.in
| MDOUT: equil8_G4malTRI_10BIL.out
| INPCRD: equil7_G4malTRI_10BIL.rst
| PARM: G4malTRI_10BIL.prmtop
| RESTRT: equil8_G4malTRI_10BIL.rst
| REFC: refc
| MDVEL: mdvel
| MDEN: mden
| MDCRD: equil8_G4malTRI_10BIL.mdcrd
| MDINFO: mdinfo


  Here is the input file:

heat ras-raf
  &cntrl
   imin=0,irest=1,ntx=5,
   nstlim=250000,dt=0.002,
   ntc=2,ntf=2,
   cut=10.0, ntb=2, ntp=1, taup=1.0,
   ntpr=5000, ntwx=5000,
   ntt=3, gamma_ln=2.0, ig=676075,
     temp0=298,
  /




|--------------------- INFORMATION ----------------------
| GPU (CUDA) Version of PMEMD in use: NVIDIA GPU IN USE.
| Version 2.1
|
| 12/20/2010
|
|
| Implementation by:
| Ross C. Walker (SDSC)
| Scott Le Grand (nVIDIA)
| Duncan Poole (nVIDIA)
|
| CAUTION: The CUDA code is currently experimental.
| You use it at your own risk. Be sure to
| check ALL results carefully.
|
| Precision model in use:
| [SPDP] - Hybrid Single/Double Precision (Default).
|
|--------------------------------------------------------

|------------------- GPU DEVICE INFO --------------------
|
| CUDA Capable Devices Detected: 1
| CUDA Device ID in use: 0
| CUDA Device Name: GeForce GTX 470
| CUDA Device Global Mem Size: 1279 MB
| CUDA Device Num Multiprocessors: 14
| CUDA Device Core Freq: 1.22 GHz
|
|--------------------------------------------------------


| Conditional Compilation Defines Used:
| DIRFRC_COMTRANS
| DIRFRC_EFS
| DIRFRC_NOVEC
| PUBFFT
| FFTLOADBAL_2PROC
| BINTRAJ
| CUDA

| Largest sphere to fit in unit cell has radius = 43.365

| New format PARM file being parsed.
| Version = 1.000 Date = 01/21/11 Time = 18:45:59

| Note: 1-4 EEL scale factors were NOT found in the topology file.
| Using default value of 1.2.

| Note: 1-4 VDW scale factors were NOT found in the topology file.
| Using default value of 2.0.
| Duplicated 0 dihedrals

| Duplicated 0 dihedrals

--------------------------------------------------------------------------------
    1. RESOURCE USE:
--------------------------------------------------------------------------------

  getting new box info from bottom of inpcrd

  NATOM = 92246 NTYPES = 16 NBONH = 89280 MBONA = 2945
  NTHETH = 6302 MTHETA = 4240 NPHIH = 11120 MPHIA = 8029
  NHPARM = 0 NPARM = 0 NNB = 147846 NRES = 29261
  NBONA = 2945 NTHETA = 4240 NPHIA = 8029 NUMBND = 33
  NUMANG = 67 NPTRA = 38 NATYP = 25 NPHB = 1
  IFBOX = 2 NMXRS = 77 IFCAP = 0 NEXTRA = 0
  NCOPY = 0

| Coordinate Index Table dimensions: 17 17 17
| Direct force subcell size = 6.2484 6.2484 6.2484

      BOX TYPE: TRUNCATED OCTAHEDRON

--------------------------------------------------------------------------------
    2. CONTROL DATA FOR THE RUN
--------------------------------------------------------------------------------



General flags:
      imin = 0, nmropt = 0

Nature and format of input:
      ntx = 5, irest = 1, ntrx = 1

Nature and format of output:
      ntxo = 1, ntpr = 5000, ntrx = 1, ntwr
= 500
      iwrap = 0, ntwx = 5000, ntwv = 0, ntwe
= 0
      ioutfm = 0, ntwprt = 0, idecomp = 0,
rbornstat= 0

Potential function:
      ntf = 2, ntb = 2, igb = 0, nsnb
= 25
      ipol = 0, gbsa = 0, iesp = 0
      dielc = 1.00000, cut = 10.00000, intdiel = 1.00000

Frozen or restrained atoms:
      ibelly = 0, ntr = 0

Molecular dynamics:
      nstlim = 250000, nscm = 1000, nrespa = 1
      t = 0.00000, dt = 0.00200, vlimit = -1.00000

Langevin dynamics temperature regulation:
      ig = 676075
      temp0 = 298.00000, tempi = 0.00000, gamma_ln= 2.00000

Pressure regulation:
      ntp = 1
      pres0 = 1.00000, comp = 44.60000, taup = 1.00000

SHAKE:
      ntc = 2, jfastw = 0
      tol = 0.00001

| Intermolecular bonds treatment:
| no_intermolecular_bonds = 1

| Energy averages sample interval:
| ene_avg_sampling = 5000

Ewald parameters:
      verbose = 0, ew_type = 0, nbflag = 1, use_pme
= 1
      vdwmeth = 1, eedmeth = 1, netfrc = 1
      Box X = 106.223 Box Y = 106.223 Box Z = 106.223
      Alpha = 109.471 Beta = 109.471 Gamma = 109.471
      NFFT1 = 108 NFFT2 = 108 NFFT3 = 108
      Cutoff= 10.000 Tol =0.100E-04
      Ewald Coefficient = 0.27511
      Interpolation order = 4

--------------------------------------------------------------------------------
    3. ATOMIC COORDINATES AND VELOCITIES
--------------------------------------------------------------------------------


  begin time read from input coords = 3700.000 ps


  Number of triangulated 3-point waters found: 28820

      Sum of charges from parm topology file = -0.00266701
      Forcing neutrality...

| Dynamic Memory, Types Used:
| Reals 2846082
| Integers 2613755

| Nonbonded Pairs Initial Allocation: 27902108

| GPU memory information:
| KB of GPU memory in use: 876823
| KB of CPU memory in use: 77099

--------------------------------------------------------------------------------
    4. RESULTS
--------------------------------------------------------------------------------

  ---------------------------------------------------
  APPROXIMATING switch and d/dx switch using CUBIC SPLINE INTERPOLATION
  using 5000.0 points per unit in tabled values
  TESTING RELATIVE ERROR over r ranging from 0.0 to cutoff
| CHECK switch(x): max rel err = 0.2738E-14 at 2.422500
| CHECK d/dx switch(x): max rel err = 0.8314E-11 at 2.736960
  ---------------------------------------------------
|---------------------------------------------------
| APPROXIMATING direct energy using CUBIC SPLINE INTERPOLATION
| with 50.0 points per unit in tabled values
| Relative Error Limit not exceeded for r .gt. 2.33
| APPROXIMATING direct force using CUBIC SPLINE INTERPOLATION
| with 50.0 points per unit in tabled values
| Relative Error Limit not exceeded for r .gt. 2.80
|---------------------------------------------------

  NSTEP = 5000 TIME(PS) = 3710.000 TEMP(K) = 298.54 PRESS =
19.7
  Etot = -208138.5945 EKtot = 55606.0703 EPtot =
-263744.6648
  BOND = 12770.3337 ANGLE = 5327.0090 DIHED =
1796.1712
  1-4 NB = 1220.6297 1-4 EEL = 32614.9838 VDWAALS =
44968.0029
  EELEC = -362441.7951 EHBOND = 0.0000 RESTRAINT =
0.0000
  EKCMT = 25890.2734 VIRIAL = 25498.7611 VOLUME =
920970.7237
                                                     Density =
1.0185
  ------------------------------------------------------------------------------


  NSTEP = 10000 TIME(PS) = 3720.000 TEMP(K) = 296.22 PRESS =
73.8
  Etot = -208072.6867 EKtot = 55172.7422 EPtot =
-263245.4289
  BOND = 12797.2804 ANGLE = 5280.3633 DIHED =
1793.7009
  1-4 NB = 1236.9252 1-4 EEL = 32530.0890 VDWAALS =
45147.6190
  EELEC = -362031.4066 EHBOND = 0.0000 RESTRAINT =
0.0000
  EKCMT = 25827.2976 VIRIAL = 24357.5606 VOLUME =
921836.9062
                                                     Density =
1.0176
  ------------------------------------------------------------------------------


  NSTEP = 15000 TIME(PS) = 3730.000 TEMP(K) = 299.32 PRESS =
133.8
  Etot = -207140.0864 EKtot = 55750.0508 EPtot =
-262890.1372
  BOND = 12813.3463 ANGLE = 5316.1231 DIHED =
1777.2270
  1-4 NB = 1214.8564 1-4 EEL = 32504.7227 VDWAALS =
45134.9078
  EELEC = -361651.3206 EHBOND = 0.0000 RESTRAINT =
0.0000
  EKCMT = 26061.3868 VIRIAL = 23397.2429 VOLUME =
922304.1794
                                                     Density =
1.0171
  ------------------------------------------------------------------------------


  NSTEP = 20000 TIME(PS) = 3740.000 TEMP(K) = NaN PRESS
= NaN
  Etot = NaN EKtot = NaN EPtot
= NaN
  BOND = 0.0000 ANGLE = 795708.3759 DIHED =
0.0000
  1-4 NB = 0.0000 1-4 EEL = 0.0000 VDWAALS =
-1408.8764
  EELEC = NaN EHBOND = 0.0000 RESTRAINT =
0.0000
  EKCMT = 0.0000 VIRIAL = NaN VOLUME
= NaN
                                                     Density
= NaN
  ------------------------------------------------------------------------------


  NSTEP = 25000 TIME(PS) = 3750.000 TEMP(K) = NaN PRESS
= NaN
  Etot = NaN EKtot = NaN EPtot
= NaN
  BOND = 0.0000 ANGLE = 795708.3759 DIHED =
0.0000
  1-4 NB = 0.0000 1-4 EEL = 0.0000 VDWAALS =
-1408.8764
  EELEC = NaN EHBOND = 0.0000 RESTRAINT =
0.0000
  EKCMT = 0.0000 VIRIAL = NaN VOLUME
= NaN
                                                     Density
= NaN
  ------------------------------------------------------------------------------






Dne Wed, 26 Jan 2011 15:04:16 +0100 case <case.biomaps.rutgers.edu>
napsal/-a:

> On Wed, Jan 26, 2011, Marek Maly wrote:
>>
>> I recently also obtained NaN problem in one of my simulation using
>> pmemd.cuda on GTX 470:
>> see - relevant data below. Unfortunately it is unreproducable as I am
>> using ig=-1.
>
> You *can* re-run things in this circumstance: pmemd will print out the
> value
> of the random seed that was actually used, and you can put that back into
> your input file. This can be helpful to see if the problem is the same
> every
> time.
>
> ....dac
>
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
>
> __________ Informace od ESET NOD32 Antivirus, verze databaze 5820
> (20110126) __________
>
> Tuto zpravu proveril ESET NOD32 Antivirus.
>
> http://www.eset.cz
>
>
>


-- 
Tato zpráva byla vytvořena převratným poštovním klientem Opery:  
http://www.opera.com/mail/
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Wed Jan 26 2011 - 09:00:02 PST
Custom Search