Re: [AMBER] cudaMemcpy GpuBuffer::Download failed unspecified launch failure

From: Hailin Huang <hailin.huang.my.liu.edu>
Date: Wed, 17 Jul 2013 19:22:31 -0400

This is a GTX 690..

|------------------- GPU DEVICE INFO --------------------
|
| CUDA Capable Devices Detected: 2
| CUDA Device ID in use: 1
| CUDA Device Name: GeForce GTX 690
| CUDA Device Global Mem Size: 2047 MB
| CUDA Device Num Multiprocessors: 8
| CUDA Device Core Freq: 1.02 GHz
|
|--------------------------------------------------------



On Wed, Jul 17, 2013 at 7:19 PM, Scott Le Grand <varelse2005.gmail.com>wrote:

> This is a GTX 780 right?
>
>
> On Wed, Jul 17, 2013 at 4:01 PM, Hailin Huang <hailin.huang.my.liu.edu
> >wrote:
>
> > Hi Ross,
> >
> > Yes we are using the version 12.3 of the GPU code.
> >
> > |--------------------- INFORMATION ----------------------
> > | GPU (CUDA) Version of PMEMD in use: NVIDIA GPU IN USE.
> > | Version 12.3
> > |
> > | 04/24/2013
> > |
> > | Implementation by:
> > | Ross C. Walker (SDSC)
> > | Scott Le Grand (nVIDIA)
> > | Duncan Poole (nVIDIA)
> > |
> > | CAUTION: The CUDA code is currently experimental.
> > | You use it at your own risk. Be sure to
> > | check ALL results carefully.
> > |
> > | Precision model in use:
> > | [SPFP] - Mixed Single/Double/Fixed Point Precision.
> > | (Default)
> > |
> > |--------------------------------------------------------
> >
> > |----------------- CITATION INFORMATION -----------------
> > |
> > | When publishing work that utilized the CUDA version
> > | of AMBER, please cite the following in addition to
> > | the regular AMBER citations:
> > |
> > | - Romelia Salomon-Ferrer; Andreas W. Goetz; Duncan
> > | Poole; Scott Le Grand; Ross C. Walker "Routine
> > | microsecond molecular dynamics simulations with
> > | AMBER - Part II: Particle Mesh Ewald", J. Chem.
> > | Theory Comput., 2012, (In review).
> > |
> > | - Andreas W. Goetz; Mark J. Williamson; Dong Xu;
> > | Duncan Poole; Scott Le Grand; Ross C. Walker
> > | "Routine microsecond molecular dynamics simulations
> > | with AMBER - Part I: Generalized Born", J. Chem.
> > | Theory Comput., 2012, 8 (5), pp1542-1555.
> > |
> > | - Scott Le Grand; Andreas W. Goetz; Ross C. Walker
> > | "SPFP: Speed without compromise - a mixed precision
> > | model for GPU accelerated molecular dynamics
> > | simulations.", Comp. Phys. Comm., 2013, 184
> > | pp374-380, DOI: 10.1016/j.cpc.2012.09.022
> > |
> > |--------------------------------------------------------
> >
> >
> > I am sending the input files to your email.
> >
> > Thanks!
> > Richard
> >
> >
> > On Wed, Jul 17, 2013 at 5:53 PM, Ross Walker <ross.rosswalker.co.uk>
> > wrote:
> >
> > > Hi Richard,
> > >
> > > Can you confirm that you are running version 12.3 of the GPU code
> please.
> > > Check the mdout file where it writes info about the GPUs in use and
> > > citation information. This corresponds to AMBER 12 up to bugfix.18. If
> > you
> > > have an earlier version please upgrade and this should fix the problem.
> > If
> > > you still see the problem with the latest version of the code please
> post
> > > your input files for the run that takes the shortest time to crash and
> > > we'll try and debug it.
> > >
> > > All the best
> > > Ross
> > >
> > >
> > >
> > > On 7/17/13 2:23 PM, "Hailin Huang" <hailin.huang.my.liu.edu> wrote:
> > >
> > > >Dear Amber Users,
> > > >
> > > >I am simulating 4 variations of the protein kinase A (PKA) tetramer
> > using
> > > >explicit solvent. One of them has 1 Mg ion between each R-C subunits.
> > > >First
> > > >I did the minimization with 10 kcal/mol restraint weight on the
> > backbone.
> > > >Then, when I tried to run the heating simulation, the calculation
> > stopped
> > > >at step 0, without leaving any error messages in the output file.
> > > >
> > > >The end of the output file looks like this:
> > > >
> > > >NSTEP = 0 TIME(PS) = 0.000 TEMP(K) = 0.00 PRESS =
> > > >-3062.5
> > > > Etot = -409445.2805 EKtot = 0.0000 EPtot =
> > > >-409445.2805
> > > > BOND = 49371.8329 ANGLE = 1469.1848 DIHED =
> > > >10100.8048
> > > > 1-4 NB = 2093.2049 1-4 EEL = 24175.6479 VDWAALS =
> > > >97254.9978
> > > > EELEC = -593910.9535 EHBOND = 0.0000 RESTRAINT =
> > > >0.0000
> > > > EKCMT = 0.0000 VIRIAL = 70377.8192 VOLUME =
> > > >1064336.6826
> > > > Density =
> > > >0.8515
> > > >
> > >
> >
> >--------------------------------------------------------------------------
> > > >----
> > > >
> > > > NMR restraints: Bond = 0.000 Angle = 0.000 Torsion =
> > 0.000
> > >
> >
> >==========================================================================
> > > >=====
> > > >
> > > >
> > > >In the log file, I got the message "cudaMemcpy GpuBuffer::Download
> > failed
> > > >unspecified launch failure". This problem did not occur when I run the
> > > >other 3 variations of PKA structures (one with 2 Mg ions, one with no
> Mg
> > > >ions and one with no ADPs).
> > > >
> > > >I have tried to run the simulation using GTX 680, 690, 780, and on the
> > > >Stampede supercomputers that use Tesla K20m, I got the same result.
> > > >
> > > >Then I changed the size of the solvent box (from 12A to 10A), the
> > problem
> > > >went away and the calculation completed successfully. However, same
> > > >problem
> > > >happened again in the later NVT equilibration simulation. This time
> the
> > > >calculation stopped in the middle:
> > > >
> > > >NSTEP = 1599000 TIME(PS) = 3598.000 TEMP(K) = 299.24 PRESS =
> > > >22.1
> > > > Etot = -432512.9372 EKtot = 90851.5391 EPtot =
> > > >-523364.4763
> > > > BOND = 3740.6596 ANGLE = 10689.2110 DIHED =
> > > >21485.9623
> > > > 1-4 NB = 4601.7696 1-4 EEL = 47865.1134 VDWAALS =
> > > >74622.8763
> > > > EELEC = -687021.9668 EHBOND = 0.0000 RESTRAINT =
> > > >651.8984
> > > > EAMBER (non-restraint) = -524016.3746
> > > > EKCMT = 38312.0157 VIRIAL = 37612.0350 VOLUME =
> > > >1465745.9675
> > > > Density =
> > > >1.0384
> > > >
> > >
> >
> >--------------------------------------------------------------------------
> > > >----
> > > >
> > > >Same message in the log file "cudaMemcpy GpuBuffer::Download failed
> > > >unspecified launch failure".
> > > >
> > > >
> > > >When I was simulating 4 variations of the heterodimer of PKA (half
> size
> > of
> > > >the tetramer), same error occurred in the heating stage for the
> > structure
> > > >with no Mg ions and the structure with no ADPs, while the ones with 1
> Mg
> > > >and 2 Mg ions are running just fine. So I changed the size of solvent
> > box
> > > >from 10A to 8A and this got rid of the problem again. Now I am running
> > the
> > > >NVT equilibration and "expecting" the calculations to stop in the
> middle
> > > >just like what happened to the tetramer with 1 Mg (hopefully not).
> > > >
> > > >
> > > >I would be really grateful if anyone could give me an idea what is
> going
> > > >on
> > > >and possibly with a solution to this annoying "cudaMemcpy
> > > >GpuBuffer::Download failed unspecified launch failure" problem.
> > > >
> > > >
> > > >Thank you!
> > > >
> > > >Best,
> > > >Richard
> > > >_______________________________________________
> > > >AMBER mailing list
> > > >AMBER.ambermd.org
> > > >http://lists.ambermd.org/mailman/listinfo/amber
> > >
> > >
> > >
> > > _______________________________________________
> > > AMBER mailing list
> > > AMBER.ambermd.org
> > > http://lists.ambermd.org/mailman/listinfo/amber
> > >
> > _______________________________________________
> > AMBER mailing list
> > AMBER.ambermd.org
> > http://lists.ambermd.org/mailman/listinfo/amber
> >
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
>
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Wed Jul 17 2013 - 16:30:04 PDT
Custom Search