Re: [AMBER] cudaMemcpy GpuBuffer::Download failed unspecified launch failure

From: Scott Le Grand <varelse2005.gmail.com>
Date: Wed, 17 Jul 2013 16:28:03 -0700

I guess Ross should gather the input files and file a bug...



On Wed, Jul 17, 2013 at 4:22 PM, Hailin Huang <hailin.huang.my.liu.edu>wrote:

> This is a GTX 690..
>
> |------------------- GPU DEVICE INFO --------------------
> |
> | CUDA Capable Devices Detected: 2
> | CUDA Device ID in use: 1
> | CUDA Device Name: GeForce GTX 690
> | CUDA Device Global Mem Size: 2047 MB
> | CUDA Device Num Multiprocessors: 8
> | CUDA Device Core Freq: 1.02 GHz
> |
> |--------------------------------------------------------
>
>
>
> On Wed, Jul 17, 2013 at 7:19 PM, Scott Le Grand <varelse2005.gmail.com
> >wrote:
>
> > This is a GTX 780 right?
> >
> >
> > On Wed, Jul 17, 2013 at 4:01 PM, Hailin Huang <hailin.huang.my.liu.edu
> > >wrote:
> >
> > > Hi Ross,
> > >
> > > Yes we are using the version 12.3 of the GPU code.
> > >
> > > |--------------------- INFORMATION ----------------------
> > > | GPU (CUDA) Version of PMEMD in use: NVIDIA GPU IN USE.
> > > | Version 12.3
> > > |
> > > | 04/24/2013
> > > |
> > > | Implementation by:
> > > | Ross C. Walker (SDSC)
> > > | Scott Le Grand (nVIDIA)
> > > | Duncan Poole (nVIDIA)
> > > |
> > > | CAUTION: The CUDA code is currently experimental.
> > > | You use it at your own risk. Be sure to
> > > | check ALL results carefully.
> > > |
> > > | Precision model in use:
> > > | [SPFP] - Mixed Single/Double/Fixed Point Precision.
> > > | (Default)
> > > |
> > > |--------------------------------------------------------
> > >
> > > |----------------- CITATION INFORMATION -----------------
> > > |
> > > | When publishing work that utilized the CUDA version
> > > | of AMBER, please cite the following in addition to
> > > | the regular AMBER citations:
> > > |
> > > | - Romelia Salomon-Ferrer; Andreas W. Goetz; Duncan
> > > | Poole; Scott Le Grand; Ross C. Walker "Routine
> > > | microsecond molecular dynamics simulations with
> > > | AMBER - Part II: Particle Mesh Ewald", J. Chem.
> > > | Theory Comput., 2012, (In review).
> > > |
> > > | - Andreas W. Goetz; Mark J. Williamson; Dong Xu;
> > > | Duncan Poole; Scott Le Grand; Ross C. Walker
> > > | "Routine microsecond molecular dynamics simulations
> > > | with AMBER - Part I: Generalized Born", J. Chem.
> > > | Theory Comput., 2012, 8 (5), pp1542-1555.
> > > |
> > > | - Scott Le Grand; Andreas W. Goetz; Ross C. Walker
> > > | "SPFP: Speed without compromise - a mixed precision
> > > | model for GPU accelerated molecular dynamics
> > > | simulations.", Comp. Phys. Comm., 2013, 184
> > > | pp374-380, DOI: 10.1016/j.cpc.2012.09.022
> > > |
> > > |--------------------------------------------------------
> > >
> > >
> > > I am sending the input files to your email.
> > >
> > > Thanks!
> > > Richard
> > >
> > >
> > > On Wed, Jul 17, 2013 at 5:53 PM, Ross Walker <ross.rosswalker.co.uk>
> > > wrote:
> > >
> > > > Hi Richard,
> > > >
> > > > Can you confirm that you are running version 12.3 of the GPU code
> > please.
> > > > Check the mdout file where it writes info about the GPUs in use and
> > > > citation information. This corresponds to AMBER 12 up to bugfix.18.
> If
> > > you
> > > > have an earlier version please upgrade and this should fix the
> problem.
> > > If
> > > > you still see the problem with the latest version of the code please
> > post
> > > > your input files for the run that takes the shortest time to crash
> and
> > > > we'll try and debug it.
> > > >
> > > > All the best
> > > > Ross
> > > >
> > > >
> > > >
> > > > On 7/17/13 2:23 PM, "Hailin Huang" <hailin.huang.my.liu.edu> wrote:
> > > >
> > > > >Dear Amber Users,
> > > > >
> > > > >I am simulating 4 variations of the protein kinase A (PKA) tetramer
> > > using
> > > > >explicit solvent. One of them has 1 Mg ion between each R-C
> subunits.
> > > > >First
> > > > >I did the minimization with 10 kcal/mol restraint weight on the
> > > backbone.
> > > > >Then, when I tried to run the heating simulation, the calculation
> > > stopped
> > > > >at step 0, without leaving any error messages in the output file.
> > > > >
> > > > >The end of the output file looks like this:
> > > > >
> > > > >NSTEP = 0 TIME(PS) = 0.000 TEMP(K) = 0.00
> PRESS =
> > > > >-3062.5
> > > > > Etot = -409445.2805 EKtot = 0.0000 EPtot =
> > > > >-409445.2805
> > > > > BOND = 49371.8329 ANGLE = 1469.1848 DIHED =
> > > > >10100.8048
> > > > > 1-4 NB = 2093.2049 1-4 EEL = 24175.6479 VDWAALS =
> > > > >97254.9978
> > > > > EELEC = -593910.9535 EHBOND = 0.0000 RESTRAINT =
> > > > >0.0000
> > > > > EKCMT = 0.0000 VIRIAL = 70377.8192 VOLUME =
> > > > >1064336.6826
> > > > > Density =
> > > > >0.8515
> > > > >
> > > >
> > >
> >
> >--------------------------------------------------------------------------
> > > > >----
> > > > >
> > > > > NMR restraints: Bond = 0.000 Angle = 0.000 Torsion =
> > > 0.000
> > > >
> > >
> >
> >==========================================================================
> > > > >=====
> > > > >
> > > > >
> > > > >In the log file, I got the message "cudaMemcpy GpuBuffer::Download
> > > failed
> > > > >unspecified launch failure". This problem did not occur when I run
> the
> > > > >other 3 variations of PKA structures (one with 2 Mg ions, one with
> no
> > Mg
> > > > >ions and one with no ADPs).
> > > > >
> > > > >I have tried to run the simulation using GTX 680, 690, 780, and on
> the
> > > > >Stampede supercomputers that use Tesla K20m, I got the same result.
> > > > >
> > > > >Then I changed the size of the solvent box (from 12A to 10A), the
> > > problem
> > > > >went away and the calculation completed successfully. However, same
> > > > >problem
> > > > >happened again in the later NVT equilibration simulation. This time
> > the
> > > > >calculation stopped in the middle:
> > > > >
> > > > >NSTEP = 1599000 TIME(PS) = 3598.000 TEMP(K) = 299.24
> PRESS =
> > > > >22.1
> > > > > Etot = -432512.9372 EKtot = 90851.5391 EPtot =
> > > > >-523364.4763
> > > > > BOND = 3740.6596 ANGLE = 10689.2110 DIHED =
> > > > >21485.9623
> > > > > 1-4 NB = 4601.7696 1-4 EEL = 47865.1134 VDWAALS =
> > > > >74622.8763
> > > > > EELEC = -687021.9668 EHBOND = 0.0000 RESTRAINT =
> > > > >651.8984
> > > > > EAMBER (non-restraint) = -524016.3746
> > > > > EKCMT = 38312.0157 VIRIAL = 37612.0350 VOLUME =
> > > > >1465745.9675
> > > > > Density =
> > > > >1.0384
> > > > >
> > > >
> > >
> >
> >--------------------------------------------------------------------------
> > > > >----
> > > > >
> > > > >Same message in the log file "cudaMemcpy GpuBuffer::Download failed
> > > > >unspecified launch failure".
> > > > >
> > > > >
> > > > >When I was simulating 4 variations of the heterodimer of PKA (half
> > size
> > > of
> > > > >the tetramer), same error occurred in the heating stage for the
> > > structure
> > > > >with no Mg ions and the structure with no ADPs, while the ones with
> 1
> > Mg
> > > > >and 2 Mg ions are running just fine. So I changed the size of
> solvent
> > > box
> > > > >from 10A to 8A and this got rid of the problem again. Now I am
> running
> > > the
> > > > >NVT equilibration and "expecting" the calculations to stop in the
> > middle
> > > > >just like what happened to the tetramer with 1 Mg (hopefully not).
> > > > >
> > > > >
> > > > >I would be really grateful if anyone could give me an idea what is
> > going
> > > > >on
> > > > >and possibly with a solution to this annoying "cudaMemcpy
> > > > >GpuBuffer::Download failed unspecified launch failure" problem.
> > > > >
> > > > >
> > > > >Thank you!
> > > > >
> > > > >Best,
> > > > >Richard
> > > > >_______________________________________________
> > > > >AMBER mailing list
> > > > >AMBER.ambermd.org
> > > > >http://lists.ambermd.org/mailman/listinfo/amber
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > AMBER mailing list
> > > > AMBER.ambermd.org
> > > > http://lists.ambermd.org/mailman/listinfo/amber
> > > >
> > > _______________________________________________
> > > AMBER mailing list
> > > AMBER.ambermd.org
> > > http://lists.ambermd.org/mailman/listinfo/amber
> > >
> > _______________________________________________
> > AMBER mailing list
> > AMBER.ambermd.org
> > http://lists.ambermd.org/mailman/listinfo/amber
> >
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
>
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Wed Jul 17 2013 - 17:00:02 PDT
Custom Search