Sounds like the Titan is freaking out actually...
Sigh...
On Thu, Jun 20, 2013 at 11:42 AM, ET <sketchfoot.gmail.com> wrote:
> Hi Ross,
>
> What you say makes sense. The error does occur at the start of the
> production segment. However, IMO it is GPU related as if you look in the
> previous segment that generated the restart file for the segment that
> failed, there are ******'s written in the outfile.
>
> Restarting the run with the nfft error inevitably fails, but restarting
> from the previous run works fine in all the cases I have tried so far. So
> the error that lead to NAN in the co-ordinate file only happens
> intermittently.
>
> I can play the trajectories fine so the my input parameters are fine. It's
> just that along the way the written co-ordinates get messed up. This occurs
> with very high frequency (i.e.inevitable) in my dual GPU-setup. The problem
> happens very rarely (but still occurs) in single-GPU.
>
> What do you think?
>
>
> br,
> g
>
>
>
>
> On 20 June 2013 17:37, Ross Walker <ross.rosswalker.co.uk> wrote:
>
> > Hi ET,
> >
> > This cannot possibly be bandwidth limited. This error is triggered in the
> > CPU code (vanilla Fortran) long before any GPU calculations are fired up.
> > It is an initial check by the CPU at the time it is reading in the
> > coordinates. Have you tried this with the CPU version of PMEMD? What does
> > that report?
> >
> > All the best
> > Ross
> >
> >
> >
> > On 6/20/13 7:56 AM, "ET" <sketchfoot.gmail.com> wrote:
> >
> > >Hi Scott,
> > >
> > >Hmmm. I'm thinking this is maybe a bandwidth issue. The problem occurs
> > >when
> > >running the card in dual GPU-config and the fail in the single-GPU mode
> > >occurred on a machine with strangely low results.
> > >
> > >If I load the other card back into dual mode, I'll run the bandwidthtest
> > >again. The slots should both have x16 bandwidth even if both are
> > >populated.
> > >Do you think an unusual number of peripherals such as HDDs will make a
> > >difference?
> > >
> > >Also you say that it is not CUDA code or the GPU. This isn't an OS error
> > >is
> > >it? Or is it AMBER reporting an underlying OS error?
> > >
> > >
> > >##########################################
> > >
> > >[CUDA Bandwidth Test] - Starting...
> > >Running on...
> > >
> > > Device 0: GeForce GTX TITAN
> > > Quick Mode
> > >
> > > Host to Device Bandwidth, 1 Device(s)
> > > PINNED Memory Transfers
> > > Transfer Size (Bytes) Bandwidth(MB/s)
> > > 33554432 3930.7
> > >
> > > Device to Host Bandwidth, 1 Device(s)
> > > PINNED Memory Transfers
> > > Transfer Size (Bytes) Bandwidth(MB/s)
> > > 33554432 2100.4
> > >
> > > Device to Device Bandwidth, 1 Device(s)
> > > PINNED Memory Transfers
> > > Transfer Size (Bytes) Bandwidth(MB/s)
> > > 33554432 220731.1
> > >
> > >
> > >
> > >br,
> > >g
> > >
> > >
> > >On 19 June 2013 18:15, Scott Le Grand <varelse2005.gmail.com> wrote:
> > >
> > >> | ERROR: nfft1 must be in the range of 6 to 512!
> > >> | ERROR: nfft2 must be in the range of 6 to 512!
> > >> | ERROR: nfft3 must be in the range of 6 to 512!
> > >> | ERROR: a must be in the range of 0.10000E+01 to 0.10000E+04!
> > >> | ERROR: b must be in the range of 0.10000E+01 to 0.10000E+04!
> > >> | ERROR: c must be in the range of 0.10000E+01 to 0.10000E+04!
> > >>
> > >> That's not the CUDA code or the GPU. That's something *very* bad with
> > >>your
> > >> machine having somehow convinced itself of something even worse. No
> > >>idea
> > >> what though
> > >>
> > >>
> > >>
> > >>
> > >> On Mon, Jun 17, 2013 at 10:24 PM, ET <sketchfoot.gmail.com> wrote:
> > >>
> > >> > ps. The machine is running in headless mode on centos 6
> > >> >
> > >> >
> > >> > #### bandwitdh test for currently installed TITAN-b:
> > >> >
> > >> > [CUDA Bandwidth Test] - Starting...
> > >> > Running on...
> > >> >
> > >> > Device 0: GeForce GTX TITAN
> > >> > Quick Mode
> > >> >
> > >> > Host to Device Bandwidth, 1 Device(s)
> > >> > PINNED Memory Transfers
> > >> > Transfer Size (Bytes) Bandwidth(MB/s)
> > >> > 33554432 6002.5
> > >> >
> > >> > Device to Host Bandwidth, 1 Device(s)
> > >> > PINNED Memory Transfers
> > >> > Transfer Size (Bytes) Bandwidth(MB/s)
> > >> > 33554432 6165.5
> > >> >
> > >> > Device to Device Bandwidth, 1 Device(s)
> > >> > PINNED Memory Transfers
> > >> > Transfer Size (Bytes) Bandwidth(MB/s)
> > >> > 33554432 220723.8
> > >> >
> > >> >
> > >> >
> > >> > ### deviceQuery
> > >> >
> > >> > deviceQuery Starting...
> > >> >
> > >> > CUDA Device Query (Runtime API) version (CUDART static linking)
> > >> >
> > >> > Detected 1 CUDA Capable device(s)
> > >> >
> > >> > Device 0: "GeForce GTX TITAN"
> > >> > CUDA Driver Version / Runtime Version 5.5 / 5.0
> > >> > CUDA Capability Major/Minor version number: 3.5
> > >> > Total amount of global memory: 6143 MBytes
> > >>(6441730048
> > >> > bytes)
> > >> > (14) Multiprocessors x (192) CUDA Cores/MP: 2688 CUDA Cores
> > >> > GPU Clock rate: 928 MHz (0.93 GHz)
> > >> > Memory Clock rate: 3004 Mhz
> > >> > Memory Bus Width: 384-bit
> > >> > L2 Cache Size: 1572864 bytes
> > >> > Max Texture Dimension Size (x,y,z) 1D=(65536),
> > >> > 2D=(65536,65536), 3D=(4096,4096,4096)
> > >> > Max Layered Texture Size (dim) x layers 1D=(16384) x 2048,
> > >> > 2D=(16384,16384) x 2048
> > >> > Total amount of constant memory: 65536 bytes
> > >> > Total amount of shared memory per block: 49152 bytes
> > >> > Total number of registers available per block: 65536
> > >> > Warp size: 32
> > >> > Maximum number of threads per multiprocessor: 2048
> > >> > Maximum number of threads per block: 1024
> > >> > Maximum sizes of each dimension of a block: 1024 x 1024 x 64
> > >> > Maximum sizes of each dimension of a grid: 2147483647 x 65535
> x
> > >> 65535
> > >> > Maximum memory pitch: 2147483647 bytes
> > >> > Texture alignment: 512 bytes
> > >> > Concurrent copy and kernel execution: Yes with 1 copy
> > >> engine(s)
> > >> > Run time limit on kernels: No
> > >> > Integrated GPU sharing Host Memory: No
> > >> > Support host page-locked memory mapping: Yes
> > >> > Alignment requirement for Surfaces: Yes
> > >> > Device has ECC support: Disabled
> > >> > Device supports Unified Addressing (UVA): Yes
> > >> > Device PCI Bus ID / PCI location ID: 3 / 0
> > >> > Compute Mode:
> > >> > < Exclusive Process (many threads in one process is able to use
> > >> > ::cudaSetDevice() with this device) >
> > >> >
> > >> > deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 5.5, CUDA
> > >> Runtime
> > >> > Version = 5.0, NumDevs = 1, Device0 = GeForce GTX TITAN
> > >> >
> > >> >
> > >> >
> > >> > On 18 June 2013 06:21, ET <sketchfoot.gmail.com> wrote:
> > >> >
> > >> > > Hi,
> > >> > >
> > >> > > I am trying to run NPT simulations using pmemd.cuda using TITAN
> > >> graphics
> > >> > > cards. The equilibration & steps were completed with the CPU
> > >>version
> > >> of
> > >> > > sander.
> > >> > >
> > >> > > I have 2x EVGA superclocked TITAN cards.There have been problems
> > >>with
> > >> the
> > >> > > TITAN graphics cards and I RMA'd one. I have benchmarked both the
> > >>cards
> > >> > > after the RMA and determined that they have no obvious problems
> that
> > >> > would
> > >> > > warrant them being RMA'd again. Though there is an issue with the
> > >>AMBER
> > >> > > cuda code and TITANs in general as discussed in the following
> > >>thread:
> > >> > >
> > >> > > < sorry, can't find it, but it's a ~200 posts long and titled:
> > >> > experiences
> > >> > > with EVGA GTX TITAN Superclocked - memtestG80 - UNDERclocking in
> > >>Linux
> > >> ?
> > >> > >
> > >> > >
> > >> > > As I'm not sure whether this is the same issue, I'm posting this
> in
> > >>a
> > >> new
> > >> > > thread.
> > >> > >
> > >> > > I began running 12 100ns production run using TITAN-a. There were
> no
> > >> > > problems. After waiting for and testing the replacement card
> > >> (TITAN-b), I
> > >> > > put that into the machine as well. So both cards were working on
> > >> > finishing
> > >> > > the total of 300 segments.
> > >> > >
> > >> > > Very shortly, all the segments had failed, though the cards still
> > >> showed
> > >> > a
> > >> > > 100% utilisation and I did not realise until I checked the
> outfiles
> > >> which
> > >> > > showed "ERROR: nfft1 must be in the range of blah, blah, blah"
> > >> (error
> > >> > > posted below). This was pretty weird as I am used to the jobs
> > >>visibly
> > >> > > failing and not carrying on eating resources, whilst doing
> nothing.
> > >> > >
> > >> > > So I pulled the TITAN-a out and restarted the calculations with
> > >>TITAN-b
> > >> > > from the last good rst. Usually 2 back. There have been no
> problems
> > >>at
> > >> > all
> > >> > > and all the simulations have completed.
> > >> > >
> > >> > > My hardware specs are:
> > >> > > Gigabyte GA-X58-UD7 mobo
> > >> > > i7-930 processor
> > >> > > 6GB RAM
> > >> > > 1200 Watt Bequiet power supply
> > >> > >
> > >> > >
> > >> > >
> > >> > > Does anyone have any idea as to what's going on?
> > >> > >
> > >> > >
> > >> > > br,
> > >> > > g
> > >> > >
> > >> > > ############################################################
> > >> > > ############################################################
> > >> > > -------------------------------------------------------
> > >> > > Amber 12 SANDER 2012
> > >> > > -------------------------------------------------------
> > >> > >
> > >> > > | PMEMD implementation of SANDER, Release 12
> > >> > >
> > >> > > | Run on 06/09/2013 at 16:26:10
> > >> > >
> > >> > > [-O]verwriting output
> > >> > >
> > >> > > File Assignments:
> > >> > > | MDIN: prod.in
> > >> > >
> > >> > > | MDOUT: md_4.out
> > >> > >
> > >> > > | INPCRD: md_3.rst
> > >> > >
> > >> > > | PARM: ../leap/TMC_I54V-V82S_Complex_25.parm
> > >> > >
> > >> > > | RESTRT: md_4.rst
> > >> > >
> > >> > > | REFC: refc
> > >> > >
> > >> > > | MDVEL: mdvel
> > >> > >
> > >> > > | MDEN: mden
> > >> > >
> > >> > > | MDCRD: md_4.ncdf
> > >> > >
> > >> > > | MDINFO: mdinfo
> > >> > >
> > >> > >
> > >> > >
> > >> > > Here is the input file:
> > >> > >
> > >> > > Constant pressure constant temperature production run
> > >> > >
> > >> > > &cntrl
> > >> > >
> > >> > > nstlim=2000000, dt=0.002, ntx=5, irest=1, ntpr=250, ntwr=1000,
> > >> > ntwx=500,
> > >> > >
> > >> > > temp0=300.0, ntt=1, tautp=2.0, ioutfm=1, ig=-1, ntxo=2,
> > >> > >
> > >> > >
> > >> > >
> > >> > > ntb=2, ntp=1,
> > >> > >
> > >> > >
> > >> > >
> > >> > > ntc=2, ntf=2,
> > >> > >
> > >> > >
> > >> > >
> > >> > > nrespa=1,
> > >> > >
> > >> > > &end
> > >> > >
> > >> > >
> > >> > >
> > >> > > Note: ig = -1. Setting random seed based on wallclock time in
> > >> > microseconds.
> > >> > >
> > >> > > |--------------------- INFORMATION ----------------------
> > >> > > | GPU (CUDA) Version of PMEMD in use: NVIDIA GPU IN USE.
> > >> > > | Version 12.3
> > >> > > |
> > >> > > | 04/24/2013
> > >> > > |
> > >> > > | Implementation by:
> > >> > > | Ross C. Walker (SDSC)
> > >> > > | Scott Le Grand (nVIDIA)
> > >> > > | Duncan Poole (nVIDIA)
> > >> > > |
> > >> > > | CAUTION: The CUDA code is currently experimental.
> > >> > > | You use it at your own risk. Be sure to
> > >> > > | check ALL results carefully.
> > >> > > |
> > >> > > | Precision model in use:
> > >> > > | [SPFP] - Mixed Single/Double/Fixed Point Precision.
> > >> > > | (Default)
> > >> > > |
> > >> > > |--------------------------------------------------------
> > >> > >
> > >> > > |----------------- CITATION INFORMATION -----------------
> > >> > > |
> > >> > > | When publishing work that utilized the CUDA version
> > >> > > | of AMBER, please cite the following in addition to
> > >> > > | the regular AMBER citations:
> > >> > > |
> > >> > > | - Romelia Salomon-Ferrer; Andreas W. Goetz; Duncan
> > >> > > | Poole; Scott Le Grand; Ross C. Walker "Routine
> > >> > > | microsecond molecular dynamics simulations with
> > >> > > | AMBER - Part II: Particle Mesh Ewald", J. Chem.
> > >> > > | Theory Comput., 2012, (In review).
> > >> > > |
> > >> > > | - Andreas W. Goetz; Mark J. Williamson; Dong Xu;
> > >> > > | Duncan Poole; Scott Le Grand; Ross C. Walker
> > >> > > | "Routine microsecond molecular dynamics simulations
> > >> > > | with AMBER - Part I: Generalized Born", J. Chem.
> > >> > > | Theory Comput., 2012, 8 (5), pp1542-1555.
> > >> > > |
> > >> > > | - Scott Le Grand; Andreas W. Goetz; Ross C. Walker
> > >> > > | "SPFP: Speed without compromise - a mixed precision
> > >> > > | model for GPU accelerated molecular dynamics
> > >> > > | simulations.", Comp. Phys. Comm., 2013, 184
> > >> > > | pp374-380, DOI: 10.1016/j.cpc.2012.09.022
> > >> > > |
> > >> > > |--------------------------------------------------------
> > >> > >
> > >> > > |------------------- GPU DEVICE INFO --------------------
> > >> > > |
> > >> > > | CUDA Capable Devices Detected: 2
> > >> > > | CUDA Device ID in use: 0
> > >> > > | CUDA Device Name: GeForce GTX TITAN
> > >> > > | CUDA Device Global Mem Size: 6143 MB
> > >> > > | CUDA Device Num Multiprocessors: 14
> > >> > > | CUDA Device Core Freq: 0.93 GHz
> > >> > > |
> > >> > > |--------------------------------------------------------
> > >> > >
> > >> > > | ERROR: nfft1 must be in the range of 6 to 512!
> > >> > > | ERROR: nfft2 must be in the range of 6 to 512!
> > >> > > | ERROR: nfft3 must be in the range of 6 to 512!
> > >> > > | ERROR: a must be in the range of 0.10000E+01 to 0.10000E+04!
> > >> > > | ERROR: b must be in the range of 0.10000E+01 to 0.10000E+04!
> > >> > > | ERROR: c must be in the range of 0.10000E+01 to 0.10000E+04!
> > >> > >
> > >> > > Input errors occurred. Terminating execution.
> > >> > > ############################################################
> > >> > > ############################################################
> > >> > >
> > >> > _______________________________________________
> > >> > AMBER mailing list
> > >> > AMBER.ambermd.org
> > >> > http://lists.ambermd.org/mailman/listinfo/amber
> > >> >
> > >> _______________________________________________
> > >> AMBER mailing list
> > >> AMBER.ambermd.org
> > >> http://lists.ambermd.org/mailman/listinfo/amber
> > >>
> > >_______________________________________________
> > >AMBER mailing list
> > >AMBER.ambermd.org
> > >http://lists.ambermd.org/mailman/listinfo/amber
> >
> >
> >
> > _______________________________________________
> > AMBER mailing list
> > AMBER.ambermd.org
> > http://lists.ambermd.org/mailman/listinfo/amber
> >
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
>
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Thu Jun 20 2013 - 12:30:02 PDT
