Re: [AMBER] GTX780Ti consistent error "cudaMemcpy GpuBuffer::Download failed unspecified launch failure."

From: Keiran Rowell <k.rowell.unsw.edu.au>
Date: Fri, 30 May 2014 05:43:53 +0000

Hi Ross and Filip, Apologies for not thanking you for your help earlier, we sent the card in for warranty and I wanted to wait for results before reviving the thread and the process ended up taking a while. I didn't manage to get coolbits working for our old card unfortunately, so I can't tell you what downclocking would have done. They approved the replacement, and we requested a standard 780 but they just did a direct exchange with the same model. The downside is the new GTX780Ti is still having errors, on the plus side I'm getting more informative error reports. My systems (which run fine on our TeslaC2050s) are still crashing almost immediately. However the tutorial 1B explicit water simulation ran for 10ns without any errors reported at all. This didn't work before so.. progress, but I'm not sure if that's down to hardware or updated Amber. The error which is being reported to stdout for my systems is "cudaMemcpy GpuBuffer::Download failed an illegal memory access was encountered" or "cudaMemcpy GpuBuffer::Download failed unspecified launch failure" So I submitted jobs with cuda-memcheck and get "Invalid __global__ ... Address xxxx is out of bounds" type errors, so memory access errors of the form listed in section 3.4 here: http://docs.nvidia.com/cuda/cuda-memcheck/#axzz32zciX8kj My computer science is rusty, but does this indicate a software not hardware issue? Have I got something wrong in my set-up? I've pasted excerpts of a cuda-memcheck error at the bottom of this message, as well as attaching the nohup.out I've updated and am now running AmberTools v14.02, Amber v14.00, Nvidia driver v337.12 (beta), CUDA v6.0.1. pmemd.cuda was compiled with ./configure -cuda gnu. OS is CentOS 6.5 Thank you heaps! Keiran ________________________________________ From: filip fratev [filipfratev.yahoo.com] Sent: 18 April 2014 02:14 To: AMBER Mailing List Subject: Re: [AMBER] GTX780Ti consistent error "cudaMemcpy GpuBuffer::Download failed unspecified launch failure." Hi Keiran and Ross, >I've seen lots of issues with 780Ti since they appear to be right on >the edge of stability with regards to clock speed. In fact this was exactly my case, but my was the SC version. You might also prove that if you use 337 driver and downclock your GPU. BTW 337 is the first driver with this option avalible. Regards, Filip On Wednesday, April 16, 2014 8:45 PM, Ross Walker <ross.rosswalker.co.uk> wrote: Hi Keiran, If driver 337.12 does not fix things then it is almost certainly a bad GPU. I've seen lots of issues with 780Ti since they appear to be right on the edge of stability with regards to clock speed. I recommend returning it for exchange. All the best Ross On 4/15/14, 11:14 PM, "Keiran Rowell" <k.rowell.unsw.edu.au> wrote: >Dear Amber users, > >My research group recently bought a GTX780Ti (model number:GV-N78TOC-3GD) >for running pmemd.cuda, and it's been consistently printing out "NaN" in >the TEMP field of an mdcrd.out and reporting "cudaMemcpy >GpuBuffer::Download failed unspecified launch failure." It will run for a >little bit, often a couple thousand frames, which look normal when >visualised in VMD, before suddenly cutting out. > >I've managed to find a few threads on this issue, however they seemed >mostly to be caused by out-of-date Amber builds, and the only one I saw >about a GTX780Ti (http://archive.ambermd.org/201401/0378.html) seemed to >be just due to a bad card. > >I've made sure my Amber build is up-to-date. "update_amber --update" says >no new updates and "update_amber -v" gives AmberTools version 13.24, >Amber version 12.21 > >This exact same system (and a bunch of analogous ones) has run without >problems on the pair of Tesla C2050's we have. I also tried running the >longer ambermd.org tutorial B1 runs with pmemd.cuda and I get the same >types of crashes. > >To my unexperienced eyes the "cudaMemcpy buffer" error seemed to be a >memory error with the card, so I ran the jobs with cuda-memcheck. However >every time I do this the job runs fine (albiet very slowly) and with 0 >errors in the summary. When it moves onto the next job without memcheck I >then get a crash. > >Suspecting overheating I monitored the card's temprature with nvidia-smi >while on full load, and it gets up to 83C with 70% fan which I don't >think is out of tolerance. > >When I run tests with make test.cuda I get 89 passes and 8 fails, and on >inspection of the .diff file they are just errors in the last decimal >place. Similarly the benchmark suite passes fine and gives ns/day in line >with gpu benchmarks on ambermd.org. > >For completeness sake I've also recompiled Amber with -cuda_SPDP, >-cuda_DPD and the intel compilers and I get the same story. > >I'm running CentOS 6.5, and CUDA 6.0/5.5, with nvidia driver version >v337.12. I installed these by hand as sadly I can't yum install directly >from the cuda and elrepo repositories, as the cuda repo provides v319.37 >which the Amber website says is incompatible. I previously tried v331.49 >but now that ambermd.org/gpu says GTX780Ti is supported I updated the >drivers, and still get the issue. > >My supervisor also mentioned that the motherboard (Gigabyte P35-DS3P) >seems to only support PCI-E 1.x. Could this be the cause of the issue? > >I'm now a bit of a loss as to what the issue is. Is there something I am >missing, or is this just a bad card? > >Gratefully, > >Keiran Rowell > >_______________________________________________ >AMBER mailing list >AMBER.ambermd.org >http://lists.ambermd.org/mailman/listinfo/amber _______________________________________________ AMBER mailing list AMBER.ambermd.org http://lists.ambermd.org/mailman/listinfo/amber _______________________________________________ AMBER mailing list AMBER.ambermd.org http://lists.ambermd.org/mailman/listinfo/amber

_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber

Received on Thu May 29 2014 - 23:00:02 PDT
Custom Search