Hi Keiran,
In the absence of a hardware issue 'Invalid write etc' do indeed imply a
software error. However if the hardware is faulty then the instruction
stack may be getting corrupted in which case all bets are off. It's hard
to know what it going on here but I still suspect a bad GPU. The 780Ti's
have been awful in terms of reliability. I think I'll update the AMBER
website to make it clear that these are not recommended. If you look at
the performance they come in faster than even the GTX-Titan-Black and yet
are lower binned silicon. Crazy overclocking. :-( A few things I would try
just to see if it helps.
Upgrade to the very latest driver - it is 337.19 I believe. Switch to CUDA
5.0 (this is what we recommend and test with), update your motherboard
bios, do a clean power off reboot and then see what happens. Try the GPU
Validation suite I built here:
https://dl.dropboxusercontent.com/u/708185/GPU_Validation_Test.tar.gz
All the best
Ross
On 5/29/14, 10:43 PM, "Keiran Rowell" <k.rowell.unsw.edu.au> wrote:
>Hi Ross and Filip,
>
>Apologies for not thanking you for your help earlier, we sent the card in
>for warranty and I wanted to wait for results before reviving the thread
>and the process ended up taking a while.
>
>I didn't manage to get coolbits working for our old card unfortunately,
>so I can't tell you what downclocking would have done. They approved the
>replacement, and we requested a standard 780 but they just did a direct
>exchange with the same model.
>
>The downside is the new GTX780Ti is still having errors, on the plus side
>I'm getting more informative error reports.
>
>My systems (which run fine on our TeslaC2050s) are still crashing almost
>immediately. However the tutorial 1B explicit water simulation ran for
>10ns without any errors reported at all. This didn't work before so..
>progress, but I'm not sure if that's down to hardware or updated Amber.
>
>The error which is being reported to stdout for my systems is "cudaMemcpy
>GpuBuffer::Download failed an illegal memory access was encountered" or
>"cudaMemcpy GpuBuffer::Download failed unspecified launch failure"
>
>So I submitted jobs with cuda-memcheck and get "Invalid __global__ ...
>Address xxxx is out of bounds" type errors, so memory access errors of
>the form listed in section 3.4 here:
>http://docs.nvidia.com/cuda/cuda-memcheck/#axzz32zciX8kj
>
>My computer science is rusty, but does this indicate a software not
>hardware issue? Have I got something wrong in my set-up? I've pasted
>excerpts of a cuda-memcheck error at the bottom of this message, as well
>as attaching the nohup.out
>
>I've updated and am now running AmberTools v14.02, Amber v14.00, Nvidia
>driver v337.12 (beta), CUDA v6.0.1. pmemd.cuda was compiled with
>./configure -cuda gnu. OS is CentOS 6.5
>
>Thank you heaps!
>
>Keiran
>
>________________________________________
>From: filip fratev [filipfratev.yahoo.com]
>Sent: 18 April 2014 02:14
>To: AMBER Mailing List
>Subject: Re: [AMBER] GTX780Ti consistent error "cudaMemcpy
>GpuBuffer::Download failed unspecified launch failure."
>
>Hi Keiran and Ross,
>>I've seen lots of issues with 780Ti since they appear to be right on
>>the edge of stability with regards to clock speed.
>
>In fact this was exactly my case, but my was the SC version. You might
>also prove that if you use 337 driver and downclock your GPU. BTW 337 is
>the first driver with this option avalible.
>
>
>Regards,
>Filip
>
>On Wednesday, April 16, 2014 8:45 PM, Ross Walker <ross.rosswalker.co.uk>
>wrote:
>
>Hi Keiran,
>
>If driver 337.12 does not fix things then it is almost certainly a bad
>GPU. I've seen lots of issues with 780Ti since they appear to be right on
>the edge of stability with regards to clock speed. I recommend returning
>it for exchange.
>
>All the best
>Ross
>
>
>On 4/15/14, 11:14 PM, "Keiran Rowell" <k.rowell.unsw.edu.au> wrote:
>
>>Dear Amber users,
>>
>>My research group recently bought a GTX780Ti (model number:GV-N78TOC-3GD)
>>for running pmemd.cuda, and it's been consistently printing out "NaN" in
>>the TEMP field of an mdcrd.out and reporting "cudaMemcpy
>>GpuBuffer::Download failed unspecified launch failure." It will run for a
>>little bit, often a couple thousand frames, which look normal when
>>visualised in VMD, before suddenly cutting out.
>>
>>I've managed to find a few threads on this issue, however they seemed
>>mostly to be caused by out-of-date Amber builds, and the only one I saw
>>about a GTX780Ti (http://archive.ambermd.org/201401/0378.html) seemed to
>>be just due to a bad card.
>>
>>I've made sure my Amber build is up-to-date. "update_amber --update" says
>>no new updates and "update_amber -v" gives AmberTools version 13.24,
>>Amber version 12.21
>>
>>This exact same system (and a bunch of analogous ones) has run without
>>problems on the pair of Tesla C2050's we have. I also tried running the
>>longer ambermd.org tutorial B1 runs with pmemd.cuda and I get the same
>>types of crashes.
>>
>>To my unexperienced eyes the "cudaMemcpy buffer" error seemed to be a
>>memory error with the card, so I ran the jobs with cuda-memcheck. However
>>every time I do this the job runs fine (albiet very slowly) and with 0
>>errors in the summary. When it moves onto the next job without memcheck I
>>then get a crash.
>>
>>Suspecting overheating I monitored the card's temprature with nvidia-smi
>>while on full load, and it gets up to 83C with 70% fan which I don't
>>think is out of tolerance.
>>
>>When I run tests with make test.cuda I get 89 passes and 8 fails, and on
>>inspection of the .diff file they are just errors in the last decimal
>>place. Similarly the benchmark suite passes fine and gives ns/day in line
>>with gpu benchmarks on ambermd.org.
>>
>>For completeness sake I've also recompiled Amber with -cuda_SPDP,
>>-cuda_DPD and the intel compilers and I get the same story.
>>
>>I'm running CentOS 6.5, and CUDA 6.0/5.5, with nvidia driver version
>>v337.12. I installed these by hand as sadly I can't yum install directly
>>from the cuda and elrepo repositories, as the cuda repo provides v319.37
>>which the Amber website says is incompatible. I previously tried v331.49
>>but now that ambermd.org/gpu says GTX780Ti is supported I updated the
>>drivers, and still get the issue.
>>
>>My supervisor also mentioned that the motherboard (Gigabyte P35-DS3P)
>>seems to only support PCI-E 1.x. Could this be the cause of the issue?
>>
>>I'm now a bit of a loss as to what the issue is. Is there something I am
>>missing, or is this just a bad card?
>>
>>Gratefully,
>>
>>Keiran Rowell
>>
>>_______________________________________________
>>AMBER mailing list
>>AMBER.ambermd.org
>>http://lists.ambermd.org/mailman/listinfo/amber
>
>
>
>_______________________________________________
>AMBER mailing list
>AMBER.ambermd.org
>http://lists.ambermd.org/mailman/listinfo/amber
>_______________________________________________
>AMBER mailing list
>AMBER.ambermd.org
>http://lists.ambermd.org/mailman/listinfo/amber
>_______________________________________________
>AMBER mailing list
>AMBER.ambermd.org
>http://lists.ambermd.org/mailman/listinfo/amber
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Fri May 30 2014 - 09:00:03 PDT
