Ps try running for 100k steps before comparing energies and I suspect no
two simulations will match.
On May 29, 2013 1:41 PM, "Scott Le Grand" <varelse2005.gmail.com> wrote:
> Your Titan setup is hosed. Your results were not 100% deterministic for
> the same inputs.
>
> Energies + Forces use a different subroutine than just Forces hence the
> ntpr dependence. Hence changing ntpr effectively is changing the input.
>
> It's 100% ironclad reproducibility that matters and you demonstrated it's
> not happening.
> On May 29, 2013 1:30 PM, "Marek Maly" <marek.maly.ujep.cz> wrote:
>
>> Hi all,
>>
>> First of all thanks to Ross for his update ! although it is question
>> if it helps to solve all the reported Amber issues with Titan/OC Titan
>> GPUs .
>> So let's see and hope :))
>>
>> Here are my results - see the attached TXT file with tables where
>> the results from the tests are summarised. I did twice the same
>> Amber benchmark tests on each GPU (both titans, GTX 680 and GTX 580)
>> to see reproducibility of the results after 100K steps at ig=default
>> (so ig not present in mdin file).
>>
>> The first table contains ns/day estimates obtained for each molecular
>> system
>> for each TITAN GPU. Interestingly estimates obtained for the same system
>> in different
>> round slightly differ, but maybe that's OK.
>>
>> In the second table there are values of the total energy after 100k steps
>> to check
>> reproducibility of the results.
>>
>> Here is summarisation :
>>
>> #1 - simulation crashes on TITANs
>>
>> Interestingly there was just one simulation crash in JAC_NPT (TITAN_0,
>> ROUND_1) the remaining
>> 3 TITAN JAC_NPT simulations were finished. There were also 3 times
>> crashes in CELLULOSE_NVE
>> test but the last simulation (TITAN_1,ROUND_2) was finished without any
>> problem. All the remaining
>> simulations were always finished without any problem. So the simulation
>> crashes seem to be
>> not-reproducible/unpredictible on some moleacular systems/(mdin setups).
>>
>> CRASH ERRORS:
>>
>> a) JAC_NPT (TITAN_0, ROUND_1)
>> Here 11k steps were successfully done before crash, I found this error
>> in mdout file:
>>
>> | ERROR: max pairlist cutoff must be less than unit cell max sphere
>> radius!
>>
>> b) CELLULOSE_NVE (TITAN_0, ROUND_1, ROUND_2; TITAN_1, ROUND_1 )
>> Here I did not find any error in mdout file. Just this error was written
>> on standard output
>> (screen/nohup.out file):
>>
>> ------
>> Error: unspecified launch failure launching kernel kNLSkinTest
>> cudaFree GpuBuffer::Deallocate failed unspecified launch failure
>> grep: mdinfo.1GTX_TITAN: No such file or directory
>> -----
>>
>> in all three cases.
>>
>> Here on CELLULOSE_NVE case I started to play with NTPR parameter
>> (originally just
>> on TITAN-0 GPU), to see how many steps were successfully done here before
>> crash, then this my
>> small research started to be more interesting than I ever thought :)) see
>> here
>> chronologically my results for E_tot after 2000 steps for different GPUs
>> (machines) - I repeated calculation several times for the given NTPR just
>> to be sure.
>>
>> TITAN-0, Etot after 2000 steps
>>
>> NTPR=10
>>
>> -443256.6867
>> -443256.6867
>> -443256.6867
>>
>> NTPR=100
>>
>> -443250.1350
>> -443250.1350
>> -443250.1350
>>
>> NTPR=200
>>
>> -443261.0705
>> -443261.0705
>> -443072.3097
>> -443261.0705
>> -443261.0705
>> -443261.0705
>> -443261.0705
>>
>> NTPR=10 (again just to verify)
>>
>> -443256.6867
>> -443256.6867
>>
>>
>> Then I tried with TITAN-1
>>
>> NTPR=10
>>
>> -443256.6867
>> -443256.6867
>>
>> NTPR=100
>>
>> -443250.1350
>> -443250.1350
>>
>> NTPR=200
>>
>> -443261.0705
>> -443261.0705
>>
>>
>> Then I tried with GTX-580
>>
>> NTPR=10
>>
>> -443256.6867
>> -443256.6867
>>
>> NTPR=200
>>
>> -443261.0705
>> -443261.0705
>>
>> then I tried with GTX-680
>>
>> NTPR=10 Etot after 2000 steps
>>
>> -443256.6711
>> -443256.6711
>>
>> NTPR=200 Etot after 2000 steps
>>
>> -443261.0705
>> -443261.0705
>>
>> Any idea why energies should depend on frequency of energy records (NTPR)
>> ?
>>
>>
>>
>> #2 - reproducibility on TITANs (see attached table.txt)
>>
>> Also here are differences depending on concrete systems/setups.
>> While in case of FACTOR_IX_NVE, FACTOR_IX_NPT, TRPCAGE, MYOGLOBIN systems
>> I have obtained
>> 100% reproducibility (the results for the given system were identical for
>> both cards/all ROUNDs)
>> on systems JAC_NVE, JAC_NPT, NUCLEOSOME I obtained small differences in
>> general however in case
>> of TITAN_1 GPU also NUCLEOSOME results were 100% reproducible. Moreover
>> for the TITAN_1 card which succeeded to finish CELLULOSE test at least in
>> ROUND_2 I did 3rd additional round and I got the identical result as from
>> the ROUND_2 (i.e. -443246.3206 ) so regarding TITAN_1 GPU I can say that
>> it is able to 100% reproduce 100k steps CELLULOSE_NVE test result perhaps
>> on all eventually successfully finished runs :))
>>
>>
>> #3 - GTX-580, GTX-680 controls
>>
>> Here the simulations were done without any problems and were 100%
>> reproducible on each card however
>> the results for the given system slightly differ between those two cards
>> with exception of the
>> CELLULOSE system where both cards GTX-580, GTX-680 provided identical
>> result which is moreover
>> nearly identical with result obtained with TITAN_1 during ROUND_2
>> (relative difference 2e-6).
>>
>>
>> TO ET:
>> a)
>> I had no problems with minimization stages in my own simul. bigger than
>> 100k which crashed
>> during heat NVT phase.
>>
>> b)
>> 313.30 driver ??? OK so after 319.23 I will try experiment with this a
>> bit "outdated" version :))
>> Actually I am working under 319.17. (and CUDA 5.0)
>>
>> c)
>> Can you please do at least JAC_NPT, JAC_NVE, NUCLEOSOME and CELLULOSE_NVE
>> tests using 100 000 steps
>> (same random seed e.g. default = ig deleted from mdin if is there) twice
>> to confirm 100% reproducibility on your TITAN GPU ?
>>
>> TO Divi:
>>
>> This is also my usual approach to divide whole simulation into many
>> subtrajectories (in my case 0.5 ns = 250k 2fs steps) but it does not seem
>> to help here it self. Can you please also do the same tests which I asked
>> ET (point c) )
>>
>>
>> BTW CUDA release candidate 5.5 was just released (
>> https://developer.nvidia.com/**cuda-toolkit<https://developer.nvidia.com/cuda-toolkit>)
>> would it be reasonable idea to try compile/run pmemd.cuda with this brand
>> new cuda version ?
>>
>> Thanks !
>>
>> Best wishes,
>>
>> Marek
>>
>>
>>
>>
>>
>>
>> Dne Wed, 29 May 2013 03:44:33 +0200 Ross Walker <ross.rosswalker.co.uk>
>> napsal/-a:
>>
>> Hi All,
>>>
>>> Just an update that we will have some fixes out soon that address some
>>> errors we have been noticing with simulations crashing during NPT runs.
>>> It
>>> is possible that this is confusing the issue here as to whether the
>>> problem is related to the GTX Titan or to a possible bug in the code. I
>>> hope to have the patch released within a few days at which point it would
>>> be good to repeat these tests and then hopefully we can try to track down
>>> what is going on. I find it hard to believe that so many cards are faulty
>>> so I suspect that there may be something funky in the code with regards
>>> to
>>> GTX Titans. We'll try and get it fixed as soon as possible but for now
>>> please just wait until we get the update released for AMBER 12 in a few
>>> days and see if that helps at all.
>>>
>>> All the best
>>> Ross
>>>
>>>
>>> On 5/28/13 5:12 PM, "Divi/GMAIL" <dvenkatlu.gmail.com> wrote:
>>>
>>> I have two TITANs in my Gigabyte workstation. I have had similar
>>>> issues
>>>> of NANs for some of the simulation setups. Never could figure out why
>>>> the
>>>> simulations failed for no reason. I tried 10, 12 ang. box sizes. same
>>>> random breakdowns. Thought of returning them suspecting memory errors.
>>>> But
>>>> some simulations ran perfectly fine. Currently running two calculations
>>>> without any problems. Both are running pretty stable for over 100ns. I
>>>> suspect AMBER CUDA code may have some issues under some simulation
>>>> conditions such as NPT. In general, NVT setup is more successful than
>>>> NPT,
>>>> in my case.
>>>>
>>>> These are 287426 atoms simulation on one card (9 ns/day)
>>>> On other card: 129049 atom setup (20 ns/day)
>>>>
>>>> Both using same NVT setup. (AMBER12/INTEL-12.x
>>>> compilers/CentOS-6.3/Drivers 319.17/CUDA5.0)
>>>>
>>>> Input is below:
>>>> &cntrl
>>>> nstlim=500000, dt=0.002,
>>>> ntx=5, irest=1, ig=-1,
>>>> ntpr=1000, ntwr=10000, ntwx=10000,
>>>> ntt=1, tautp=2, ntb=1, ntp=0, ntc=2, ntf=2,
>>>> iwrap=1, ioutfm=1, ntxo=2,
>>>> &end
>>>>
>>>> One suggestion If I may add: If you could run short simulations for no
>>>> more
>>>> than 500,000 steps (or 1ns with 2 fs), you might find some stability.
>>>> Again,
>>>> not scientific rationale from my side. But it worked in some cases for
>>>> me.
>>>>
>>>> This is self-assembled system with GIGABYTE GA-Z77X-UP7 (with core i5
>>>> processor) and 1200W PS/16GB memory.
>>>>
>>>>
>>>> Best regards
>>>> Divi
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Scott Le Grand
>>>> Sent: Tuesday, May 28, 2013 4:46 PM
>>>> To: AMBER Mailing List
>>>> Subject: Re: [AMBER] experiences with EVGA GTX TITAN Superclocked -
>>>> memtestG80 - UNDERclocking in Linux ?
>>>>
>>>> You can play Russian Roulette a whole bunch of rounds without blowing
>>>> your
>>>> head off.
>>>>
>>>> Similarly, when you have a GPU that occasionally flips a bit the wrong
>>>> way,
>>>> most of the time it will be some low order perturbation to the
>>>> coordinates
>>>> that does little more than make the trajectory nondeterministic...
>>>> Except
>>>> when it doesn't...
>>>>
>>>> You can't even detect this kind of misbehavior in GROMACS, ACEMD, or
>>>> NAMD
>>>> because *none* of them (to my knowledge) are capable of producing
>>>> deterministic output at production-level performance.
>>>>
>>>> Titans and 680s are consumer cards. I love them to death, but if you're
>>>> going to do production work with them, you need to qual them thoroughly
>>>> before proceeding or you need to pay up and use Teslas instead. I'd
>>>> still
>>>> build a cluster with Titans myself, but I'd ruthlessly RMA them until I
>>>> got
>>>> satisfaction if they couldn't pass a test consisting of running an AMBER
>>>> simulation for 100K iterations without either crashing or producing a
>>>> nondeterministic result. The customer is always right.
>>>>
>>>>
>>>> On Tue, May 28, 2013 at 1:20 PM, Marek Maly <marek.maly.ujep.cz> wrote:
>>>>
>>>> I would wait for the results of my GOPU0, GPU1 double tests before
>>>>> any serious conclusions.
>>>>>
>>>>> BTW what exactly means "GPU is hosed" ? Something like GPU is damaged
>>>>> or
>>>>> so ?
>>>>>
>>>>> Also would be strange (not probable) to buy 2 somehow damaged GPUs
>>>>> (even
>>>>> in the same way).
>>>>>
>>>>> As I wrote, memtestG80 tests were negative on both cards, if moreover
>>>>> both cards will perfectly reproduce both repetitions of the Amber
>>>>> benchmarks
>>>>> and eventually pass some another GPU tests (can you recommend any
>>>>> except
>>>>> memtestG80 ?)
>>>>> I still believe that the GPU cards are OK (also thank to particular
>>>>> successes in my Amb. simulations and actual A. benchmarks). So maybe I
>>>>> will eventually try downclock, but there might be some another
>>>>> variables,
>>>>> e.g. driver, OS, motherboard (I will perhaps test one card in another
>>>>> MB
>>>>> just to be sure, that problem is not MB based) etc. that's why I asked
>>>>> before that guy "ET" for the info about driver version, would be also
>>>>> interesting OS info or MB.
>>>>>
>>>>> M.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Dne Tue, 28 May 2013 22:13:36 +0200 Scott Le Grand
>>>>> <varelse2005.gmail.com>
>>>>> napsal/-a:
>>>>>
>>>>> > Marek,
>>>>> > Your GPU is hosed. I don't have anything else to add. I'm not going
>>>>> to
>>>>> > go
>>>>> > snark hunting for a bug that doesn't exist.
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Tue, May 28, 2013 at 12:24 PM, Marek Maly <marek.maly.ujep.cz>
>>>>> wrote:
>>>>> >
>>>>> >> Hi, just for the curiosity which driver are you using
>>>>> >> on that machine with perfectly working with OC TITAN,
>>>>> >> 319.17 or some more actual e.g. 319.23 ?
>>>>> >>
>>>>> >> RMA is a good idea but it could be also long time story and
>>>>> >> also to succeed here you need to have strong arguments
>>>>> >> especially if you are going to RMA two OC TITANs.
>>>>> >>
>>>>> >> I am not sure if my arguments "The cards have problems with some
>>>>> Amber
>>>>> >> calculations"
>>>>> >> would be strong enough here. Would be much better to have clear
>>>>> results
>>>>> >> from
>>>>> >> respected GPU tests and as it seems you may do extensive GPU tests
>>>>> also
>>>>> >> with
>>>>> >> multiple routines without any errors but still have problems with
>>>>> >> particular
>>>>> >> Amber simulations...
>>>>> >>
>>>>> >> BTW I am now doing Amber benchmarks with nstlim=100K and ig=default
>>>>> for
>>>>> >> each card
>>>>> >> twice. The tests will be done in cca 3 hours (due to slow nucleosome
>>>>> GB
>>>>> >> test).
>>>>> >>
>>>>> >> But even now I have interesting results from the first test on GPU0
>>>>> >> (nucleosome is still running) see below.
>>>>> >>
>>>>> >> As you can see JAC_NPT crashed around 11000 step, here is the last
>>>>> >> md.out
>>>>> >> record:
>>>>> >>
>>>>> >> *********
>>>>> >>
>>>>> >>
>>>>>
>>>>> ------------------------------**------------------------------**
>>>>> -------------
>>>>> -----
>>>>> >>
>>>>> >> check COM velocity, temp: 0.000021 0.00(Removed)
>>>>> >>
>>>>> >> NSTEP = 11000 TIME(PS) = 28.000 TEMP(K) = 300.39
>>>>> PRESS
>>>>> >> =
>>>>> >> -9.4
>>>>> >> Etot = -58092.8958 EKtot = 14440.2520 EPtot =
>>>>> >> -72533.1478
>>>>> >> BOND = 443.3912 ANGLE = 1253.5177 DIHED =
>>>>> >> 970.1275
>>>>> >> 1-4 NB = 567.2497 1-4 EEL = 6586.9007 VDWAALS =
>>>>> >> 8664.9960
>>>>> >> EELEC = -91019.3306 EHBOND = 0.0000 RESTRAINT =
>>>>> >> 0.0000
>>>>> >> EKCMT = 6274.0354 VIRIAL = 6321.9969 VOLUME =
>>>>> >> 236141.9494
>>>>> >> Density =
>>>>> >> 1.0162
>>>>> >>
>>>>> >>
>>>>>
>>>>> ------------------------------**------------------------------**
>>>>> -------------
>>>>> -----
>>>>> >>
>>>>> >> | ERROR: max pairlist cutoff must be less than unit cell max
>>>>> sphere
>>>>> >> radius!
>>>>> >>
>>>>> >> ********
>>>>> >>
>>>>> >> Any idea about that ERROR ?
>>>>> >>
>>>>> >> On the other hand FACTOR_IX_NPT which has much more atoms passed
>>>>> >> without
>>>>> >> any issue.
>>>>> >>
>>>>> >> Cellulose crashed on the beginning without any ERROR message in
>>>>> md.out
>>>>> >> file.
>>>>> >>
>>>>> >>
>>>>> >> I am very curious regarding exact reproducibility of the results at
>>>>> >> least
>>>>> >> in the
>>>>> >> framework of both tests on individual cards.
>>>>> >>
>>>>> >> BTW regarding eventual downclocking, has anyone idea about some
>>>>> NVclock
>>>>> >> alternative or
>>>>> >> I will be really eventually forced to edit frequency value in GPU
>>>>> BIOS
>>>>> >> ?
>>>>> >>
>>>>> >> Best,
>>>>> >>
>>>>> >> Marek
>>>>> >>
>>>>> >> HERE ARE THE FIRST DATA FROM MY 2x2 Bench tests
>>>>> >>
>>>>> >> JAC_PRODUCTION_NVE - 23,558 atoms PME
>>>>> >> ------------------------------**-------
>>>>> >>
>>>>> >> 1 x GTX_TITAN: | ns/day = 115.91 seconds/ns =
>>>>> >> 745.39
>>>>> >>
>>>>> >> JAC_PRODUCTION_NPT - 23,558 atoms PME
>>>>> >> ------------------------------**-------
>>>>> >>
>>>>> >> 1 x GTX_TITAN: STOP PMEMD Terminated Abnormally!
>>>>> >> | ns/day = 90.72 seconds/ns = 952.42
>>>>> >>
>>>>> >> FACTOR_IX_PRODUCTION_NVE - 90,906 atoms PME
>>>>> >> ------------------------------**-------------
>>>>> >>
>>>>> >> 1 x GTX_TITAN: | ns/day = 30.56
>>>>> seconds/ns =
>>>>> >> 2827.33
>>>>> >>
>>>>> >> FACTOR_IX_PRODUCTION_NPT - 90,906 atoms PME
>>>>> >> ------------------------------**-------------
>>>>> >>
>>>>> >> 1 x GTX_TITAN: | ns/day = 25.01
>>>>> seconds/ns =
>>>>> >> 3454.56
>>>>> >>
>>>>> >> CELLULOSE_PRODUCTION_NVE - 408,609 atoms PME
>>>>> >> ------------------------------**--------------
>>>>> >>
>>>>> >> 1 x GTX_TITAN: Error: unspecified launch failure launching
>>>>> >> kernel
>>>>> >> kNLSkinTest
>>>>> >> cudaFree GpuBuffer::Deallocate failed unspecified launch failure
>>>>> >> grep: mdinfo.1GTX_TITAN: No such file or directory
>>>>> >>
>>>>> >> TRPCAGE_PRODUCTION - 304 atoms GB
>>>>> >> ------------------------------**---
>>>>> >> 1 x GTX_TITAN: | ns/day = 595.09 seconds/ns =
>>>>> >> 145.19
>>>>> >>
>>>>> >> MYOGLOBIN_PRODUCTION - 2,492 atoms GB
>>>>> >> ------------------------------**-------
>>>>> >>
>>>>> >> 1 x GTX_TITAN: | ns/day = 202.56 seconds/ns =
>>>>> >> 426.53
>>>>> >>
>>>>> >> NUCLEOSOME_PRODUCTION - 25,095 atoms GB
>>>>> >> ------------------------------**---------
>>>>> >>
>>>>> >> 1 x GTX_TITAN:
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> Dne Tue, 28 May 2013 20:42:32 +0200 ET <sketchfoot.gmail.com>
>>>>> napsal/-a:
>>>>> >>
>>>>> >> > Hi,
>>>>> >> >
>>>>> >> > I just got a superclocked Titan and one at normal freq. The first
>>>>> one
>>>>> >> ran
>>>>> >> > like a charm with no issues so far. The other standard clocked one
>>>>> >> could
>>>>> >> > never get past the constant pressure stage in an NPT simulation.
>>>>> It
>>>>> >> kept
>>>>> >> > writing NAN or ********* in the outfile. I swapped them about in
>>>>> the
>>>>> >> pcie
>>>>> >> > lanes then ran it solo in each one of the lanes. Despite all this
>>>>> it
>>>>> >> was
>>>>> >> > still failing the benchmark that the other one had no problems
>>>>> with.
>>>>> >> >
>>>>> >> > I couldn't find any memory errors with GPU-burn either, but as
>>>>> they
>>>>> >> cost
>>>>> >> > near a grand a piece, I RMA'd it today. I recommend you to do the
>>>>> >> same if
>>>>> >> > its not giving you any joy. Life's too short. :)
>>>>> >> >
>>>>> >> > br,
>>>>> >> > g
>>>>> >> >
>>>>> >> >
>>>>> >> > On 28 May 2013 16:57, Scott Le Grand <varelse2005.gmail.com>
>>>>> wrote:
>>>>> >> >
>>>>> >> >> AMBER != NAMD...
>>>>> >> >>
>>>>> >> >> GTX 680 != GTX Titan...
>>>>> >> >>
>>>>> >> >> Ian's suggestion is a good one. But even then, you need to test
>>>>> >> >> your
>>>>> >> >> GPUs
>>>>> >> >> as the Titans are running right on the edge of stability. Like I
>>>>> >> told
>>>>> >> >> Marek, try running 100K iterations of Cellulose NVE twice with
>>>>> the
>>>>> >> same
>>>>> >> >> random seed. if you don't get identically bit accurate output,
>>>>> your
>>>>> >> >> GPU is
>>>>> >> >> not working. Memtest programs do not catch this because (I am
>>>>> >> guessing)
>>>>> >> >> they are designed for a uniform memory hierarchy and only one
>>>>> path
>>>>> >> >> to
>>>>> >> >> read
>>>>> >> >> and write data. I have a stock GTX Titan that cannot pass the
>>>>> >> Cellulose
>>>>> >> >> NVE test and another one that does. I spent a couple days on the
>>>>> >> former
>>>>> >> >> GPU looking for the imaginary bug that went away like magic the
>>>>> >> second I
>>>>> >> >> switched out the GPU.
>>>>> >> >>
>>>>> >> >> Scott
>>>>> >> >>
>>>>> >> >>
>>>>> >> >>
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> On Tue, May 28, 2013 at 8:11 AM, Robert Konecny <rok.ucsd.edu>
>>>>> wrote:
>>>>> >> >>
>>>>> >> >> > Hi Scott,
>>>>> >> >> >
>>>>> >> >> > unfortunately we are seeing similar Amber instability on GTX
>>>>> >> Titans as
>>>>> >> >> > Marek is. We have a box with four GTX Titans (not oveclocked)
>>>>> >> running
>>>>> >> >> > CentOS 6.3 with NVidia 319.17 driver and Amber 12.2. Any Amber
>>>>> >> >> simulation
>>>>> >> >> > longer than 10-15 min eventually crashes on these cards,
>>>>> including
>>>>> >> >> both
>>>>> >> >> JAC
>>>>> >> >> > benchmarks (with extended run time). This is reproducible on
>>>>> all
>>>>> >> four
>>>>> >> >> > cards.
>>>>> >> >> >
>>>>> >> >> > To eliminate the possible hardware error we ran extended GPU
>>>>> >> >> > memory
>>>>> >> >> tests
>>>>> >> >> > on all four Titans with memtestG80, cuda_memtest and also
>>>>> gpu_burn
>>>>> >> -
>>>>> >> >> all
>>>>> >> >> > finished without errors. Since I agree that these programs may
>>>>> not
>>>>> >> >> test
>>>>> >> >> the
>>>>> >> >> > GPU completely we also set up simulations with NAMD. We can run
>>>>> >> four
>>>>> >> >> NAMD
>>>>> >> >> > simulations simultaneously for many days without any errors on
>>>>> >> >> > this
>>>>> >> >> > hardware. For reference - we also have exactly the same server
>>>>> >> >> > with
>>>>> >> >> the
>>>>> >> >> > same hardware components but with four GTX680s and this setup
>>>>> >> >> > works
>>>>> >> >> just
>>>>> >> >> > fine for Amber. So all this leads me to believe that a hardware
>>>>> >> error
>>>>> >> >> is
>>>>> >> >> > not very likely.
>>>>> >> >> >
>>>>> >> >> > I would appreciate your comments on this, perhaps there is
>>>>> >> something
>>>>> >> >> else
>>>>> >> >> > causing these errors which we are not seeing.
>>>>> >> >> >
>>>>> >> >> > Thanks,
>>>>> >> >> >
>>>>> >> >> > Robert
>>>>> >> >> >
>>>>> >> >> >
>>>>> >> >> > On Mon, May 27, 2013 at 04:25:24PM -0700, Scott Le Grand wrote:
>>>>> >> >> > > I have two GTX Titans. One is defective, the other is not.
>>>>> >> >> > Unfortunately,
>>>>> >> >> > > they both pass all standard GPU memory tests.
>>>>> >> >> > >
>>>>> >> >> > > What the defective one doesn't do is generate reproducibly
>>>>> >> >> bit-accurate
>>>>> >> >> > > outputs for simulations of Factor IX (90,986 atoms) or
>>>>> larger,
>>>>> >> >> > > of
>>>>> >> >> 100K
>>>>> >> >> or
>>>>> >> >> > > so iterations.
>>>>> >> >> > >
>>>>> >> >> > > Which is yet another reason why I insist on MD algorithms
>>>>> >> >> (especially
>>>>> >> >> on
>>>>> >> >> > > GPUS) being deterministic. Besides its ability to find
>>>>> software
>>>>> >> >> bugs,
>>>>> >> >> > and
>>>>> >> >> > > fulfilling one of the most important tenets of science, it's
>>>>> a
>>>>> >> great
>>>>> >> >> way
>>>>> >> >> > to
>>>>> >> >> > > diagnose defective hardware with very little effort.
>>>>> >> >> > >
>>>>> >> >> > > 928 MHz? That's 6% above the boost clock of a stock Titan.
>>>>> >> Titan
>>>>> >> >> is
>>>>> >> >> > > pushing the performance envelope as is. If you're going to
>>>>> pay
>>>>> >> the
>>>>> >> >> > premium
>>>>> >> >> > > for such chips, I'd send them back until you get one that
>>>>> runs
>>>>> >> >> correctly.
>>>>> >> >> > > I'm very curious how fast you can push one of these things
>>>>> >> >> > > before
>>>>> >> >> they
>>>>> >> >> > give
>>>>> >> >> > > out.
>>>>> >> >> > >
>>>>> >> >> > >
>>>>> >> >> > >
>>>>> >> >> > >
>>>>> >> >> > >
>>>>> >> >> > >
>>>>> >> >> > >
>>>>> >> >> > > On Mon, May 27, 2013 at 10:01 AM, Marek Maly
>>>>> <marek.maly.ujep.cz
>>>>> >
>>>>> >> >> wrote:
>>>>> >> >> > >
>>>>> >> >> > > > Dear all,
>>>>> >> >> > > >
>>>>> >> >> > > > I have recently bought two "EVGA GTX TITAN Superclocked"
>>>>> GPUs.
>>>>> >> >> > > >
>>>>> >> >> > > > I did the first calculations (pmemd.cuda in Amber12) with
>>>>> >> systems
>>>>> >> >> > around
>>>>> >> >> > > > 60K atoms without any problems (NPT, Langevin), but when I
>>>>> >> later
>>>>> >> >> tried
>>>>> >> >> > > > with bigger systems (around 100K atoms) I obtained
>>>>> "classical"
>>>>> >> >> > irritating
>>>>> >> >> > > > errors
>>>>> >> >> > > >
>>>>> >> >> > > > cudaMemcpy GpuBuffer::Download failed unspecified launch
>>>>> >> failure
>>>>> >> >> > > >
>>>>> >> >> > > > just after few thousands of MD steps.
>>>>> >> >> > > >
>>>>> >> >> > > > So this was obviously the reason for memtestG80 tests.
>>>>> >> >> > > > ( https://simtk.org/home/memtest ).
>>>>> >> >> > > >
>>>>> >> >> > > > So I compiled memtestG80 from sources (
>>>>> >> memtestG80-1.1-src.tar.gz
>>>>> >> >> )
>>>>> >> >> and
>>>>> >> >> > > > then tested
>>>>> >> >> > > > just small part of memory GPU (200 MB) using 100
>>>>> iterations.
>>>>> >> >> > > >
>>>>> >> >> > > > On both cards I have obtained huge amount of errors but
>>>>> "just"
>>>>> >> on
>>>>> >> >> > > > "Random blocks:". 0 errors in all remaining tests in all
>>>>> >> >> iterations.
>>>>> >> >> > > >
>>>>> >> >> > > > ------THE LAST ITERATION AND FINAL RESULTS-------
>>>>> >> >> > > >
>>>>> >> >> > > > Test iteration 100 (GPU 0, 200 MiB): 169736847 errors so
>>>>> far
>>>>> >> >> > > > Moving Inversions (ones and zeros): 0 errors (6 ms)
>>>>> >> >> > > > Memtest86 Walking 8-bit: 0 errors (53 ms)
>>>>> >> >> > > > True Walking zeros (8-bit): 0 errors (26 ms)
>>>>> >> >> > > > True Walking ones (8-bit): 0 errors (26 ms)
>>>>> >> >> > > > Moving Inversions (random): 0 errors (6 ms)
>>>>> >> >> > > > Memtest86 Walking zeros (32-bit): 0 errors (105 ms)
>>>>> >> >> > > > Memtest86 Walking ones (32-bit): 0 errors (104 ms)
>>>>> >> >> > > > Random blocks: 1369863 errors (27 ms)
>>>>> >> >> > > > Memtest86 Modulo-20: 0 errors (215 ms)
>>>>> >> >> > > > Logic (one iteration): 0 errors (4 ms)
>>>>> >> >> > > > Logic (4 iterations): 0 errors (8 ms)
>>>>> >> >> > > > Logic (shared memory, one iteration): 0 errors (8
>>>>> ms)
>>>>> >> >> > > > Logic (shared-memory, 4 iterations): 0 errors (25
>>>>> ms)
>>>>> >> >> > > >
>>>>> >> >> > > > Final error count after 100 iterations over 200 MiB of GPU
>>>>> >> memory:
>>>>> >> >> > > > 171106710 errors
>>>>> >> >> > > >
>>>>> >> >> > > > ------------------------------**------------
>>>>> >> >> > > >
>>>>> >> >> > > > I have some questions and would be really grateful for any
>>>>> >> >> comments.
>>>>> >> >> > > >
>>>>> >> >> > > > Regarding overclocking, using the deviceQuery I found out
>>>>> that
>>>>> >> >> under
>>>>> >> >> > linux
>>>>> >> >> > > > both cards run
>>>>> >> >> > > > automatically using boost shader/GPU frequency which is
>>>>> here
>>>>> >> 928
>>>>> >> >> MHz
>>>>> >> >> > (the
>>>>> >> >> > > > basic value for these factory OC cards is 876 MHz).
>>>>> >> >> > > > deviceQuery
>>>>> >> >> > reported
>>>>> >> >> > > > Memory Clock rate is 3004 MHz although "it" should be 6008
>>>>> MHz
>>>>> >> but
>>>>> >> >> > maybe
>>>>> >> >> > > > the quantity which is reported by deviceQuery "Memory Clock
>>>>> >> rate"
>>>>> >> >> is
>>>>> >> >> > > > different from the product specification "Memory Clock" .
>>>>> It
>>>>> >> seems
>>>>> >> >> that
>>>>> >> >> > > > "Memory Clock rate" = "Memory Clock"/2. Am I right ? Or
>>>>> just
>>>>> >> >> > deviceQuery
>>>>> >> >> > > > is not able to read this spec. properly
>>>>> >> >> > > > in Titan GPU ?
>>>>> >> >> > > >
>>>>> >> >> > > > Anyway for the moment I assume that the problem might be
>>>>> due
>>>>> >> >> > > > to
>>>>> >> >> the
>>>>> >> >> > high
>>>>> >> >> > > > shader/GPU frequency.
>>>>> >> >> > > > (see here :
>>>>> http://folding.stanford.edu/**English/DownloadUtils<http://folding.stanford.edu/English/DownloadUtils>
>>>>> )
>>>>> >> >> > > >
>>>>> >> >> > > > To verify this hypothesis one should perhaps UNDERclock to
>>>>> >> basic
>>>>> >> >> > frequency
>>>>> >> >> > > > which is in this
>>>>> >> >> > > > model 876 MHz or even to the TITAN REFERENCE frequency
>>>>> which
>>>>> >> >> > > > is
>>>>> >> >> 837
>>>>> >> >> > MHz.
>>>>> >> >> > > >
>>>>> >> >> > > > Obviously I am working with these cards under linux (CentOS
>>>>> >> >> > > > 2.6.32-358.6.1.el6.x86_64) and as I found, the OC tools
>>>>> under
>>>>> >> >> linux
>>>>> >> >> > are in
>>>>> >> >> > > > fact limited just to NVclock utility, which is
>>>>> unfortunately
>>>>> >> >> > > > out of date (at least speaking about the GTX Titan ). I
>>>>> have
>>>>> >> >> obtained
>>>>> >> >> > this
>>>>> >> >> > > > message when I wanted
>>>>> >> >> > > > just to let NVclock utility to read and print shader and
>>>>> >> >> > > > memory
>>>>> >> >> > > > frequencies of my Titan's:
>>>>> >> >> > > >
>>>>> >> >> > > >
>>>>> >> >>
>>>>> ------------------------------**------------------------------**
>>>>> -------
>>>>> >> >> > > >
>>>>> >> >> > > > [root.dyn-138-272 NVCLOCK]# nvclock -s --speeds
>>>>> >> >> > > > Card: Unknown Nvidia card
>>>>> >> >> > > > Card number: 1
>>>>> >> >> > > > Memory clock: -2147483.750 MHz
>>>>> >> >> > > > GPU clock: -2147483.750 MHz
>>>>> >> >> > > >
>>>>> >> >> > > > Card: Unknown Nvidia card
>>>>> >> >> > > > Card number: 2
>>>>> >> >> > > > Memory clock: -2147483.750 MHz
>>>>> >> >> > > > GPU clock: -2147483.750 MHz
>>>>> >> >> > > >
>>>>> >> >> > > >
>>>>> >> >> > > >
>>>>> >> >>
>>>>> ------------------------------**------------------------------**
>>>>> -------
>>>>> >> >> > > >
>>>>> >> >> > > >
>>>>> >> >> > > > I would be really grateful for some tips regarding
>>>>> "NVclock
>>>>> >> >> > alternatives",
>>>>> >> >> > > > but after wasting some hours with googling it seems that
>>>>> there
>>>>> >> is
>>>>> >> >> no
>>>>> >> >> > other
>>>>> >> >> > > > Linux
>>>>> >> >> > > > tool with NVclock functionality. So the only possibility is
>>>>> >> here
>>>>> >> >> > perhaps
>>>>> >> >> > > > to edit
>>>>> >> >> > > > GPU bios with some Lin/DOS/Win tools like (Kepler BIOS
>>>>> >> >> > > > Tweaker,
>>>>> >> >> > NVflash)
>>>>> >> >> > > > but obviously
>>>>> >> >> > > > I would like to rather avoid such approach as using it
>>>>> means
>>>>> >> >> perhaps
>>>>> >> >> > also
>>>>> >> >> > > > to void the warranty even if I am going to underclock the
>>>>> GPUs
>>>>> >> >> not to
>>>>> >> >> > > > overclock them.
>>>>> >> >> > > > So before this eventual step (GPU bios editing) I would
>>>>> like
>>>>> >> >> > > > to
>>>>> >> >> have
>>>>> >> >> > some
>>>>> >> >> > > > approximative estimate
>>>>> >> >> > > > of the probability, that the problems are here really
>>>>> because
>>>>> >> of
>>>>> >> >> the
>>>>> >> >> > > > overclocking
>>>>> >> >> > > > (too high (boost) default shader frequency).
>>>>> >> >> > > >
>>>>> >> >> > > > This probability I hope to estimate from the eventual
>>>>> >> responses of
>>>>> >> >> > another
>>>>> >> >> > > > Amber/Titan SC users, if I am not the only crazy guy who
>>>>> >> >> > > > bought
>>>>> >> >> this
>>>>> >> >> > model
>>>>> >> >> > > > for Amber calculations :)) But of course any eventual
>>>>> >> experiences
>>>>> >> >> with
>>>>> >> >> > > > Titan cards related to their memtestG80 results and
>>>>> >> >> UNDER/OVERclocking
>>>>> >> >> > > > (if possible in Linux OS) are of course welcomed as well !
>>>>> >> >> > > >
>>>>> >> >> > > > My HW/SW configuration
>>>>> >> >> > > >
>>>>> >> >> > > > motherboard: ASUS P9X79 PRO
>>>>> >> >> > > > CPU: Intel Core i7-3930K
>>>>> >> >> > > > RAM: CRUCIAL Ballistix Sport 32GB (4x8GB) DDR3 1600 VLP
>>>>> >> >> > > > CASE: CoolerMaster Dominator CM-690 II Advanced,
>>>>> >> >> > > > Power:Enermax PLATIMAX EPM1200EWT 1200W, 80+, Platinum
>>>>> >> >> > > > GPUs : 2 x EVGA GTX TITAN Superclocked 6GB
>>>>> >> >> > > > cooler: Cooler Master Hyper 412 SLIM
>>>>> >> >> > > >
>>>>> >> >> > > > OS: CentOS (2.6.32-358.6.1.el6.x86_64)
>>>>> >> >> > > > driver version: 319.17
>>>>> >> >> > > > cudatoolkit_5.0.35_linux_64_**rhel6.x
>>>>> >> >> > > >
>>>>> >> >> > > > The computer is in air-conditioned room with permanent
>>>>> >> >> > > > external
>>>>> >> >> > > > temperature around 18°C
>>>>> >> >> > > >
>>>>> >> >> > > >
>>>>> >> >> > > > Thanks a lot in advance for any comment/experience !
>>>>> >> >> > > >
>>>>> >> >> > > > Best wishes,
>>>>> >> >> > > >
>>>>> >> >> > > > Marek
>>>>> >> >> > > >
>>>>> >> >> > > > --
>>>>> >> >> > > > Tato zpráva byla vytvořena převratným poštovním klientem
>>>>> >> >> > > > Opery:
>>>>> >> >> > > > http://www.opera.com/mail/
>>>>> >> >> > > >
>>>>> >> >> > > > ______________________________**_________________
>>>>> >> >> > > > AMBER mailing list
>>>>> >> >> > > > AMBER.ambermd.org
>>>>> >> >> > > > http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>> >> >> > > >
>>>>> >> >> > > ______________________________**_________________
>>>>> >> >> > > AMBER mailing list
>>>>> >> >> > > AMBER.ambermd.org
>>>>> >> >> > > http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>> >> >> >
>>>>> >> >> > ______________________________**_________________
>>>>> >> >> > AMBER mailing list
>>>>> >> >> > AMBER.ambermd.org
>>>>> >> >> > http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>> >> >> >
>>>>> >> >> ______________________________**_________________
>>>>> >> >> AMBER mailing list
>>>>> >> >> AMBER.ambermd.org
>>>>> >> >> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>> >> >>
>>>>> >> > ______________________________**_________________
>>>>> >> > AMBER mailing list
>>>>> >> > AMBER.ambermd.org
>>>>> >> > http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>> >> >
>>>>> >> > __________ Informace od ESET NOD32 Antivirus, verze databaze 8385
>>>>> >> > (20130528) __________
>>>>> >> >
>>>>> >> > Tuto zpravu proveril ESET NOD32 Antivirus.
>>>>> >> >
>>>>> >> > http://www.eset.cz
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
>>>>> >> http://www.opera.com/mail/
>>>>> >>
>>>>> >> ______________________________**_________________
>>>>> >> AMBER mailing list
>>>>> >> AMBER.ambermd.org
>>>>> >> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>> >>
>>>>> > ______________________________**_________________
>>>>> > AMBER mailing list
>>>>> > AMBER.ambermd.org
>>>>> > http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>> >
>>>>> > __________ Informace od ESET NOD32 Antivirus, verze databaze 8386
>>>>> > (20130528) __________
>>>>> >
>>>>> > Tuto zpravu proveril ESET NOD32 Antivirus.
>>>>> >
>>>>> > http://www.eset.cz
>>>>> >
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>>> --
>>>>> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
>>>>> http://www.opera.com/mail/
>>>>>
>>>>> ______________________________**_________________
>>>>> AMBER mailing list
>>>>> AMBER.ambermd.org
>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>
>>>>> ______________________________**_________________
>>>> AMBER mailing list
>>>> AMBER.ambermd.org
>>>> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>
>>>>
>>>> ______________________________**_________________
>>>> AMBER mailing list
>>>> AMBER.ambermd.org
>>>> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>
>>>
>>>
>>>
>>> ______________________________**_________________
>>> AMBER mailing list
>>> AMBER.ambermd.org
>>> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>
>>> __________ Informace od ESET NOD32 Antivirus, verze databaze 8386
>>> (20130528) __________
>>>
>>> Tuto zpravu proveril ESET NOD32 Antivirus.
>>>
>>> http://www.eset.cz
>>>
>>>
>>>
>>>
>>
>> --
>> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
>> http://www.opera.com/mail/
>> _______________________________________________
>> AMBER mailing list
>> AMBER.ambermd.org
>> http://lists.ambermd.org/mailman/listinfo/amber
>>
>>
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Wed May 29 2013 - 14:00:04 PDT
