OK,
BTW I just run 500K steps benchmarks (again 2 repetitions for each test)
:))
but before I installed the newest driver 319.23 (with reboot) so let see
...
What about that Cuda 5.5 ? Anyway 319.23 driver has already some
"Cuda 5.5 support/part= CUDA driver" as deviceQuery printed (after 319.23
installation)
--------
...CUDA Driver = CUDART, CUDA Driver Version = 5.5, CUDA Runtime Version =
5.0...
--------
M.
Dne Thu, 30 May 2013 01:05:24 +0200 Scott Le Grand <varelse2005.gmail.com>
napsal/-a:
> Sorry, I missed the attachment because I'm on a cellphone and on a
> business
> trip. Anyway, neither Titan showed fully deterministic behavior and that
> is worrisome. Notice that the 680 runs were indeed so. The latter is
> the
> expected behavior and exactly what I see with one of my Titans and my
> K20.
>
> Which means we need to figure this out. For now, could you take it on
> faith that changing ntpr changes the trajectory by changing the code
> executed and that it doing so is not a bug? Playing around with it is
> just
> confusing the issue right now.
>
> What would help clarify is if someone could try these tests on K20 or
> K20X. I would love for someone to demonstrate this is a coding error on
> my part because I can fix that. The evidence just isn't leading me that
> way right now.
>
> Scott
> On May 29, 2013 2:41 PM, "Marek Maly" <marek.maly.ujep.cz> wrote:
>
>> Hi Scott,
>>
>> what do you mean by "try running for 100k steps before comparing
>> energies". In all
>> tests I have actually done I did exactly 100k steps before comparing
>> energies
>> E_tot(at step 100 000). So you mean to extend tests to 200k steps now ?
>>
>> M.
>>
>>
>> Dne Wed, 29 May 2013 22:46:58 +0200 Scott Le Grand
>> <varelse2005.gmail.com>
>> napsal/-a:
>>
>> > Ps try running for 100k steps before comparing energies and I suspect
>> no
>> > two simulations will match.
>> > On May 29, 2013 1:41 PM, "Scott Le Grand" <varelse2005.gmail.com>
>> wrote:
>> >
>> >> Your Titan setup is hosed. Your results were not 100% deterministic
>> for
>> >> the same inputs.
>> >>
>> >> Energies + Forces use a different subroutine than just Forces hence
>> the
>> >> ntpr dependence. Hence changing ntpr effectively is changing the
>> input.
>> >>
>> >> It's 100% ironclad reproducibility that matters and you demonstrated
>> >> it's
>> >> not happening.
>> >> On May 29, 2013 1:30 PM, "Marek Maly" <marek.maly.ujep.cz> wrote:
>> >>
>> >>> Hi all,
>> >>>
>> >>> First of all thanks to Ross for his update ! although it is question
>> >>> if it helps to solve all the reported Amber issues with Titan/OC
>> Titan
>> >>> GPUs .
>> >>> So let's see and hope :))
>> >>>
>> >>> Here are my results - see the attached TXT file with tables where
>> >>> the results from the tests are summarised. I did twice the same
>> >>> Amber benchmark tests on each GPU (both titans, GTX 680 and GTX 580)
>> >>> to see reproducibility of the results after 100K steps at ig=default
>> >>> (so ig not present in mdin file).
>> >>>
>> >>> The first table contains ns/day estimates obtained for each
>> molecular
>> >>> system
>> >>> for each TITAN GPU. Interestingly estimates obtained for the same
>> >>> system
>> >>> in different
>> >>> round slightly differ, but maybe that's OK.
>> >>>
>> >>> In the second table there are values of the total energy after 100k
>> >>> steps
>> >>> to check
>> >>> reproducibility of the results.
>> >>>
>> >>> Here is summarisation :
>> >>>
>> >>> #1 - simulation crashes on TITANs
>> >>>
>> >>> Interestingly there was just one simulation crash in JAC_NPT
>> (TITAN_0,
>> >>> ROUND_1) the remaining
>> >>> 3 TITAN JAC_NPT simulations were finished. There were also 3 times
>> >>> crashes in CELLULOSE_NVE
>> >>> test but the last simulation (TITAN_1,ROUND_2) was finished without
>> any
>> >>> problem. All the remaining
>> >>> simulations were always finished without any problem. So the
>> simulation
>> >>> crashes seem to be
>> >>> not-reproducible/unpredictible on some moleacular systems/(mdin
>> >>> setups).
>> >>>
>> >>> CRASH ERRORS:
>> >>>
>> >>> a) JAC_NPT (TITAN_0, ROUND_1)
>> >>> Here 11k steps were successfully done before crash, I found this
>> error
>> >>> in mdout file:
>> >>>
>> >>> | ERROR: max pairlist cutoff must be less than unit cell max
>> sphere
>> >>> radius!
>> >>>
>> >>> b) CELLULOSE_NVE (TITAN_0, ROUND_1, ROUND_2; TITAN_1, ROUND_1 )
>> >>> Here I did not find any error in mdout file. Just this error was
>> >>> written
>> >>> on standard output
>> >>> (screen/nohup.out file):
>> >>>
>> >>> ------
>> >>> Error: unspecified launch failure launching kernel kNLSkinTest
>> >>> cudaFree GpuBuffer::Deallocate failed unspecified launch failure
>> >>> grep: mdinfo.1GTX_TITAN: No such file or directory
>> >>> -----
>> >>>
>> >>> in all three cases.
>> >>>
>> >>> Here on CELLULOSE_NVE case I started to play with NTPR parameter
>> >>> (originally just
>> >>> on TITAN-0 GPU), to see how many steps were successfully done here
>> >>> before
>> >>> crash, then this my
>> >>> small research started to be more interesting than I ever thought
>> :))
>> >>> see
>> >>> here
>> >>> chronologically my results for E_tot after 2000 steps for different
>> >>> GPUs
>> >>> (machines) - I repeated calculation several times for the given NTPR
>> >>> just
>> >>> to be sure.
>> >>>
>> >>> TITAN-0, Etot after 2000 steps
>> >>>
>> >>> NTPR=10
>> >>>
>> >>> -443256.6867
>> >>> -443256.6867
>> >>> -443256.6867
>> >>>
>> >>> NTPR=100
>> >>>
>> >>> -443250.1350
>> >>> -443250.1350
>> >>> -443250.1350
>> >>>
>> >>> NTPR=200
>> >>>
>> >>> -443261.0705
>> >>> -443261.0705
>> >>> -443072.3097
>> >>> -443261.0705
>> >>> -443261.0705
>> >>> -443261.0705
>> >>> -443261.0705
>> >>>
>> >>> NTPR=10 (again just to verify)
>> >>>
>> >>> -443256.6867
>> >>> -443256.6867
>> >>>
>> >>>
>> >>> Then I tried with TITAN-1
>> >>>
>> >>> NTPR=10
>> >>>
>> >>> -443256.6867
>> >>> -443256.6867
>> >>>
>> >>> NTPR=100
>> >>>
>> >>> -443250.1350
>> >>> -443250.1350
>> >>>
>> >>> NTPR=200
>> >>>
>> >>> -443261.0705
>> >>> -443261.0705
>> >>>
>> >>>
>> >>> Then I tried with GTX-580
>> >>>
>> >>> NTPR=10
>> >>>
>> >>> -443256.6867
>> >>> -443256.6867
>> >>>
>> >>> NTPR=200
>> >>>
>> >>> -443261.0705
>> >>> -443261.0705
>> >>>
>> >>> then I tried with GTX-680
>> >>>
>> >>> NTPR=10 Etot after 2000 steps
>> >>>
>> >>> -443256.6711
>> >>> -443256.6711
>> >>>
>> >>> NTPR=200 Etot after 2000 steps
>> >>>
>> >>> -443261.0705
>> >>> -443261.0705
>> >>>
>> >>> Any idea why energies should depend on frequency of energy records
>> >>> (NTPR)
>> >>> ?
>> >>>
>> >>>
>> >>>
>> >>> #2 - reproducibility on TITANs (see attached table.txt)
>> >>>
>> >>> Also here are differences depending on concrete systems/setups.
>> >>> While in case of FACTOR_IX_NVE, FACTOR_IX_NPT, TRPCAGE, MYOGLOBIN
>> >>> systems
>> >>> I have obtained
>> >>> 100% reproducibility (the results for the given system were
>> identical
>> >>> for
>> >>> both cards/all ROUNDs)
>> >>> on systems JAC_NVE, JAC_NPT, NUCLEOSOME I obtained small
>> differences
>> >>> in
>> >>> general however in case
>> >>> of TITAN_1 GPU also NUCLEOSOME results were 100% reproducible.
>> >>> Moreover
>> >>> for the TITAN_1 card which succeeded to finish CELLULOSE test at
>> least
>> >>> in
>> >>> ROUND_2 I did 3rd additional round and I got the identical result as
>> >>> from
>> >>> the ROUND_2 (i.e. -443246.3206 ) so regarding TITAN_1 GPU I can say
>> >>> that
>> >>> it is able to 100% reproduce 100k steps CELLULOSE_NVE test result
>> >>> perhaps
>> >>> on all eventually successfully finished runs :))
>> >>>
>> >>>
>> >>> #3 - GTX-580, GTX-680 controls
>> >>>
>> >>> Here the simulations were done without any problems and were 100%
>> >>> reproducible on each card however
>> >>> the results for the given system slightly differ between those two
>> >>> cards
>> >>> with exception of the
>> >>> CELLULOSE system where both cards GTX-580, GTX-680 provided
>> identical
>> >>> result which is moreover
>> >>> nearly identical with result obtained with TITAN_1 during ROUND_2
>> >>> (relative difference 2e-6).
>> >>>
>> >>>
>> >>> TO ET:
>> >>> a)
>> >>> I had no problems with minimization stages in my own simul. bigger
>> than
>> >>> 100k which crashed
>> >>> during heat NVT phase.
>> >>>
>> >>> b)
>> >>> 313.30 driver ??? OK so after 319.23 I will try experiment with
>> this a
>> >>> bit "outdated" version :))
>> >>> Actually I am working under 319.17. (and CUDA 5.0)
>> >>>
>> >>> c)
>> >>> Can you please do at least JAC_NPT, JAC_NVE, NUCLEOSOME and
>> >>> CELLULOSE_NVE
>> >>> tests using 100 000 steps
>> >>> (same random seed e.g. default = ig deleted from mdin if is there)
>> >>> twice
>> >>> to confirm 100% reproducibility on your TITAN GPU ?
>> >>>
>> >>> TO Divi:
>> >>>
>> >>> This is also my usual approach to divide whole simulation into many
>> >>> subtrajectories (in my case 0.5 ns = 250k 2fs steps) but it does not
>> >>> seem
>> >>> to help here it self. Can you please also do the same tests which I
>> >>> asked
>> >>> ET (point c) )
>> >>>
>> >>>
>> >>> BTW CUDA release candidate 5.5 was just released (
>> >>> https://developer.nvidia.com/**cuda-toolkit<
>> https://developer.nvidia.com/cuda-toolkit>)
>> >>> would it be reasonable idea to try compile/run pmemd.cuda with this
>> >>> brand
>> >>> new cuda version ?
>> >>>
>> >>> Thanks !
>> >>>
>> >>> Best wishes,
>> >>>
>> >>> Marek
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> Dne Wed, 29 May 2013 03:44:33 +0200 Ross Walker
>> <ross.rosswalker.co.uk
>> >
>> >>> napsal/-a:
>> >>>
>> >>> Hi All,
>> >>>>
>> >>>> Just an update that we will have some fixes out soon that address
>> some
>> >>>> errors we have been noticing with simulations crashing during NPT
>> >>>> runs.
>> >>>> It
>> >>>> is possible that this is confusing the issue here as to whether the
>> >>>> problem is related to the GTX Titan or to a possible bug in the
>> code.
>> >>>> I
>> >>>> hope to have the patch released within a few days at which point it
>> >>>> would
>> >>>> be good to repeat these tests and then hopefully we can try to
>> track
>> >>>> down
>> >>>> what is going on. I find it hard to believe that so many cards are
>> >>>> faulty
>> >>>> so I suspect that there may be something funky in the code with
>> >>>> regards
>> >>>> to
>> >>>> GTX Titans. We'll try and get it fixed as soon as possible but for
>> now
>> >>>> please just wait until we get the update released for AMBER 12 in a
>> >>>> few
>> >>>> days and see if that helps at all.
>> >>>>
>> >>>> All the best
>> >>>> Ross
>> >>>>
>> >>>>
>> >>>> On 5/28/13 5:12 PM, "Divi/GMAIL" <dvenkatlu.gmail.com> wrote:
>> >>>>
>> >>>> I have two TITANs in my Gigabyte workstation. I have had similar
>> >>>>> issues
>> >>>>> of NANs for some of the simulation setups. Never could figure out
>> why
>> >>>>> the
>> >>>>> simulations failed for no reason. I tried 10, 12 ang. box sizes.
>> >>>>> same
>> >>>>> random breakdowns. Thought of returning them suspecting memory
>> >>>>> errors.
>> >>>>> But
>> >>>>> some simulations ran perfectly fine. Currently running two
>> >>>>> calculations
>> >>>>> without any problems. Both are running pretty stable for over
>> 100ns.
>> >>>>> I
>> >>>>> suspect AMBER CUDA code may have some issues under some simulation
>> >>>>> conditions such as NPT. In general, NVT setup is more successful
>> than
>> >>>>> NPT,
>> >>>>> in my case.
>> >>>>>
>> >>>>> These are 287426 atoms simulation on one card (9 ns/day)
>> >>>>> On other card: 129049 atom setup (20 ns/day)
>> >>>>>
>> >>>>> Both using same NVT setup. (AMBER12/INTEL-12.x
>> >>>>> compilers/CentOS-6.3/Drivers 319.17/CUDA5.0)
>> >>>>>
>> >>>>> Input is below:
>> >>>>> &cntrl
>> >>>>> nstlim=500000, dt=0.002,
>> >>>>> ntx=5, irest=1, ig=-1,
>> >>>>> ntpr=1000, ntwr=10000, ntwx=10000,
>> >>>>> ntt=1, tautp=2, ntb=1, ntp=0, ntc=2, ntf=2,
>> >>>>> iwrap=1, ioutfm=1, ntxo=2,
>> >>>>> &end
>> >>>>>
>> >>>>> One suggestion If I may add: If you could run short simulations
>> for
>> >>>>> no
>> >>>>> more
>> >>>>> than 500,000 steps (or 1ns with 2 fs), you might find some
>> stability.
>> >>>>> Again,
>> >>>>> not scientific rationale from my side. But it worked in some cases
>> >>>>> for
>> >>>>> me.
>> >>>>>
>> >>>>> This is self-assembled system with GIGABYTE GA-Z77X-UP7 (with
>> core
>> >>>>> i5
>> >>>>> processor) and 1200W PS/16GB memory.
>> >>>>>
>> >>>>>
>> >>>>> Best regards
>> >>>>> Divi
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> -----Original Message-----
>> >>>>> From: Scott Le Grand
>> >>>>> Sent: Tuesday, May 28, 2013 4:46 PM
>> >>>>> To: AMBER Mailing List
>> >>>>> Subject: Re: [AMBER] experiences with EVGA GTX TITAN Superclocked
>> -
>> >>>>> memtestG80 - UNDERclocking in Linux ?
>> >>>>>
>> >>>>> You can play Russian Roulette a whole bunch of rounds without
>> blowing
>> >>>>> your
>> >>>>> head off.
>> >>>>>
>> >>>>> Similarly, when you have a GPU that occasionally flips a bit the
>> >>>>> wrong
>> >>>>> way,
>> >>>>> most of the time it will be some low order perturbation to the
>> >>>>> coordinates
>> >>>>> that does little more than make the trajectory nondeterministic...
>> >>>>> Except
>> >>>>> when it doesn't...
>> >>>>>
>> >>>>> You can't even detect this kind of misbehavior in GROMACS, ACEMD,
>> or
>> >>>>> NAMD
>> >>>>> because *none* of them (to my knowledge) are capable of producing
>> >>>>> deterministic output at production-level performance.
>> >>>>>
>> >>>>> Titans and 680s are consumer cards. I love them to death, but if
>> >>>>> you're
>> >>>>> going to do production work with them, you need to qual them
>> >>>>> thoroughly
>> >>>>> before proceeding or you need to pay up and use Teslas instead.
>> I'd
>> >>>>> still
>> >>>>> build a cluster with Titans myself, but I'd ruthlessly RMA them
>> >>>>> until I
>> >>>>> got
>> >>>>> satisfaction if they couldn't pass a test consisting of running an
>> >>>>> AMBER
>> >>>>> simulation for 100K iterations without either crashing or
>> producing a
>> >>>>> nondeterministic result. The customer is always right.
>> >>>>>
>> >>>>>
>> >>>>> On Tue, May 28, 2013 at 1:20 PM, Marek Maly <marek.maly.ujep.cz>
>> >>>>> wrote:
>> >>>>>
>> >>>>> I would wait for the results of my GOPU0, GPU1 double tests
>> before
>> >>>>>> any serious conclusions.
>> >>>>>>
>> >>>>>> BTW what exactly means "GPU is hosed" ? Something like GPU is
>> >>>>>> damaged
>> >>>>>> or
>> >>>>>> so ?
>> >>>>>>
>> >>>>>> Also would be strange (not probable) to buy 2 somehow damaged
>> GPUs
>> >>>>>> (even
>> >>>>>> in the same way).
>> >>>>>>
>> >>>>>> As I wrote, memtestG80 tests were negative on both cards, if
>> >>>>>> moreover
>> >>>>>> both cards will perfectly reproduce both repetitions of the Amber
>> >>>>>> benchmarks
>> >>>>>> and eventually pass some another GPU tests (can you recommend any
>> >>>>>> except
>> >>>>>> memtestG80 ?)
>> >>>>>> I still believe that the GPU cards are OK (also thank to
>> particular
>> >>>>>> successes in my Amb. simulations and actual A. benchmarks). So
>> >>>>>> maybe I
>> >>>>>> will eventually try downclock, but there might be some another
>> >>>>>> variables,
>> >>>>>> e.g. driver, OS, motherboard (I will perhaps test one card in
>> >>>>>> another
>> >>>>>> MB
>> >>>>>> just to be sure, that problem is not MB based) etc. that's why I
>> >>>>>> asked
>> >>>>>> before that guy "ET" for the info about driver version, would be
>> >>>>>> also
>> >>>>>> interesting OS info or MB.
>> >>>>>>
>> >>>>>> M.
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> Dne Tue, 28 May 2013 22:13:36 +0200 Scott Le Grand
>> >>>>>> <varelse2005.gmail.com>
>> >>>>>> napsal/-a:
>> >>>>>>
>> >>>>>> > Marek,
>> >>>>>> > Your GPU is hosed. I don't have anything else to add. I'm not
>> >>>>>> going
>> >>>>>> to
>> >>>>>> > go
>> >>>>>> > snark hunting for a bug that doesn't exist.
>> >>>>>> >
>> >>>>>> >
>> >>>>>> >
>> >>>>>> > On Tue, May 28, 2013 at 12:24 PM, Marek Maly
>> <marek.maly.ujep.cz>
>> >>>>>> wrote:
>> >>>>>> >
>> >>>>>> >> Hi, just for the curiosity which driver are you using
>> >>>>>> >> on that machine with perfectly working with OC TITAN,
>> >>>>>> >> 319.17 or some more actual e.g. 319.23 ?
>> >>>>>> >>
>> >>>>>> >> RMA is a good idea but it could be also long time story and
>> >>>>>> >> also to succeed here you need to have strong arguments
>> >>>>>> >> especially if you are going to RMA two OC TITANs.
>> >>>>>> >>
>> >>>>>> >> I am not sure if my arguments "The cards have problems with
>> some
>> >>>>>> Amber
>> >>>>>> >> calculations"
>> >>>>>> >> would be strong enough here. Would be much better to have
>> clear
>> >>>>>> results
>> >>>>>> >> from
>> >>>>>> >> respected GPU tests and as it seems you may do extensive GPU
>> >>>>>> tests
>> >>>>>> also
>> >>>>>> >> with
>> >>>>>> >> multiple routines without any errors but still have problems
>> with
>> >>>>>> >> particular
>> >>>>>> >> Amber simulations...
>> >>>>>> >>
>> >>>>>> >> BTW I am now doing Amber benchmarks with nstlim=100K and
>> >>>>>> ig=default
>> >>>>>> for
>> >>>>>> >> each card
>> >>>>>> >> twice. The tests will be done in cca 3 hours (due to slow
>> >>>>>> nucleosome
>> >>>>>> GB
>> >>>>>> >> test).
>> >>>>>> >>
>> >>>>>> >> But even now I have interesting results from the first test on
>> >>>>>> GPU0
>> >>>>>> >> (nucleosome is still running) see below.
>> >>>>>> >>
>> >>>>>> >> As you can see JAC_NPT crashed around 11000 step, here is the
>> >>>>>> last
>> >>>>>> >> md.out
>> >>>>>> >> record:
>> >>>>>> >>
>> >>>>>> >> *********
>> >>>>>> >>
>> >>>>>> >>
>> >>>>>>
>> >>>>>> ------------------------------**------------------------------**
>> >>>>>> -------------
>> >>>>>> -----
>> >>>>>> >>
>> >>>>>> >> check COM velocity, temp: 0.000021 0.00(Removed)
>> >>>>>> >>
>> >>>>>> >> NSTEP = 11000 TIME(PS) = 28.000 TEMP(K) =
>> 300.39
>> >>>>>> PRESS
>> >>>>>> >> =
>> >>>>>> >> -9.4
>> >>>>>> >> Etot = -58092.8958 EKtot = 14440.2520
>> EPtot =
>> >>>>>> >> -72533.1478
>> >>>>>> >> BOND = 443.3912 ANGLE = 1253.5177
>> DIHED =
>> >>>>>> >> 970.1275
>> >>>>>> >> 1-4 NB = 567.2497 1-4 EEL = 6586.9007
>> VDWAALS =
>> >>>>>> >> 8664.9960
>> >>>>>> >> EELEC = -91019.3306 EHBOND = 0.0000
>> RESTRAINT =
>> >>>>>> >> 0.0000
>> >>>>>> >> EKCMT = 6274.0354 VIRIAL = 6321.9969
>> VOLUME =
>> >>>>>> >> 236141.9494
>> >>>>>> >>
>> Density =
>> >>>>>> >> 1.0162
>> >>>>>> >>
>> >>>>>> >>
>> >>>>>>
>> >>>>>> ------------------------------**------------------------------**
>> >>>>>> -------------
>> >>>>>> -----
>> >>>>>> >>
>> >>>>>> >> | ERROR: max pairlist cutoff must be less than unit cell max
>> >>>>>> sphere
>> >>>>>> >> radius!
>> >>>>>> >>
>> >>>>>> >> ********
>> >>>>>> >>
>> >>>>>> >> Any idea about that ERROR ?
>> >>>>>> >>
>> >>>>>> >> On the other hand FACTOR_IX_NPT which has much more atoms
>> passed
>> >>>>>> >> without
>> >>>>>> >> any issue.
>> >>>>>> >>
>> >>>>>> >> Cellulose crashed on the beginning without any ERROR message
>> in
>> >>>>>> md.out
>> >>>>>> >> file.
>> >>>>>> >>
>> >>>>>> >>
>> >>>>>> >> I am very curious regarding exact reproducibility of the
>> results
>> >>>>>> at
>> >>>>>> >> least
>> >>>>>> >> in the
>> >>>>>> >> framework of both tests on individual cards.
>> >>>>>> >>
>> >>>>>> >> BTW regarding eventual downclocking, has anyone idea about
>> some
>> >>>>>> NVclock
>> >>>>>> >> alternative or
>> >>>>>> >> I will be really eventually forced to edit frequency value in
>> GPU
>> >>>>>> BIOS
>> >>>>>> >> ?
>> >>>>>> >>
>> >>>>>> >> Best,
>> >>>>>> >>
>> >>>>>> >> Marek
>> >>>>>> >>
>> >>>>>> >> HERE ARE THE FIRST DATA FROM MY 2x2 Bench tests
>> >>>>>> >>
>> >>>>>> >> JAC_PRODUCTION_NVE - 23,558 atoms PME
>> >>>>>> >> ------------------------------**-------
>> >>>>>> >>
>> >>>>>> >> 1 x GTX_TITAN: | ns/day = 115.91
>> >>>>>> seconds/ns =
>> >>>>>> >> 745.39
>> >>>>>> >>
>> >>>>>> >> JAC_PRODUCTION_NPT - 23,558 atoms PME
>> >>>>>> >> ------------------------------**-------
>> >>>>>> >>
>> >>>>>> >> 1 x GTX_TITAN: STOP PMEMD Terminated Abnormally!
>> >>>>>> >> | ns/day = 90.72 seconds/ns = 952.42
>> >>>>>> >>
>> >>>>>> >> FACTOR_IX_PRODUCTION_NVE - 90,906 atoms PME
>> >>>>>> >> ------------------------------**-------------
>> >>>>>> >>
>> >>>>>> >> 1 x GTX_TITAN: | ns/day = 30.56
>> >>>>>> seconds/ns =
>> >>>>>> >> 2827.33
>> >>>>>> >>
>> >>>>>> >> FACTOR_IX_PRODUCTION_NPT - 90,906 atoms PME
>> >>>>>> >> ------------------------------**-------------
>> >>>>>> >>
>> >>>>>> >> 1 x GTX_TITAN: | ns/day = 25.01
>> >>>>>> seconds/ns =
>> >>>>>> >> 3454.56
>> >>>>>> >>
>> >>>>>> >> CELLULOSE_PRODUCTION_NVE - 408,609 atoms PME
>> >>>>>> >> ------------------------------**--------------
>> >>>>>> >>
>> >>>>>> >> 1 x GTX_TITAN: Error: unspecified launch failure
>> >>>>>> launching
>> >>>>>> >> kernel
>> >>>>>> >> kNLSkinTest
>> >>>>>> >> cudaFree GpuBuffer::Deallocate failed unspecified launch
>> failure
>> >>>>>> >> grep: mdinfo.1GTX_TITAN: No such file or directory
>> >>>>>> >>
>> >>>>>> >> TRPCAGE_PRODUCTION - 304 atoms GB
>> >>>>>> >> ------------------------------**---
>> >>>>>> >> 1 x GTX_TITAN: | ns/day = 595.09
>> >>>>>> seconds/ns =
>> >>>>>> >> 145.19
>> >>>>>> >>
>> >>>>>> >> MYOGLOBIN_PRODUCTION - 2,492 atoms GB
>> >>>>>> >> ------------------------------**-------
>> >>>>>> >>
>> >>>>>> >> 1 x GTX_TITAN: | ns/day = 202.56
>> >>>>>> seconds/ns =
>> >>>>>> >> 426.53
>> >>>>>> >>
>> >>>>>> >> NUCLEOSOME_PRODUCTION - 25,095 atoms GB
>> >>>>>> >> ------------------------------**---------
>> >>>>>> >>
>> >>>>>> >> 1 x GTX_TITAN:
>> >>>>>> >>
>> >>>>>> >>
>> >>>>>> >>
>> >>>>>> >>
>> >>>>>> >>
>> >>>>>> >>
>> >>>>>> >>
>> >>>>>> >> Dne Tue, 28 May 2013 20:42:32 +0200 ET <sketchfoot.gmail.com>
>> >>>>>> napsal/-a:
>> >>>>>> >>
>> >>>>>> >> > Hi,
>> >>>>>> >> >
>> >>>>>> >> > I just got a superclocked Titan and one at normal freq. The
>> >>>>>> first
>> >>>>>> one
>> >>>>>> >> ran
>> >>>>>> >> > like a charm with no issues so far. The other standard
>> clocked
>> >>>>>> one
>> >>>>>> >> could
>> >>>>>> >> > never get past the constant pressure stage in an NPT
>> >>>>>> simulation.
>> >>>>>> It
>> >>>>>> >> kept
>> >>>>>> >> > writing NAN or ********* in the outfile. I swapped them
>> about
>> >>>>>> in
>> >>>>>> the
>> >>>>>> >> pcie
>> >>>>>> >> > lanes then ran it solo in each one of the lanes. Despite all
>> >>>>>> this
>> >>>>>> it
>> >>>>>> >> was
>> >>>>>> >> > still failing the benchmark that the other one had no
>> problems
>> >>>>>> with.
>> >>>>>> >> >
>> >>>>>> >> > I couldn't find any memory errors with GPU-burn either, but
>> as
>> >>>>>> they
>> >>>>>> >> cost
>> >>>>>> >> > near a grand a piece, I RMA'd it today. I recommend you to
>> do
>> >>>>>> the
>> >>>>>> >> same if
>> >>>>>> >> > its not giving you any joy. Life's too short. :)
>> >>>>>> >> >
>> >>>>>> >> > br,
>> >>>>>> >> > g
>> >>>>>> >> >
>> >>>>>> >> >
>> >>>>>> >> > On 28 May 2013 16:57, Scott Le Grand <varelse2005.gmail.com>
>> >>>>>> wrote:
>> >>>>>> >> >
>> >>>>>> >> >> AMBER != NAMD...
>> >>>>>> >> >>
>> >>>>>> >> >> GTX 680 != GTX Titan...
>> >>>>>> >> >>
>> >>>>>> >> >> Ian's suggestion is a good one. But even then, you need to
>> >>>>>> test
>> >>>>>> >> >> your
>> >>>>>> >> >> GPUs
>> >>>>>> >> >> as the Titans are running right on the edge of stability.
>> >>>>>> Like I
>> >>>>>> >> told
>> >>>>>> >> >> Marek, try running 100K iterations of Cellulose NVE twice
>> with
>> >>>>>> the
>> >>>>>> >> same
>> >>>>>> >> >> random seed. if you don't get identically bit accurate
>> >>>>>> output,
>> >>>>>> your
>> >>>>>> >> >> GPU is
>> >>>>>> >> >> not working. Memtest programs do not catch this because
>> (I am
>> >>>>>> >> guessing)
>> >>>>>> >> >> they are designed for a uniform memory hierarchy and only
>> one
>> >>>>>> path
>> >>>>>> >> >> to
>> >>>>>> >> >> read
>> >>>>>> >> >> and write data. I have a stock GTX Titan that cannot pass
>> the
>> >>>>>> >> Cellulose
>> >>>>>> >> >> NVE test and another one that does. I spent a couple days
>> on
>> >>>>>> the
>> >>>>>> >> former
>> >>>>>> >> >> GPU looking for the imaginary bug that went away like magic
>> >>>>>> the
>> >>>>>> >> second I
>> >>>>>> >> >> switched out the GPU.
>> >>>>>> >> >>
>> >>>>>> >> >> Scott
>> >>>>>> >> >>
>> >>>>>> >> >>
>> >>>>>> >> >>
>> >>>>>> >> >>
>> >>>>>> >> >>
>> >>>>>> >> >> On Tue, May 28, 2013 at 8:11 AM, Robert Konecny
>> <rok.ucsd.edu
>> >
>> >>>>>> wrote:
>> >>>>>> >> >>
>> >>>>>> >> >> > Hi Scott,
>> >>>>>> >> >> >
>> >>>>>> >> >> > unfortunately we are seeing similar Amber instability on
>> GTX
>> >>>>>> >> Titans as
>> >>>>>> >> >> > Marek is. We have a box with four GTX Titans (not
>> >>>>>> oveclocked)
>> >>>>>> >> running
>> >>>>>> >> >> > CentOS 6.3 with NVidia 319.17 driver and Amber 12.2. Any
>> >>>>>> Amber
>> >>>>>> >> >> simulation
>> >>>>>> >> >> > longer than 10-15 min eventually crashes on these cards,
>> >>>>>> including
>> >>>>>> >> >> both
>> >>>>>> >> >> JAC
>> >>>>>> >> >> > benchmarks (with extended run time). This is
>> reproducible on
>> >>>>>> all
>> >>>>>> >> four
>> >>>>>> >> >> > cards.
>> >>>>>> >> >> >
>> >>>>>> >> >> > To eliminate the possible hardware error we ran extended
>> GPU
>> >>>>>> >> >> > memory
>> >>>>>> >> >> tests
>> >>>>>> >> >> > on all four Titans with memtestG80, cuda_memtest and also
>> >>>>>> gpu_burn
>> >>>>>> >> -
>> >>>>>> >> >> all
>> >>>>>> >> >> > finished without errors. Since I agree that these
>> programs
>> >>>>>> may
>> >>>>>> not
>> >>>>>> >> >> test
>> >>>>>> >> >> the
>> >>>>>> >> >> > GPU completely we also set up simulations with NAMD. We
>> can
>> >>>>>> run
>> >>>>>> >> four
>> >>>>>> >> >> NAMD
>> >>>>>> >> >> > simulations simultaneously for many days without any
>> errors
>> >>>>>> on
>> >>>>>> >> >> > this
>> >>>>>> >> >> > hardware. For reference - we also have exactly the same
>> >>>>>> server
>> >>>>>> >> >> > with
>> >>>>>> >> >> the
>> >>>>>> >> >> > same hardware components but with four GTX680s and this
>> >>>>>> setup
>> >>>>>> >> >> > works
>> >>>>>> >> >> just
>> >>>>>> >> >> > fine for Amber. So all this leads me to believe that a
>> >>>>>> hardware
>> >>>>>> >> error
>> >>>>>> >> >> is
>> >>>>>> >> >> > not very likely.
>> >>>>>> >> >> >
>> >>>>>> >> >> > I would appreciate your comments on this, perhaps there
>> is
>> >>>>>> >> something
>> >>>>>> >> >> else
>> >>>>>> >> >> > causing these errors which we are not seeing.
>> >>>>>> >> >> >
>> >>>>>> >> >> > Thanks,
>> >>>>>> >> >> >
>> >>>>>> >> >> > Robert
>> >>>>>> >> >> >
>> >>>>>> >> >> >
>> >>>>>> >> >> > On Mon, May 27, 2013 at 04:25:24PM -0700, Scott Le Grand
>> >>>>>> wrote:
>> >>>>>> >> >> > > I have two GTX Titans. One is defective, the other is
>> >>>>>> not.
>> >>>>>> >> >> > Unfortunately,
>> >>>>>> >> >> > > they both pass all standard GPU memory tests.
>> >>>>>> >> >> > >
>> >>>>>> >> >> > > What the defective one doesn't do is generate
>> reproducibly
>> >>>>>> >> >> bit-accurate
>> >>>>>> >> >> > > outputs for simulations of Factor IX (90,986 atoms) or
>> >>>>>> larger,
>> >>>>>> >> >> > > of
>> >>>>>> >> >> 100K
>> >>>>>> >> >> or
>> >>>>>> >> >> > > so iterations.
>> >>>>>> >> >> > >
>> >>>>>> >> >> > > Which is yet another reason why I insist on MD
>> algorithms
>> >>>>>> >> >> (especially
>> >>>>>> >> >> on
>> >>>>>> >> >> > > GPUS) being deterministic. Besides its ability to find
>> >>>>>> software
>> >>>>>> >> >> bugs,
>> >>>>>> >> >> > and
>> >>>>>> >> >> > > fulfilling one of the most important tenets of science,
>> >>>>>> it's
>> >>>>>> a
>> >>>>>> >> great
>> >>>>>> >> >> way
>> >>>>>> >> >> > to
>> >>>>>> >> >> > > diagnose defective hardware with very little effort.
>> >>>>>> >> >> > >
>> >>>>>> >> >> > > 928 MHz? That's 6% above the boost clock of a stock
>> >>>>>> Titan.
>> >>>>>> >> Titan
>> >>>>>> >> >> is
>> >>>>>> >> >> > > pushing the performance envelope as is. If you're
>> going
>> >>>>>> to
>> >>>>>> pay
>> >>>>>> >> the
>> >>>>>> >> >> > premium
>> >>>>>> >> >> > > for such chips, I'd send them back until you get one
>> that
>> >>>>>> runs
>> >>>>>> >> >> correctly.
>> >>>>>> >> >> > > I'm very curious how fast you can push one of these
>> things
>> >>>>>> >> >> > > before
>> >>>>>> >> >> they
>> >>>>>> >> >> > give
>> >>>>>> >> >> > > out.
>> >>>>>> >> >> > >
>> >>>>>> >> >> > >
>> >>>>>> >> >> > >
>> >>>>>> >> >> > >
>> >>>>>> >> >> > >
>> >>>>>> >> >> > >
>> >>>>>> >> >> > >
>> >>>>>> >> >> > > On Mon, May 27, 2013 at 10:01 AM, Marek Maly
>> >>>>>> <marek.maly.ujep.cz
>> >>>>>> >
>> >>>>>> >> >> wrote:
>> >>>>>> >> >> > >
>> >>>>>> >> >> > > > Dear all,
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > I have recently bought two "EVGA GTX TITAN
>> Superclocked"
>> >>>>>> GPUs.
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > I did the first calculations (pmemd.cuda in Amber12)
>> >>>>>> with
>> >>>>>> >> systems
>> >>>>>> >> >> > around
>> >>>>>> >> >> > > > 60K atoms without any problems (NPT, Langevin), but
>> >>>>>> when I
>> >>>>>> >> later
>> >>>>>> >> >> tried
>> >>>>>> >> >> > > > with bigger systems (around 100K atoms) I obtained
>> >>>>>> "classical"
>> >>>>>> >> >> > irritating
>> >>>>>> >> >> > > > errors
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > cudaMemcpy GpuBuffer::Download failed unspecified
>> launch
>> >>>>>> >> failure
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > just after few thousands of MD steps.
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > So this was obviously the reason for memtestG80
>> tests.
>> >>>>>> >> >> > > > ( https://simtk.org/home/memtest ).
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > So I compiled memtestG80 from sources (
>> >>>>>> >> memtestG80-1.1-src.tar.gz
>> >>>>>> >> >> )
>> >>>>>> >> >> and
>> >>>>>> >> >> > > > then tested
>> >>>>>> >> >> > > > just small part of memory GPU (200 MB) using 100
>> >>>>>> iterations.
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > On both cards I have obtained huge amount of errors
>> but
>> >>>>>> "just"
>> >>>>>> >> on
>> >>>>>> >> >> > > > "Random blocks:". 0 errors in all remaining tests in
>> all
>> >>>>>> >> >> iterations.
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > ------THE LAST ITERATION AND FINAL RESULTS-------
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > Test iteration 100 (GPU 0, 200 MiB): 169736847
>> errors so
>> >>>>>> far
>> >>>>>> >> >> > > > Moving Inversions (ones and zeros): 0 errors
>> (6
>> >>>>>> ms)
>> >>>>>> >> >> > > > Memtest86 Walking 8-bit: 0 errors (53 ms)
>> >>>>>> >> >> > > > True Walking zeros (8-bit): 0 errors (26 ms)
>> >>>>>> >> >> > > > True Walking ones (8-bit): 0 errors (26 ms)
>> >>>>>> >> >> > > > Moving Inversions (random): 0 errors (6 ms)
>> >>>>>> >> >> > > > Memtest86 Walking zeros (32-bit): 0 errors
>> (105
>> >>>>>> ms)
>> >>>>>> >> >> > > > Memtest86 Walking ones (32-bit): 0 errors
>> (104
>> >>>>>> ms)
>> >>>>>> >> >> > > > Random blocks: 1369863 errors (27 ms)
>> >>>>>> >> >> > > > Memtest86 Modulo-20: 0 errors (215 ms)
>> >>>>>> >> >> > > > Logic (one iteration): 0 errors (4 ms)
>> >>>>>> >> >> > > > Logic (4 iterations): 0 errors (8 ms)
>> >>>>>> >> >> > > > Logic (shared memory, one iteration): 0
>> errors
>> >>>>>> (8
>> >>>>>> ms)
>> >>>>>> >> >> > > > Logic (shared-memory, 4 iterations): 0 errors
>> >>>>>> (25
>> >>>>>> ms)
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > Final error count after 100 iterations over 200 MiB
>> of
>> >>>>>> GPU
>> >>>>>> >> memory:
>> >>>>>> >> >> > > > 171106710 errors
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > ------------------------------**------------
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > I have some questions and would be really grateful
>> for
>> >>>>>> any
>> >>>>>> >> >> comments.
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > Regarding overclocking, using the deviceQuery I found
>> >>>>>> out
>> >>>>>> that
>> >>>>>> >> >> under
>> >>>>>> >> >> > linux
>> >>>>>> >> >> > > > both cards run
>> >>>>>> >> >> > > > automatically using boost shader/GPU frequency which
>> is
>> >>>>>> here
>> >>>>>> >> 928
>> >>>>>> >> >> MHz
>> >>>>>> >> >> > (the
>> >>>>>> >> >> > > > basic value for these factory OC cards is 876 MHz).
>> >>>>>> >> >> > > > deviceQuery
>> >>>>>> >> >> > reported
>> >>>>>> >> >> > > > Memory Clock rate is 3004 MHz although "it" should be
>> >>>>>> 6008
>> >>>>>> MHz
>> >>>>>> >> but
>> >>>>>> >> >> > maybe
>> >>>>>> >> >> > > > the quantity which is reported by deviceQuery "Memory
>> >>>>>> Clock
>> >>>>>> >> rate"
>> >>>>>> >> >> is
>> >>>>>> >> >> > > > different from the product specification "Memory
>> Clock"
>> >>>>>> .
>> >>>>>> It
>> >>>>>> >> seems
>> >>>>>> >> >> that
>> >>>>>> >> >> > > > "Memory Clock rate" = "Memory Clock"/2. Am I right ?
>> Or
>> >>>>>> just
>> >>>>>> >> >> > deviceQuery
>> >>>>>> >> >> > > > is not able to read this spec. properly
>> >>>>>> >> >> > > > in Titan GPU ?
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > Anyway for the moment I assume that the problem
>> might be
>> >>>>>> due
>> >>>>>> >> >> > > > to
>> >>>>>> >> >> the
>> >>>>>> >> >> > high
>> >>>>>> >> >> > > > shader/GPU frequency.
>> >>>>>> >> >> > > > (see here :
>> >>>>>> http://folding.stanford.edu/**English/DownloadUtils<
>> http://folding.stanford.edu/English/DownloadUtils>
>> >>>>>> )
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > To verify this hypothesis one should perhaps
>> UNDERclock
>> >>>>>> to
>> >>>>>> >> basic
>> >>>>>> >> >> > frequency
>> >>>>>> >> >> > > > which is in this
>> >>>>>> >> >> > > > model 876 MHz or even to the TITAN REFERENCE
>> frequency
>> >>>>>> which
>> >>>>>> >> >> > > > is
>> >>>>>> >> >> 837
>> >>>>>> >> >> > MHz.
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > Obviously I am working with these cards under linux
>> >>>>>> (CentOS
>> >>>>>> >> >> > > > 2.6.32-358.6.1.el6.x86_64) and as I found, the OC
>> tools
>> >>>>>> under
>> >>>>>> >> >> linux
>> >>>>>> >> >> > are in
>> >>>>>> >> >> > > > fact limited just to NVclock utility, which is
>> >>>>>> unfortunately
>> >>>>>> >> >> > > > out of date (at least speaking about the GTX Titan
>> ). I
>> >>>>>> have
>> >>>>>> >> >> obtained
>> >>>>>> >> >> > this
>> >>>>>> >> >> > > > message when I wanted
>> >>>>>> >> >> > > > just to let NVclock utility to read and print shader
>> and
>> >>>>>> >> >> > > > memory
>> >>>>>> >> >> > > > frequencies of my Titan's:
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > >
>> >>>>>> >> >>
>> >>>>>> ------------------------------**------------------------------**
>> >>>>>> -------
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > [root.dyn-138-272 NVCLOCK]# nvclock -s --speeds
>> >>>>>> >> >> > > > Card: Unknown Nvidia card
>> >>>>>> >> >> > > > Card number: 1
>> >>>>>> >> >> > > > Memory clock: -2147483.750 MHz
>> >>>>>> >> >> > > > GPU clock: -2147483.750 MHz
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > Card: Unknown Nvidia card
>> >>>>>> >> >> > > > Card number: 2
>> >>>>>> >> >> > > > Memory clock: -2147483.750 MHz
>> >>>>>> >> >> > > > GPU clock: -2147483.750 MHz
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > >
>> >>>>>> >> >>
>> >>>>>> ------------------------------**------------------------------**
>> >>>>>> -------
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > I would be really grateful for some tips regarding
>> >>>>>> "NVclock
>> >>>>>> >> >> > alternatives",
>> >>>>>> >> >> > > > but after wasting some hours with googling it seems
>> that
>> >>>>>> there
>> >>>>>> >> is
>> >>>>>> >> >> no
>> >>>>>> >> >> > other
>> >>>>>> >> >> > > > Linux
>> >>>>>> >> >> > > > tool with NVclock functionality. So the only
>> >>>>>> possibility is
>> >>>>>> >> here
>> >>>>>> >> >> > perhaps
>> >>>>>> >> >> > > > to edit
>> >>>>>> >> >> > > > GPU bios with some Lin/DOS/Win tools like (Kepler
>> BIOS
>> >>>>>> >> >> > > > Tweaker,
>> >>>>>> >> >> > NVflash)
>> >>>>>> >> >> > > > but obviously
>> >>>>>> >> >> > > > I would like to rather avoid such approach as using
>> it
>> >>>>>> means
>> >>>>>> >> >> perhaps
>> >>>>>> >> >> > also
>> >>>>>> >> >> > > > to void the warranty even if I am going to underclock
>> >>>>>> the
>> >>>>>> GPUs
>> >>>>>> >> >> not to
>> >>>>>> >> >> > > > overclock them.
>> >>>>>> >> >> > > > So before this eventual step (GPU bios editing) I
>> would
>> >>>>>> like
>> >>>>>> >> >> > > > to
>> >>>>>> >> >> have
>> >>>>>> >> >> > some
>> >>>>>> >> >> > > > approximative estimate
>> >>>>>> >> >> > > > of the probability, that the problems are here really
>> >>>>>> because
>> >>>>>> >> of
>> >>>>>> >> >> the
>> >>>>>> >> >> > > > overclocking
>> >>>>>> >> >> > > > (too high (boost) default shader frequency).
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > This probability I hope to estimate from the eventual
>> >>>>>> >> responses of
>> >>>>>> >> >> > another
>> >>>>>> >> >> > > > Amber/Titan SC users, if I am not the only crazy guy
>> who
>> >>>>>> >> >> > > > bought
>> >>>>>> >> >> this
>> >>>>>> >> >> > model
>> >>>>>> >> >> > > > for Amber calculations :)) But of course any eventual
>> >>>>>> >> experiences
>> >>>>>> >> >> with
>> >>>>>> >> >> > > > Titan cards related to their memtestG80 results and
>> >>>>>> >> >> UNDER/OVERclocking
>> >>>>>> >> >> > > > (if possible in Linux OS) are of course welcomed as
>> >>>>>> well !
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > My HW/SW configuration
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > motherboard: ASUS P9X79 PRO
>> >>>>>> >> >> > > > CPU: Intel Core i7-3930K
>> >>>>>> >> >> > > > RAM: CRUCIAL Ballistix Sport 32GB (4x8GB) DDR3 1600
>> VLP
>> >>>>>> >> >> > > > CASE: CoolerMaster Dominator CM-690 II Advanced,
>> >>>>>> >> >> > > > Power:Enermax PLATIMAX EPM1200EWT 1200W, 80+,
>> Platinum
>> >>>>>> >> >> > > > GPUs : 2 x EVGA GTX TITAN Superclocked 6GB
>> >>>>>> >> >> > > > cooler: Cooler Master Hyper 412 SLIM
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > OS: CentOS (2.6.32-358.6.1.el6.x86_64)
>> >>>>>> >> >> > > > driver version: 319.17
>> >>>>>> >> >> > > > cudatoolkit_5.0.35_linux_64_**rhel6.x
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > The computer is in air-conditioned room with
>> permanent
>> >>>>>> >> >> > > > external
>> >>>>>> >> >> > > > temperature around 18°C
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > Thanks a lot in advance for any
>> comment/experience !
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > Best wishes,
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > Marek
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > --
>> >>>>>> >> >> > > > Tato zpráva byla vytvořena převratným poštovním
>> klientem
>> >>>>>> >> >> > > > Opery:
>> >>>>>> >> >> > > > http://www.opera.com/mail/
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > > ______________________________**_________________
>> >>>>>> >> >> > > > AMBER mailing list
>> >>>>>> >> >> > > > AMBER.ambermd.org
>> >>>>>> >> >> > > >
>> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>> >> >> > > >
>> >>>>>> >> >> > > ______________________________**_________________
>> >>>>>> >> >> > > AMBER mailing list
>> >>>>>> >> >> > > AMBER.ambermd.org
>> >>>>>> >> >> > >
>> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>> >> >> >
>> >>>>>> >> >> > ______________________________**_________________
>> >>>>>> >> >> > AMBER mailing list
>> >>>>>> >> >> > AMBER.ambermd.org
>> >>>>>> >> >> >
>> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>> >> >> >
>> >>>>>> >> >> ______________________________**_________________
>> >>>>>> >> >> AMBER mailing list
>> >>>>>> >> >> AMBER.ambermd.org
>> >>>>>> >> >>
>> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>> >> >>
>> >>>>>> >> > ______________________________**_________________
>> >>>>>> >> > AMBER mailing list
>> >>>>>> >> > AMBER.ambermd.org
>> >>>>>> >> >
>> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>> >> >
>> >>>>>> >> > __________ Informace od ESET NOD32 Antivirus, verze databaze
>> >>>>>> 8385
>> >>>>>> >> > (20130528) __________
>> >>>>>> >> >
>> >>>>>> >> > Tuto zpravu proveril ESET NOD32 Antivirus.
>> >>>>>> >> >
>> >>>>>> >> > http://www.eset.cz
>> >>>>>> >> >
>> >>>>>> >> >
>> >>>>>> >> >
>> >>>>>> >>
>> >>>>>> >>
>> >>>>>> >> --
>> >>>>>> >> Tato zpráva byla vytvořena převratným poštovním klientem
>> Opery:
>> >>>>>> >> http://www.opera.com/mail/
>> >>>>>> >>
>> >>>>>> >> ______________________________**_________________
>> >>>>>> >> AMBER mailing list
>> >>>>>> >> AMBER.ambermd.org
>> >>>>>> >>
>> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>> >>
>> >>>>>> > ______________________________**_________________
>> >>>>>> > AMBER mailing list
>> >>>>>> > AMBER.ambermd.org
>> >>>>>> >
>> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>> >
>> >>>>>> > __________ Informace od ESET NOD32 Antivirus, verze databaze
>> 8386
>> >>>>>> > (20130528) __________
>> >>>>>> >
>> >>>>>> > Tuto zpravu proveril ESET NOD32 Antivirus.
>> >>>>>> >
>> >>>>>> > http://www.eset.cz
>> >>>>>> >
>> >>>>>> >
>> >>>>>> >
>> >>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
>> >>>>>> http://www.opera.com/mail/
>> >>>>>>
>> >>>>>> ______________________________**_________________
>> >>>>>> AMBER mailing list
>> >>>>>> AMBER.ambermd.org
>> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>>
>> >>>>>> ______________________________**_________________
>> >>>>> AMBER mailing list
>> >>>>> AMBER.ambermd.org
>> >>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>
>> >>>>>
>> >>>>> ______________________________**_________________
>> >>>>> AMBER mailing list
>> >>>>> AMBER.ambermd.org
>> >>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> ______________________________**_________________
>> >>>> AMBER mailing list
>> >>>> AMBER.ambermd.org
>> >>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>
>> >>>> __________ Informace od ESET NOD32 Antivirus, verze databaze 8386
>> >>>> (20130528) __________
>> >>>>
>> >>>> Tuto zpravu proveril ESET NOD32 Antivirus.
>> >>>>
>> >>>> http://www.eset.cz
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>
>> >>> --
>> >>> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
>> >>> http://www.opera.com/mail/
>> >>> _______________________________________________
>> >>> AMBER mailing list
>> >>> AMBER.ambermd.org
>> >>> http://lists.ambermd.org/mailman/listinfo/amber
>> >>>
>> >>>
>> > _______________________________________________
>> > AMBER mailing list
>> > AMBER.ambermd.org
>> > http://lists.ambermd.org/mailman/listinfo/amber
>> >
>> > __________ Informace od ESET NOD32 Antivirus, verze databaze 8390
>> > (20130529) __________
>> >
>> > Tuto zpravu proveril ESET NOD32 Antivirus.
>> >
>> > http://www.eset.cz
>> >
>> >
>> >
>>
>>
>> --
>> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
>> http://www.opera.com/mail/
>>
>> _______________________________________________
>> AMBER mailing list
>> AMBER.ambermd.org
>> http://lists.ambermd.org/mailman/listinfo/amber
>>
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
>
> __________ Informace od ESET NOD32 Antivirus, verze databaze 8390
> (20130529) __________
>
> Tuto zpravu proveril ESET NOD32 Antivirus.
>
> http://www.eset.cz
>
>
>
--
Tato zpráva byla vytvořena převratným poštovním klientem Opery:
http://www.opera.com/mail/
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Wed May 29 2013 - 17:00:02 PDT
