OK, if I have enough time tomorrow/Friday I will try,
for the moment good night :))
M.
Dne Thu, 30 May 2013 02:46:59 +0200 Scott Le Grand <varelse2005.gmail.com>
napsal/-a:
> It can't hurt to try all combinations.
> On May 29, 2013 4:50 PM, "Marek Maly" <marek.maly.ujep.cz> wrote:
>
>> OK,
>>
>> BTW I just run 500K steps benchmarks (again 2 repetitions for each test)
>> :))
>> but before I installed the newest driver 319.23 (with reboot) so let
>> see
>> ...
>>
>> What about that Cuda 5.5 ? Anyway 319.23 driver has already some
>> "Cuda 5.5 support/part= CUDA driver" as deviceQuery printed (after
>> 319.23
>> installation)
>>
>> --------
>> ...CUDA Driver = CUDART, CUDA Driver Version = 5.5, CUDA Runtime
>> Version =
>> 5.0...
>> --------
>>
>> M.
>>
>>
>> Dne Thu, 30 May 2013 01:05:24 +0200 Scott Le Grand
>> <varelse2005.gmail.com>
>> napsal/-a:
>>
>> > Sorry, I missed the attachment because I'm on a cellphone and on a
>> > business
>> > trip. Anyway, neither Titan showed fully deterministic behavior and
>> that
>> > is worrisome. Notice that the 680 runs were indeed so. The latter is
>> > the
>> > expected behavior and exactly what I see with one of my Titans and my
>> > K20.
>> >
>> > Which means we need to figure this out. For now, could you take it on
>> > faith that changing ntpr changes the trajectory by changing the code
>> > executed and that it doing so is not a bug? Playing around with it is
>> > just
>> > confusing the issue right now.
>> >
>> > What would help clarify is if someone could try these tests on K20 or
>> > K20X. I would love for someone to demonstrate this is a coding
>> error on
>> > my part because I can fix that. The evidence just isn't leading me
>> that
>> > way right now.
>> >
>> > Scott
>> > On May 29, 2013 2:41 PM, "Marek Maly" <marek.maly.ujep.cz> wrote:
>> >
>> >> Hi Scott,
>> >>
>> >> what do you mean by "try running for 100k steps before comparing
>> >> energies". In all
>> >> tests I have actually done I did exactly 100k steps before comparing
>> >> energies
>> >> E_tot(at step 100 000). So you mean to extend tests to 200k steps
>> now ?
>> >>
>> >> M.
>> >>
>> >>
>> >> Dne Wed, 29 May 2013 22:46:58 +0200 Scott Le Grand
>> >> <varelse2005.gmail.com>
>> >> napsal/-a:
>> >>
>> >> > Ps try running for 100k steps before comparing energies and I
>> suspect
>> >> no
>> >> > two simulations will match.
>> >> > On May 29, 2013 1:41 PM, "Scott Le Grand" <varelse2005.gmail.com>
>> >> wrote:
>> >> >
>> >> >> Your Titan setup is hosed. Your results were not 100%
>> deterministic
>> >> for
>> >> >> the same inputs.
>> >> >>
>> >> >> Energies + Forces use a different subroutine than just Forces
>> hence
>> >> the
>> >> >> ntpr dependence. Hence changing ntpr effectively is changing the
>> >> input.
>> >> >>
>> >> >> It's 100% ironclad reproducibility that matters and you
>> demonstrated
>> >> >> it's
>> >> >> not happening.
>> >> >> On May 29, 2013 1:30 PM, "Marek Maly" <marek.maly.ujep.cz> wrote:
>> >> >>
>> >> >>> Hi all,
>> >> >>>
>> >> >>> First of all thanks to Ross for his update ! although it is
>> question
>> >> >>> if it helps to solve all the reported Amber issues with Titan/OC
>> >> Titan
>> >> >>> GPUs .
>> >> >>> So let's see and hope :))
>> >> >>>
>> >> >>> Here are my results - see the attached TXT file with tables where
>> >> >>> the results from the tests are summarised. I did twice the same
>> >> >>> Amber benchmark tests on each GPU (both titans, GTX 680 and GTX
>> 580)
>> >> >>> to see reproducibility of the results after 100K steps at
>> ig=default
>> >> >>> (so ig not present in mdin file).
>> >> >>>
>> >> >>> The first table contains ns/day estimates obtained for each
>> >> molecular
>> >> >>> system
>> >> >>> for each TITAN GPU. Interestingly estimates obtained for the same
>> >> >>> system
>> >> >>> in different
>> >> >>> round slightly differ, but maybe that's OK.
>> >> >>>
>> >> >>> In the second table there are values of the total energy after
>> 100k
>> >> >>> steps
>> >> >>> to check
>> >> >>> reproducibility of the results.
>> >> >>>
>> >> >>> Here is summarisation :
>> >> >>>
>> >> >>> #1 - simulation crashes on TITANs
>> >> >>>
>> >> >>> Interestingly there was just one simulation crash in JAC_NPT
>> >> (TITAN_0,
>> >> >>> ROUND_1) the remaining
>> >> >>> 3 TITAN JAC_NPT simulations were finished. There were also 3
>> times
>> >> >>> crashes in CELLULOSE_NVE
>> >> >>> test but the last simulation (TITAN_1,ROUND_2) was finished
>> without
>> >> any
>> >> >>> problem. All the remaining
>> >> >>> simulations were always finished without any problem. So the
>> >> simulation
>> >> >>> crashes seem to be
>> >> >>> not-reproducible/unpredictible on some moleacular systems/(mdin
>> >> >>> setups).
>> >> >>>
>> >> >>> CRASH ERRORS:
>> >> >>>
>> >> >>> a) JAC_NPT (TITAN_0, ROUND_1)
>> >> >>> Here 11k steps were successfully done before crash, I found this
>> >> error
>> >> >>> in mdout file:
>> >> >>>
>> >> >>> | ERROR: max pairlist cutoff must be less than unit cell max
>> >> sphere
>> >> >>> radius!
>> >> >>>
>> >> >>> b) CELLULOSE_NVE (TITAN_0, ROUND_1, ROUND_2; TITAN_1, ROUND_1 )
>> >> >>> Here I did not find any error in mdout file. Just this error was
>> >> >>> written
>> >> >>> on standard output
>> >> >>> (screen/nohup.out file):
>> >> >>>
>> >> >>> ------
>> >> >>> Error: unspecified launch failure launching kernel kNLSkinTest
>> >> >>> cudaFree GpuBuffer::Deallocate failed unspecified launch failure
>> >> >>> grep: mdinfo.1GTX_TITAN: No such file or directory
>> >> >>> -----
>> >> >>>
>> >> >>> in all three cases.
>> >> >>>
>> >> >>> Here on CELLULOSE_NVE case I started to play with NTPR parameter
>> >> >>> (originally just
>> >> >>> on TITAN-0 GPU), to see how many steps were successfully done
>> here
>> >> >>> before
>> >> >>> crash, then this my
>> >> >>> small research started to be more interesting than I ever thought
>> >> :))
>> >> >>> see
>> >> >>> here
>> >> >>> chronologically my results for E_tot after 2000 steps for
>> different
>> >> >>> GPUs
>> >> >>> (machines) - I repeated calculation several times for the given
>> NTPR
>> >> >>> just
>> >> >>> to be sure.
>> >> >>>
>> >> >>> TITAN-0, Etot after 2000 steps
>> >> >>>
>> >> >>> NTPR=10
>> >> >>>
>> >> >>> -443256.6867
>> >> >>> -443256.6867
>> >> >>> -443256.6867
>> >> >>>
>> >> >>> NTPR=100
>> >> >>>
>> >> >>> -443250.1350
>> >> >>> -443250.1350
>> >> >>> -443250.1350
>> >> >>>
>> >> >>> NTPR=200
>> >> >>>
>> >> >>> -443261.0705
>> >> >>> -443261.0705
>> >> >>> -443072.3097
>> >> >>> -443261.0705
>> >> >>> -443261.0705
>> >> >>> -443261.0705
>> >> >>> -443261.0705
>> >> >>>
>> >> >>> NTPR=10 (again just to verify)
>> >> >>>
>> >> >>> -443256.6867
>> >> >>> -443256.6867
>> >> >>>
>> >> >>>
>> >> >>> Then I tried with TITAN-1
>> >> >>>
>> >> >>> NTPR=10
>> >> >>>
>> >> >>> -443256.6867
>> >> >>> -443256.6867
>> >> >>>
>> >> >>> NTPR=100
>> >> >>>
>> >> >>> -443250.1350
>> >> >>> -443250.1350
>> >> >>>
>> >> >>> NTPR=200
>> >> >>>
>> >> >>> -443261.0705
>> >> >>> -443261.0705
>> >> >>>
>> >> >>>
>> >> >>> Then I tried with GTX-580
>> >> >>>
>> >> >>> NTPR=10
>> >> >>>
>> >> >>> -443256.6867
>> >> >>> -443256.6867
>> >> >>>
>> >> >>> NTPR=200
>> >> >>>
>> >> >>> -443261.0705
>> >> >>> -443261.0705
>> >> >>>
>> >> >>> then I tried with GTX-680
>> >> >>>
>> >> >>> NTPR=10 Etot after 2000 steps
>> >> >>>
>> >> >>> -443256.6711
>> >> >>> -443256.6711
>> >> >>>
>> >> >>> NTPR=200 Etot after 2000 steps
>> >> >>>
>> >> >>> -443261.0705
>> >> >>> -443261.0705
>> >> >>>
>> >> >>> Any idea why energies should depend on frequency of energy
>> records
>> >> >>> (NTPR)
>> >> >>> ?
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> #2 - reproducibility on TITANs (see attached table.txt)
>> >> >>>
>> >> >>> Also here are differences depending on concrete systems/setups.
>> >> >>> While in case of FACTOR_IX_NVE, FACTOR_IX_NPT, TRPCAGE, MYOGLOBIN
>> >> >>> systems
>> >> >>> I have obtained
>> >> >>> 100% reproducibility (the results for the given system were
>> >> identical
>> >> >>> for
>> >> >>> both cards/all ROUNDs)
>> >> >>> on systems JAC_NVE, JAC_NPT, NUCLEOSOME I obtained small
>> >> differences
>> >> >>> in
>> >> >>> general however in case
>> >> >>> of TITAN_1 GPU also NUCLEOSOME results were 100% reproducible.
>> >> >>> Moreover
>> >> >>> for the TITAN_1 card which succeeded to finish CELLULOSE test at
>> >> least
>> >> >>> in
>> >> >>> ROUND_2 I did 3rd additional round and I got the identical
>> result as
>> >> >>> from
>> >> >>> the ROUND_2 (i.e. -443246.3206 ) so regarding TITAN_1 GPU I can
>> say
>> >> >>> that
>> >> >>> it is able to 100% reproduce 100k steps CELLULOSE_NVE test result
>> >> >>> perhaps
>> >> >>> on all eventually successfully finished runs :))
>> >> >>>
>> >> >>>
>> >> >>> #3 - GTX-580, GTX-680 controls
>> >> >>>
>> >> >>> Here the simulations were done without any problems and were 100%
>> >> >>> reproducible on each card however
>> >> >>> the results for the given system slightly differ between those
>> two
>> >> >>> cards
>> >> >>> with exception of the
>> >> >>> CELLULOSE system where both cards GTX-580, GTX-680 provided
>> >> identical
>> >> >>> result which is moreover
>> >> >>> nearly identical with result obtained with TITAN_1 during ROUND_2
>> >> >>> (relative difference 2e-6).
>> >> >>>
>> >> >>>
>> >> >>> TO ET:
>> >> >>> a)
>> >> >>> I had no problems with minimization stages in my own simul.
>> bigger
>> >> than
>> >> >>> 100k which crashed
>> >> >>> during heat NVT phase.
>> >> >>>
>> >> >>> b)
>> >> >>> 313.30 driver ??? OK so after 319.23 I will try experiment with
>> >> this a
>> >> >>> bit "outdated" version :))
>> >> >>> Actually I am working under 319.17. (and CUDA 5.0)
>> >> >>>
>> >> >>> c)
>> >> >>> Can you please do at least JAC_NPT, JAC_NVE, NUCLEOSOME and
>> >> >>> CELLULOSE_NVE
>> >> >>> tests using 100 000 steps
>> >> >>> (same random seed e.g. default = ig deleted from mdin if is
>> there)
>> >> >>> twice
>> >> >>> to confirm 100% reproducibility on your TITAN GPU ?
>> >> >>>
>> >> >>> TO Divi:
>> >> >>>
>> >> >>> This is also my usual approach to divide whole simulation into
>> many
>> >> >>> subtrajectories (in my case 0.5 ns = 250k 2fs steps) but it does
>> not
>> >> >>> seem
>> >> >>> to help here it self. Can you please also do the same tests
>> which I
>> >> >>> asked
>> >> >>> ET (point c) )
>> >> >>>
>> >> >>>
>> >> >>> BTW CUDA release candidate 5.5 was just released (
>> >> >>> https://developer.nvidia.com/**cuda-toolkit<
>> >> https://developer.nvidia.com/cuda-toolkit>)
>> >> >>> would it be reasonable idea to try compile/run pmemd.cuda with
>> this
>> >> >>> brand
>> >> >>> new cuda version ?
>> >> >>>
>> >> >>> Thanks !
>> >> >>>
>> >> >>> Best wishes,
>> >> >>>
>> >> >>> Marek
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> Dne Wed, 29 May 2013 03:44:33 +0200 Ross Walker
>> >> <ross.rosswalker.co.uk
>> >> >
>> >> >>> napsal/-a:
>> >> >>>
>> >> >>> Hi All,
>> >> >>>>
>> >> >>>> Just an update that we will have some fixes out soon that
>> address
>> >> some
>> >> >>>> errors we have been noticing with simulations crashing during
>> NPT
>> >> >>>> runs.
>> >> >>>> It
>> >> >>>> is possible that this is confusing the issue here as to whether
>> the
>> >> >>>> problem is related to the GTX Titan or to a possible bug in the
>> >> code.
>> >> >>>> I
>> >> >>>> hope to have the patch released within a few days at which
>> point it
>> >> >>>> would
>> >> >>>> be good to repeat these tests and then hopefully we can try to
>> >> track
>> >> >>>> down
>> >> >>>> what is going on. I find it hard to believe that so many cards
>> are
>> >> >>>> faulty
>> >> >>>> so I suspect that there may be something funky in the code with
>> >> >>>> regards
>> >> >>>> to
>> >> >>>> GTX Titans. We'll try and get it fixed as soon as possible but
>> for
>> >> now
>> >> >>>> please just wait until we get the update released for AMBER 12
>> in a
>> >> >>>> few
>> >> >>>> days and see if that helps at all.
>> >> >>>>
>> >> >>>> All the best
>> >> >>>> Ross
>> >> >>>>
>> >> >>>>
>> >> >>>> On 5/28/13 5:12 PM, "Divi/GMAIL" <dvenkatlu.gmail.com> wrote:
>> >> >>>>
>> >> >>>> I have two TITANs in my Gigabyte workstation. I have had
>> similar
>> >> >>>>> issues
>> >> >>>>> of NANs for some of the simulation setups. Never could figure
>> out
>> >> why
>> >> >>>>> the
>> >> >>>>> simulations failed for no reason. I tried 10, 12 ang. box
>> sizes.
>> >> >>>>> same
>> >> >>>>> random breakdowns. Thought of returning them suspecting memory
>> >> >>>>> errors.
>> >> >>>>> But
>> >> >>>>> some simulations ran perfectly fine. Currently running two
>> >> >>>>> calculations
>> >> >>>>> without any problems. Both are running pretty stable for over
>> >> 100ns.
>> >> >>>>> I
>> >> >>>>> suspect AMBER CUDA code may have some issues under some
>> simulation
>> >> >>>>> conditions such as NPT. In general, NVT setup is more
>> successful
>> >> than
>> >> >>>>> NPT,
>> >> >>>>> in my case.
>> >> >>>>>
>> >> >>>>> These are 287426 atoms simulation on one card (9 ns/day)
>> >> >>>>> On other card: 129049 atom setup (20 ns/day)
>> >> >>>>>
>> >> >>>>> Both using same NVT setup. (AMBER12/INTEL-12.x
>> >> >>>>> compilers/CentOS-6.3/Drivers 319.17/CUDA5.0)
>> >> >>>>>
>> >> >>>>> Input is below:
>> >> >>>>> &cntrl
>> >> >>>>> nstlim=500000, dt=0.002,
>> >> >>>>> ntx=5, irest=1, ig=-1,
>> >> >>>>> ntpr=1000, ntwr=10000, ntwx=10000,
>> >> >>>>> ntt=1, tautp=2, ntb=1, ntp=0, ntc=2, ntf=2,
>> >> >>>>> iwrap=1, ioutfm=1, ntxo=2,
>> >> >>>>> &end
>> >> >>>>>
>> >> >>>>> One suggestion If I may add: If you could run short
>> simulations
>> >> for
>> >> >>>>> no
>> >> >>>>> more
>> >> >>>>> than 500,000 steps (or 1ns with 2 fs), you might find some
>> >> stability.
>> >> >>>>> Again,
>> >> >>>>> not scientific rationale from my side. But it worked in some
>> cases
>> >> >>>>> for
>> >> >>>>> me.
>> >> >>>>>
>> >> >>>>> This is self-assembled system with GIGABYTE GA-Z77X-UP7 (with
>> >> core
>> >> >>>>> i5
>> >> >>>>> processor) and 1200W PS/16GB memory.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Best regards
>> >> >>>>> Divi
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> -----Original Message-----
>> >> >>>>> From: Scott Le Grand
>> >> >>>>> Sent: Tuesday, May 28, 2013 4:46 PM
>> >> >>>>> To: AMBER Mailing List
>> >> >>>>> Subject: Re: [AMBER] experiences with EVGA GTX TITAN
>> Superclocked
>> >> -
>> >> >>>>> memtestG80 - UNDERclocking in Linux ?
>> >> >>>>>
>> >> >>>>> You can play Russian Roulette a whole bunch of rounds without
>> >> blowing
>> >> >>>>> your
>> >> >>>>> head off.
>> >> >>>>>
>> >> >>>>> Similarly, when you have a GPU that occasionally flips a bit
>> the
>> >> >>>>> wrong
>> >> >>>>> way,
>> >> >>>>> most of the time it will be some low order perturbation to the
>> >> >>>>> coordinates
>> >> >>>>> that does little more than make the trajectory
>> nondeterministic...
>> >> >>>>> Except
>> >> >>>>> when it doesn't...
>> >> >>>>>
>> >> >>>>> You can't even detect this kind of misbehavior in GROMACS,
>> ACEMD,
>> >> or
>> >> >>>>> NAMD
>> >> >>>>> because *none* of them (to my knowledge) are capable of
>> producing
>> >> >>>>> deterministic output at production-level performance.
>> >> >>>>>
>> >> >>>>> Titans and 680s are consumer cards. I love them to death, but
>> if
>> >> >>>>> you're
>> >> >>>>> going to do production work with them, you need to qual them
>> >> >>>>> thoroughly
>> >> >>>>> before proceeding or you need to pay up and use Teslas instead.
>> >> I'd
>> >> >>>>> still
>> >> >>>>> build a cluster with Titans myself, but I'd ruthlessly RMA them
>> >> >>>>> until I
>> >> >>>>> got
>> >> >>>>> satisfaction if they couldn't pass a test consisting of
>> running an
>> >> >>>>> AMBER
>> >> >>>>> simulation for 100K iterations without either crashing or
>> >> producing a
>> >> >>>>> nondeterministic result. The customer is always right.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On Tue, May 28, 2013 at 1:20 PM, Marek Maly
>> <marek.maly.ujep.cz>
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>> I would wait for the results of my GOPU0, GPU1 double tests
>> >> before
>> >> >>>>>> any serious conclusions.
>> >> >>>>>>
>> >> >>>>>> BTW what exactly means "GPU is hosed" ? Something like GPU is
>> >> >>>>>> damaged
>> >> >>>>>> or
>> >> >>>>>> so ?
>> >> >>>>>>
>> >> >>>>>> Also would be strange (not probable) to buy 2 somehow damaged
>> >> GPUs
>> >> >>>>>> (even
>> >> >>>>>> in the same way).
>> >> >>>>>>
>> >> >>>>>> As I wrote, memtestG80 tests were negative on both cards, if
>> >> >>>>>> moreover
>> >> >>>>>> both cards will perfectly reproduce both repetitions of the
>> Amber
>> >> >>>>>> benchmarks
>> >> >>>>>> and eventually pass some another GPU tests (can you recommend
>> any
>> >> >>>>>> except
>> >> >>>>>> memtestG80 ?)
>> >> >>>>>> I still believe that the GPU cards are OK (also thank to
>> >> particular
>> >> >>>>>> successes in my Amb. simulations and actual A. benchmarks). So
>> >> >>>>>> maybe I
>> >> >>>>>> will eventually try downclock, but there might be some another
>> >> >>>>>> variables,
>> >> >>>>>> e.g. driver, OS, motherboard (I will perhaps test one card in
>> >> >>>>>> another
>> >> >>>>>> MB
>> >> >>>>>> just to be sure, that problem is not MB based) etc. that's
>> why I
>> >> >>>>>> asked
>> >> >>>>>> before that guy "ET" for the info about driver version, would
>> be
>> >> >>>>>> also
>> >> >>>>>> interesting OS info or MB.
>> >> >>>>>>
>> >> >>>>>> M.
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>>> Dne Tue, 28 May 2013 22:13:36 +0200 Scott Le Grand
>> >> >>>>>> <varelse2005.gmail.com>
>> >> >>>>>> napsal/-a:
>> >> >>>>>>
>> >> >>>>>> > Marek,
>> >> >>>>>> > Your GPU is hosed. I don't have anything else to add. I'm
>> not
>> >> >>>>>> going
>> >> >>>>>> to
>> >> >>>>>> > go
>> >> >>>>>> > snark hunting for a bug that doesn't exist.
>> >> >>>>>> >
>> >> >>>>>> >
>> >> >>>>>> >
>> >> >>>>>> > On Tue, May 28, 2013 at 12:24 PM, Marek Maly
>> >> <marek.maly.ujep.cz>
>> >> >>>>>> wrote:
>> >> >>>>>> >
>> >> >>>>>> >> Hi, just for the curiosity which driver are you using
>> >> >>>>>> >> on that machine with perfectly working with OC TITAN,
>> >> >>>>>> >> 319.17 or some more actual e.g. 319.23 ?
>> >> >>>>>> >>
>> >> >>>>>> >> RMA is a good idea but it could be also long time story and
>> >> >>>>>> >> also to succeed here you need to have strong arguments
>> >> >>>>>> >> especially if you are going to RMA two OC TITANs.
>> >> >>>>>> >>
>> >> >>>>>> >> I am not sure if my arguments "The cards have problems with
>> >> some
>> >> >>>>>> Amber
>> >> >>>>>> >> calculations"
>> >> >>>>>> >> would be strong enough here. Would be much better to have
>> >> clear
>> >> >>>>>> results
>> >> >>>>>> >> from
>> >> >>>>>> >> respected GPU tests and as it seems you may do extensive
>> GPU
>> >> >>>>>> tests
>> >> >>>>>> also
>> >> >>>>>> >> with
>> >> >>>>>> >> multiple routines without any errors but still have
>> problems
>> >> with
>> >> >>>>>> >> particular
>> >> >>>>>> >> Amber simulations...
>> >> >>>>>> >>
>> >> >>>>>> >> BTW I am now doing Amber benchmarks with nstlim=100K and
>> >> >>>>>> ig=default
>> >> >>>>>> for
>> >> >>>>>> >> each card
>> >> >>>>>> >> twice. The tests will be done in cca 3 hours (due to slow
>> >> >>>>>> nucleosome
>> >> >>>>>> GB
>> >> >>>>>> >> test).
>> >> >>>>>> >>
>> >> >>>>>> >> But even now I have interesting results from the first
>> test on
>> >> >>>>>> GPU0
>> >> >>>>>> >> (nucleosome is still running) see below.
>> >> >>>>>> >>
>> >> >>>>>> >> As you can see JAC_NPT crashed around 11000 step, here is
>> the
>> >> >>>>>> last
>> >> >>>>>> >> md.out
>> >> >>>>>> >> record:
>> >> >>>>>> >>
>> >> >>>>>> >> *********
>> >> >>>>>> >>
>> >> >>>>>> >>
>> >> >>>>>>
>> >> >>>>>>
>> ------------------------------**------------------------------**
>> >> >>>>>> -------------
>> >> >>>>>> -----
>> >> >>>>>> >>
>> >> >>>>>> >> check COM velocity, temp: 0.000021 0.00(Removed)
>> >> >>>>>> >>
>> >> >>>>>> >> NSTEP = 11000 TIME(PS) = 28.000 TEMP(K) =
>> >> 300.39
>> >> >>>>>> PRESS
>> >> >>>>>> >> =
>> >> >>>>>> >> -9.4
>> >> >>>>>> >> Etot = -58092.8958 EKtot = 14440.2520
>> >> EPtot =
>> >> >>>>>> >> -72533.1478
>> >> >>>>>> >> BOND = 443.3912 ANGLE = 1253.5177
>> >> DIHED =
>> >> >>>>>> >> 970.1275
>> >> >>>>>> >> 1-4 NB = 567.2497 1-4 EEL = 6586.9007
>> >> VDWAALS =
>> >> >>>>>> >> 8664.9960
>> >> >>>>>> >> EELEC = -91019.3306 EHBOND = 0.0000
>> >> RESTRAINT =
>> >> >>>>>> >> 0.0000
>> >> >>>>>> >> EKCMT = 6274.0354 VIRIAL = 6321.9969
>> >> VOLUME =
>> >> >>>>>> >> 236141.9494
>> >> >>>>>> >>
>> >> Density =
>> >> >>>>>> >> 1.0162
>> >> >>>>>> >>
>> >> >>>>>> >>
>> >> >>>>>>
>> >> >>>>>>
>> ------------------------------**------------------------------**
>> >> >>>>>> -------------
>> >> >>>>>> -----
>> >> >>>>>> >>
>> >> >>>>>> >> | ERROR: max pairlist cutoff must be less than unit cell
>> max
>> >> >>>>>> sphere
>> >> >>>>>> >> radius!
>> >> >>>>>> >>
>> >> >>>>>> >> ********
>> >> >>>>>> >>
>> >> >>>>>> >> Any idea about that ERROR ?
>> >> >>>>>> >>
>> >> >>>>>> >> On the other hand FACTOR_IX_NPT which has much more atoms
>> >> passed
>> >> >>>>>> >> without
>> >> >>>>>> >> any issue.
>> >> >>>>>> >>
>> >> >>>>>> >> Cellulose crashed on the beginning without any ERROR
>> message
>> >> in
>> >> >>>>>> md.out
>> >> >>>>>> >> file.
>> >> >>>>>> >>
>> >> >>>>>> >>
>> >> >>>>>> >> I am very curious regarding exact reproducibility of the
>> >> results
>> >> >>>>>> at
>> >> >>>>>> >> least
>> >> >>>>>> >> in the
>> >> >>>>>> >> framework of both tests on individual cards.
>> >> >>>>>> >>
>> >> >>>>>> >> BTW regarding eventual downclocking, has anyone idea about
>> >> some
>> >> >>>>>> NVclock
>> >> >>>>>> >> alternative or
>> >> >>>>>> >> I will be really eventually forced to edit frequency value
>> in
>> >> GPU
>> >> >>>>>> BIOS
>> >> >>>>>> >> ?
>> >> >>>>>> >>
>> >> >>>>>> >> Best,
>> >> >>>>>> >>
>> >> >>>>>> >> Marek
>> >> >>>>>> >>
>> >> >>>>>> >> HERE ARE THE FIRST DATA FROM MY 2x2 Bench tests
>> >> >>>>>> >>
>> >> >>>>>> >> JAC_PRODUCTION_NVE - 23,558 atoms PME
>> >> >>>>>> >> ------------------------------**-------
>> >> >>>>>> >>
>> >> >>>>>> >> 1 x GTX_TITAN: | ns/day = 115.91
>> >> >>>>>> seconds/ns =
>> >> >>>>>> >> 745.39
>> >> >>>>>> >>
>> >> >>>>>> >> JAC_PRODUCTION_NPT - 23,558 atoms PME
>> >> >>>>>> >> ------------------------------**-------
>> >> >>>>>> >>
>> >> >>>>>> >> 1 x GTX_TITAN: STOP PMEMD Terminated Abnormally!
>> >> >>>>>> >> | ns/day = 90.72 seconds/ns = 952.42
>> >> >>>>>> >>
>> >> >>>>>> >> FACTOR_IX_PRODUCTION_NVE - 90,906 atoms PME
>> >> >>>>>> >> ------------------------------**-------------
>> >> >>>>>> >>
>> >> >>>>>> >> 1 x GTX_TITAN: | ns/day = 30.56
>> >> >>>>>> seconds/ns =
>> >> >>>>>> >> 2827.33
>> >> >>>>>> >>
>> >> >>>>>> >> FACTOR_IX_PRODUCTION_NPT - 90,906 atoms PME
>> >> >>>>>> >> ------------------------------**-------------
>> >> >>>>>> >>
>> >> >>>>>> >> 1 x GTX_TITAN: | ns/day = 25.01
>> >> >>>>>> seconds/ns =
>> >> >>>>>> >> 3454.56
>> >> >>>>>> >>
>> >> >>>>>> >> CELLULOSE_PRODUCTION_NVE - 408,609 atoms PME
>> >> >>>>>> >> ------------------------------**--------------
>> >> >>>>>> >>
>> >> >>>>>> >> 1 x GTX_TITAN: Error: unspecified launch failure
>> >> >>>>>> launching
>> >> >>>>>> >> kernel
>> >> >>>>>> >> kNLSkinTest
>> >> >>>>>> >> cudaFree GpuBuffer::Deallocate failed unspecified launch
>> >> failure
>> >> >>>>>> >> grep: mdinfo.1GTX_TITAN: No such file or directory
>> >> >>>>>> >>
>> >> >>>>>> >> TRPCAGE_PRODUCTION - 304 atoms GB
>> >> >>>>>> >> ------------------------------**---
>> >> >>>>>> >> 1 x GTX_TITAN: | ns/day = 595.09
>> >> >>>>>> seconds/ns =
>> >> >>>>>> >> 145.19
>> >> >>>>>> >>
>> >> >>>>>> >> MYOGLOBIN_PRODUCTION - 2,492 atoms GB
>> >> >>>>>> >> ------------------------------**-------
>> >> >>>>>> >>
>> >> >>>>>> >> 1 x GTX_TITAN: | ns/day = 202.56
>> >> >>>>>> seconds/ns =
>> >> >>>>>> >> 426.53
>> >> >>>>>> >>
>> >> >>>>>> >> NUCLEOSOME_PRODUCTION - 25,095 atoms GB
>> >> >>>>>> >> ------------------------------**---------
>> >> >>>>>> >>
>> >> >>>>>> >> 1 x GTX_TITAN:
>> >> >>>>>> >>
>> >> >>>>>> >>
>> >> >>>>>> >>
>> >> >>>>>> >>
>> >> >>>>>> >>
>> >> >>>>>> >>
>> >> >>>>>> >>
>> >> >>>>>> >> Dne Tue, 28 May 2013 20:42:32 +0200 ET
>> <sketchfoot.gmail.com>
>> >> >>>>>> napsal/-a:
>> >> >>>>>> >>
>> >> >>>>>> >> > Hi,
>> >> >>>>>> >> >
>> >> >>>>>> >> > I just got a superclocked Titan and one at normal freq.
>> The
>> >> >>>>>> first
>> >> >>>>>> one
>> >> >>>>>> >> ran
>> >> >>>>>> >> > like a charm with no issues so far. The other standard
>> >> clocked
>> >> >>>>>> one
>> >> >>>>>> >> could
>> >> >>>>>> >> > never get past the constant pressure stage in an NPT
>> >> >>>>>> simulation.
>> >> >>>>>> It
>> >> >>>>>> >> kept
>> >> >>>>>> >> > writing NAN or ********* in the outfile. I swapped them
>> >> about
>> >> >>>>>> in
>> >> >>>>>> the
>> >> >>>>>> >> pcie
>> >> >>>>>> >> > lanes then ran it solo in each one of the lanes. Despite
>> all
>> >> >>>>>> this
>> >> >>>>>> it
>> >> >>>>>> >> was
>> >> >>>>>> >> > still failing the benchmark that the other one had no
>> >> problems
>> >> >>>>>> with.
>> >> >>>>>> >> >
>> >> >>>>>> >> > I couldn't find any memory errors with GPU-burn either,
>> but
>> >> as
>> >> >>>>>> they
>> >> >>>>>> >> cost
>> >> >>>>>> >> > near a grand a piece, I RMA'd it today. I recommend you
>> to
>> >> do
>> >> >>>>>> the
>> >> >>>>>> >> same if
>> >> >>>>>> >> > its not giving you any joy. Life's too short. :)
>> >> >>>>>> >> >
>> >> >>>>>> >> > br,
>> >> >>>>>> >> > g
>> >> >>>>>> >> >
>> >> >>>>>> >> >
>> >> >>>>>> >> > On 28 May 2013 16:57, Scott Le Grand
>> <varelse2005.gmail.com
>> >
>> >> >>>>>> wrote:
>> >> >>>>>> >> >
>> >> >>>>>> >> >> AMBER != NAMD...
>> >> >>>>>> >> >>
>> >> >>>>>> >> >> GTX 680 != GTX Titan...
>> >> >>>>>> >> >>
>> >> >>>>>> >> >> Ian's suggestion is a good one. But even then, you
>> need to
>> >> >>>>>> test
>> >> >>>>>> >> >> your
>> >> >>>>>> >> >> GPUs
>> >> >>>>>> >> >> as the Titans are running right on the edge of
>> stability.
>> >> >>>>>> Like I
>> >> >>>>>> >> told
>> >> >>>>>> >> >> Marek, try running 100K iterations of Cellulose NVE
>> twice
>> >> with
>> >> >>>>>> the
>> >> >>>>>> >> same
>> >> >>>>>> >> >> random seed. if you don't get identically bit accurate
>> >> >>>>>> output,
>> >> >>>>>> your
>> >> >>>>>> >> >> GPU is
>> >> >>>>>> >> >> not working. Memtest programs do not catch this because
>> >> (I am
>> >> >>>>>> >> guessing)
>> >> >>>>>> >> >> they are designed for a uniform memory hierarchy and
>> only
>> >> one
>> >> >>>>>> path
>> >> >>>>>> >> >> to
>> >> >>>>>> >> >> read
>> >> >>>>>> >> >> and write data. I have a stock GTX Titan that cannot
>> pass
>> >> the
>> >> >>>>>> >> Cellulose
>> >> >>>>>> >> >> NVE test and another one that does. I spent a couple
>> days
>> >> on
>> >> >>>>>> the
>> >> >>>>>> >> former
>> >> >>>>>> >> >> GPU looking for the imaginary bug that went away like
>> magic
>> >> >>>>>> the
>> >> >>>>>> >> second I
>> >> >>>>>> >> >> switched out the GPU.
>> >> >>>>>> >> >>
>> >> >>>>>> >> >> Scott
>> >> >>>>>> >> >>
>> >> >>>>>> >> >>
>> >> >>>>>> >> >>
>> >> >>>>>> >> >>
>> >> >>>>>> >> >>
>> >> >>>>>> >> >> On Tue, May 28, 2013 at 8:11 AM, Robert Konecny
>> >> <rok.ucsd.edu
>> >> >
>> >> >>>>>> wrote:
>> >> >>>>>> >> >>
>> >> >>>>>> >> >> > Hi Scott,
>> >> >>>>>> >> >> >
>> >> >>>>>> >> >> > unfortunately we are seeing similar Amber instability
>> on
>> >> GTX
>> >> >>>>>> >> Titans as
>> >> >>>>>> >> >> > Marek is. We have a box with four GTX Titans (not
>> >> >>>>>> oveclocked)
>> >> >>>>>> >> running
>> >> >>>>>> >> >> > CentOS 6.3 with NVidia 319.17 driver and Amber 12.2.
>> Any
>> >> >>>>>> Amber
>> >> >>>>>> >> >> simulation
>> >> >>>>>> >> >> > longer than 10-15 min eventually crashes on these
>> cards,
>> >> >>>>>> including
>> >> >>>>>> >> >> both
>> >> >>>>>> >> >> JAC
>> >> >>>>>> >> >> > benchmarks (with extended run time). This is
>> >> reproducible on
>> >> >>>>>> all
>> >> >>>>>> >> four
>> >> >>>>>> >> >> > cards.
>> >> >>>>>> >> >> >
>> >> >>>>>> >> >> > To eliminate the possible hardware error we ran
>> extended
>> >> GPU
>> >> >>>>>> >> >> > memory
>> >> >>>>>> >> >> tests
>> >> >>>>>> >> >> > on all four Titans with memtestG80, cuda_memtest and
>> also
>> >> >>>>>> gpu_burn
>> >> >>>>>> >> -
>> >> >>>>>> >> >> all
>> >> >>>>>> >> >> > finished without errors. Since I agree that these
>> >> programs
>> >> >>>>>> may
>> >> >>>>>> not
>> >> >>>>>> >> >> test
>> >> >>>>>> >> >> the
>> >> >>>>>> >> >> > GPU completely we also set up simulations with NAMD.
>> We
>> >> can
>> >> >>>>>> run
>> >> >>>>>> >> four
>> >> >>>>>> >> >> NAMD
>> >> >>>>>> >> >> > simulations simultaneously for many days without any
>> >> errors
>> >> >>>>>> on
>> >> >>>>>> >> >> > this
>> >> >>>>>> >> >> > hardware. For reference - we also have exactly the
>> same
>> >> >>>>>> server
>> >> >>>>>> >> >> > with
>> >> >>>>>> >> >> the
>> >> >>>>>> >> >> > same hardware components but with four GTX680s and
>> this
>> >> >>>>>> setup
>> >> >>>>>> >> >> > works
>> >> >>>>>> >> >> just
>> >> >>>>>> >> >> > fine for Amber. So all this leads me to believe that a
>> >> >>>>>> hardware
>> >> >>>>>> >> error
>> >> >>>>>> >> >> is
>> >> >>>>>> >> >> > not very likely.
>> >> >>>>>> >> >> >
>> >> >>>>>> >> >> > I would appreciate your comments on this, perhaps
>> there
>> >> is
>> >> >>>>>> >> something
>> >> >>>>>> >> >> else
>> >> >>>>>> >> >> > causing these errors which we are not seeing.
>> >> >>>>>> >> >> >
>> >> >>>>>> >> >> > Thanks,
>> >> >>>>>> >> >> >
>> >> >>>>>> >> >> > Robert
>> >> >>>>>> >> >> >
>> >> >>>>>> >> >> >
>> >> >>>>>> >> >> > On Mon, May 27, 2013 at 04:25:24PM -0700, Scott Le
>> Grand
>> >> >>>>>> wrote:
>> >> >>>>>> >> >> > > I have two GTX Titans. One is defective, the other
>> is
>> >> >>>>>> not.
>> >> >>>>>> >> >> > Unfortunately,
>> >> >>>>>> >> >> > > they both pass all standard GPU memory tests.
>> >> >>>>>> >> >> > >
>> >> >>>>>> >> >> > > What the defective one doesn't do is generate
>> >> reproducibly
>> >> >>>>>> >> >> bit-accurate
>> >> >>>>>> >> >> > > outputs for simulations of Factor IX (90,986 atoms)
>> or
>> >> >>>>>> larger,
>> >> >>>>>> >> >> > > of
>> >> >>>>>> >> >> 100K
>> >> >>>>>> >> >> or
>> >> >>>>>> >> >> > > so iterations.
>> >> >>>>>> >> >> > >
>> >> >>>>>> >> >> > > Which is yet another reason why I insist on MD
>> >> algorithms
>> >> >>>>>> >> >> (especially
>> >> >>>>>> >> >> on
>> >> >>>>>> >> >> > > GPUS) being deterministic. Besides its ability to
>> find
>> >> >>>>>> software
>> >> >>>>>> >> >> bugs,
>> >> >>>>>> >> >> > and
>> >> >>>>>> >> >> > > fulfilling one of the most important tenets of
>> science,
>> >> >>>>>> it's
>> >> >>>>>> a
>> >> >>>>>> >> great
>> >> >>>>>> >> >> way
>> >> >>>>>> >> >> > to
>> >> >>>>>> >> >> > > diagnose defective hardware with very little effort.
>> >> >>>>>> >> >> > >
>> >> >>>>>> >> >> > > 928 MHz? That's 6% above the boost clock of a stock
>> >> >>>>>> Titan.
>> >> >>>>>> >> Titan
>> >> >>>>>> >> >> is
>> >> >>>>>> >> >> > > pushing the performance envelope as is. If you're
>> >> going
>> >> >>>>>> to
>> >> >>>>>> pay
>> >> >>>>>> >> the
>> >> >>>>>> >> >> > premium
>> >> >>>>>> >> >> > > for such chips, I'd send them back until you get one
>> >> that
>> >> >>>>>> runs
>> >> >>>>>> >> >> correctly.
>> >> >>>>>> >> >> > > I'm very curious how fast you can push one of these
>> >> things
>> >> >>>>>> >> >> > > before
>> >> >>>>>> >> >> they
>> >> >>>>>> >> >> > give
>> >> >>>>>> >> >> > > out.
>> >> >>>>>> >> >> > >
>> >> >>>>>> >> >> > >
>> >> >>>>>> >> >> > >
>> >> >>>>>> >> >> > >
>> >> >>>>>> >> >> > >
>> >> >>>>>> >> >> > >
>> >> >>>>>> >> >> > >
>> >> >>>>>> >> >> > > On Mon, May 27, 2013 at 10:01 AM, Marek Maly
>> >> >>>>>> <marek.maly.ujep.cz
>> >> >>>>>> >
>> >> >>>>>> >> >> wrote:
>> >> >>>>>> >> >> > >
>> >> >>>>>> >> >> > > > Dear all,
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > I have recently bought two "EVGA GTX TITAN
>> >> Superclocked"
>> >> >>>>>> GPUs.
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > I did the first calculations (pmemd.cuda in
>> Amber12)
>> >> >>>>>> with
>> >> >>>>>> >> systems
>> >> >>>>>> >> >> > around
>> >> >>>>>> >> >> > > > 60K atoms without any problems (NPT, Langevin),
>> but
>> >> >>>>>> when I
>> >> >>>>>> >> later
>> >> >>>>>> >> >> tried
>> >> >>>>>> >> >> > > > with bigger systems (around 100K atoms) I obtained
>> >> >>>>>> "classical"
>> >> >>>>>> >> >> > irritating
>> >> >>>>>> >> >> > > > errors
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > cudaMemcpy GpuBuffer::Download failed unspecified
>> >> launch
>> >> >>>>>> >> failure
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > just after few thousands of MD steps.
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > So this was obviously the reason for memtestG80
>> >> tests.
>> >> >>>>>> >> >> > > > ( https://simtk.org/home/memtest ).
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > So I compiled memtestG80 from sources (
>> >> >>>>>> >> memtestG80-1.1-src.tar.gz
>> >> >>>>>> >> >> )
>> >> >>>>>> >> >> and
>> >> >>>>>> >> >> > > > then tested
>> >> >>>>>> >> >> > > > just small part of memory GPU (200 MB) using 100
>> >> >>>>>> iterations.
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > On both cards I have obtained huge amount of
>> errors
>> >> but
>> >> >>>>>> "just"
>> >> >>>>>> >> on
>> >> >>>>>> >> >> > > > "Random blocks:". 0 errors in all remaining tests
>> in
>> >> all
>> >> >>>>>> >> >> iterations.
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > ------THE LAST ITERATION AND FINAL RESULTS-------
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > Test iteration 100 (GPU 0, 200 MiB): 169736847
>> >> errors so
>> >> >>>>>> far
>> >> >>>>>> >> >> > > > Moving Inversions (ones and zeros): 0
>> errors
>> >> (6
>> >> >>>>>> ms)
>> >> >>>>>> >> >> > > > Memtest86 Walking 8-bit: 0 errors (53 ms)
>> >> >>>>>> >> >> > > > True Walking zeros (8-bit): 0 errors (26
>> ms)
>> >> >>>>>> >> >> > > > True Walking ones (8-bit): 0 errors (26
>> ms)
>> >> >>>>>> >> >> > > > Moving Inversions (random): 0 errors (6
>> ms)
>> >> >>>>>> >> >> > > > Memtest86 Walking zeros (32-bit): 0 errors
>> >> (105
>> >> >>>>>> ms)
>> >> >>>>>> >> >> > > > Memtest86 Walking ones (32-bit): 0 errors
>> >> (104
>> >> >>>>>> ms)
>> >> >>>>>> >> >> > > > Random blocks: 1369863 errors (27 ms)
>> >> >>>>>> >> >> > > > Memtest86 Modulo-20: 0 errors (215 ms)
>> >> >>>>>> >> >> > > > Logic (one iteration): 0 errors (4 ms)
>> >> >>>>>> >> >> > > > Logic (4 iterations): 0 errors (8 ms)
>> >> >>>>>> >> >> > > > Logic (shared memory, one iteration): 0
>> >> errors
>> >> >>>>>> (8
>> >> >>>>>> ms)
>> >> >>>>>> >> >> > > > Logic (shared-memory, 4 iterations): 0
>> errors
>> >> >>>>>> (25
>> >> >>>>>> ms)
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > Final error count after 100 iterations over 200
>> MiB
>> >> of
>> >> >>>>>> GPU
>> >> >>>>>> >> memory:
>> >> >>>>>> >> >> > > > 171106710 errors
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > ------------------------------**------------
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > I have some questions and would be really grateful
>> >> for
>> >> >>>>>> any
>> >> >>>>>> >> >> comments.
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > Regarding overclocking, using the deviceQuery I
>> found
>> >> >>>>>> out
>> >> >>>>>> that
>> >> >>>>>> >> >> under
>> >> >>>>>> >> >> > linux
>> >> >>>>>> >> >> > > > both cards run
>> >> >>>>>> >> >> > > > automatically using boost shader/GPU frequency
>> which
>> >> is
>> >> >>>>>> here
>> >> >>>>>> >> 928
>> >> >>>>>> >> >> MHz
>> >> >>>>>> >> >> > (the
>> >> >>>>>> >> >> > > > basic value for these factory OC cards is 876
>> MHz).
>> >> >>>>>> >> >> > > > deviceQuery
>> >> >>>>>> >> >> > reported
>> >> >>>>>> >> >> > > > Memory Clock rate is 3004 MHz although "it"
>> should be
>> >> >>>>>> 6008
>> >> >>>>>> MHz
>> >> >>>>>> >> but
>> >> >>>>>> >> >> > maybe
>> >> >>>>>> >> >> > > > the quantity which is reported by deviceQuery
>> "Memory
>> >> >>>>>> Clock
>> >> >>>>>> >> rate"
>> >> >>>>>> >> >> is
>> >> >>>>>> >> >> > > > different from the product specification "Memory
>> >> Clock"
>> >> >>>>>> .
>> >> >>>>>> It
>> >> >>>>>> >> seems
>> >> >>>>>> >> >> that
>> >> >>>>>> >> >> > > > "Memory Clock rate" = "Memory Clock"/2. Am I
>> right ?
>> >> Or
>> >> >>>>>> just
>> >> >>>>>> >> >> > deviceQuery
>> >> >>>>>> >> >> > > > is not able to read this spec. properly
>> >> >>>>>> >> >> > > > in Titan GPU ?
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > Anyway for the moment I assume that the problem
>> >> might be
>> >> >>>>>> due
>> >> >>>>>> >> >> > > > to
>> >> >>>>>> >> >> the
>> >> >>>>>> >> >> > high
>> >> >>>>>> >> >> > > > shader/GPU frequency.
>> >> >>>>>> >> >> > > > (see here :
>> >> >>>>>> http://folding.stanford.edu/**English/DownloadUtils<
>> >> http://folding.stanford.edu/English/DownloadUtils>
>> >> >>>>>> )
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > To verify this hypothesis one should perhaps
>> >> UNDERclock
>> >> >>>>>> to
>> >> >>>>>> >> basic
>> >> >>>>>> >> >> > frequency
>> >> >>>>>> >> >> > > > which is in this
>> >> >>>>>> >> >> > > > model 876 MHz or even to the TITAN REFERENCE
>> >> frequency
>> >> >>>>>> which
>> >> >>>>>> >> >> > > > is
>> >> >>>>>> >> >> 837
>> >> >>>>>> >> >> > MHz.
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > Obviously I am working with these cards under
>> linux
>> >> >>>>>> (CentOS
>> >> >>>>>> >> >> > > > 2.6.32-358.6.1.el6.x86_64) and as I found, the OC
>> >> tools
>> >> >>>>>> under
>> >> >>>>>> >> >> linux
>> >> >>>>>> >> >> > are in
>> >> >>>>>> >> >> > > > fact limited just to NVclock utility, which is
>> >> >>>>>> unfortunately
>> >> >>>>>> >> >> > > > out of date (at least speaking about the GTX Titan
>> >> ). I
>> >> >>>>>> have
>> >> >>>>>> >> >> obtained
>> >> >>>>>> >> >> > this
>> >> >>>>>> >> >> > > > message when I wanted
>> >> >>>>>> >> >> > > > just to let NVclock utility to read and print
>> shader
>> >> and
>> >> >>>>>> >> >> > > > memory
>> >> >>>>>> >> >> > > > frequencies of my Titan's:
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >>
>> >> >>>>>>
>> ------------------------------**------------------------------**
>> >> >>>>>> -------
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > [root.dyn-138-272 NVCLOCK]# nvclock -s --speeds
>> >> >>>>>> >> >> > > > Card: Unknown Nvidia card
>> >> >>>>>> >> >> > > > Card number: 1
>> >> >>>>>> >> >> > > > Memory clock: -2147483.750 MHz
>> >> >>>>>> >> >> > > > GPU clock: -2147483.750 MHz
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > Card: Unknown Nvidia card
>> >> >>>>>> >> >> > > > Card number: 2
>> >> >>>>>> >> >> > > > Memory clock: -2147483.750 MHz
>> >> >>>>>> >> >> > > > GPU clock: -2147483.750 MHz
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >>
>> >> >>>>>>
>> ------------------------------**------------------------------**
>> >> >>>>>> -------
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > I would be really grateful for some tips regarding
>> >> >>>>>> "NVclock
>> >> >>>>>> >> >> > alternatives",
>> >> >>>>>> >> >> > > > but after wasting some hours with googling it
>> seems
>> >> that
>> >> >>>>>> there
>> >> >>>>>> >> is
>> >> >>>>>> >> >> no
>> >> >>>>>> >> >> > other
>> >> >>>>>> >> >> > > > Linux
>> >> >>>>>> >> >> > > > tool with NVclock functionality. So the only
>> >> >>>>>> possibility is
>> >> >>>>>> >> here
>> >> >>>>>> >> >> > perhaps
>> >> >>>>>> >> >> > > > to edit
>> >> >>>>>> >> >> > > > GPU bios with some Lin/DOS/Win tools like (Kepler
>> >> BIOS
>> >> >>>>>> >> >> > > > Tweaker,
>> >> >>>>>> >> >> > NVflash)
>> >> >>>>>> >> >> > > > but obviously
>> >> >>>>>> >> >> > > > I would like to rather avoid such approach as
>> using
>> >> it
>> >> >>>>>> means
>> >> >>>>>> >> >> perhaps
>> >> >>>>>> >> >> > also
>> >> >>>>>> >> >> > > > to void the warranty even if I am going to
>> underclock
>> >> >>>>>> the
>> >> >>>>>> GPUs
>> >> >>>>>> >> >> not to
>> >> >>>>>> >> >> > > > overclock them.
>> >> >>>>>> >> >> > > > So before this eventual step (GPU bios editing) I
>> >> would
>> >> >>>>>> like
>> >> >>>>>> >> >> > > > to
>> >> >>>>>> >> >> have
>> >> >>>>>> >> >> > some
>> >> >>>>>> >> >> > > > approximative estimate
>> >> >>>>>> >> >> > > > of the probability, that the problems are here
>> really
>> >> >>>>>> because
>> >> >>>>>> >> of
>> >> >>>>>> >> >> the
>> >> >>>>>> >> >> > > > overclocking
>> >> >>>>>> >> >> > > > (too high (boost) default shader frequency).
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > This probability I hope to estimate from the
>> eventual
>> >> >>>>>> >> responses of
>> >> >>>>>> >> >> > another
>> >> >>>>>> >> >> > > > Amber/Titan SC users, if I am not the only crazy
>> guy
>> >> who
>> >> >>>>>> >> >> > > > bought
>> >> >>>>>> >> >> this
>> >> >>>>>> >> >> > model
>> >> >>>>>> >> >> > > > for Amber calculations :)) But of course any
>> eventual
>> >> >>>>>> >> experiences
>> >> >>>>>> >> >> with
>> >> >>>>>> >> >> > > > Titan cards related to their memtestG80 results
>> and
>> >> >>>>>> >> >> UNDER/OVERclocking
>> >> >>>>>> >> >> > > > (if possible in Linux OS) are of course welcomed
>> as
>> >> >>>>>> well !
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > My HW/SW configuration
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > motherboard: ASUS P9X79 PRO
>> >> >>>>>> >> >> > > > CPU: Intel Core i7-3930K
>> >> >>>>>> >> >> > > > RAM: CRUCIAL Ballistix Sport 32GB (4x8GB) DDR3
>> 1600
>> >> VLP
>> >> >>>>>> >> >> > > > CASE: CoolerMaster Dominator CM-690 II Advanced,
>> >> >>>>>> >> >> > > > Power:Enermax PLATIMAX EPM1200EWT 1200W, 80+,
>> >> Platinum
>> >> >>>>>> >> >> > > > GPUs : 2 x EVGA GTX TITAN Superclocked 6GB
>> >> >>>>>> >> >> > > > cooler: Cooler Master Hyper 412 SLIM
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > OS: CentOS (2.6.32-358.6.1.el6.x86_64)
>> >> >>>>>> >> >> > > > driver version: 319.17
>> >> >>>>>> >> >> > > > cudatoolkit_5.0.35_linux_64_**rhel6.x
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > The computer is in air-conditioned room with
>> >> permanent
>> >> >>>>>> >> >> > > > external
>> >> >>>>>> >> >> > > > temperature around 18°C
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > Thanks a lot in advance for any
>> >> comment/experience !
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > Best wishes,
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > Marek
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > --
>> >> >>>>>> >> >> > > > Tato zpráva byla vytvořena převratným poštovním
>> >> klientem
>> >> >>>>>> >> >> > > > Opery:
>> >> >>>>>> >> >> > > > http://www.opera.com/mail/
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > > ______________________________**_________________
>> >> >>>>>> >> >> > > > AMBER mailing list
>> >> >>>>>> >> >> > > > AMBER.ambermd.org
>> >> >>>>>> >> >> > > >
>> >> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> >> http://lists.ambermd.org/mailman/listinfo/amber>
>> >> >>>>>> >> >> > > >
>> >> >>>>>> >> >> > > ______________________________**_________________
>> >> >>>>>> >> >> > > AMBER mailing list
>> >> >>>>>> >> >> > > AMBER.ambermd.org
>> >> >>>>>> >> >> > >
>> >> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> >> http://lists.ambermd.org/mailman/listinfo/amber>
>> >> >>>>>> >> >> >
>> >> >>>>>> >> >> > ______________________________**_________________
>> >> >>>>>> >> >> > AMBER mailing list
>> >> >>>>>> >> >> > AMBER.ambermd.org
>> >> >>>>>> >> >> >
>> >> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> >> http://lists.ambermd.org/mailman/listinfo/amber>
>> >> >>>>>> >> >> >
>> >> >>>>>> >> >> ______________________________**_________________
>> >> >>>>>> >> >> AMBER mailing list
>> >> >>>>>> >> >> AMBER.ambermd.org
>> >> >>>>>> >> >>
>> >> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> >> http://lists.ambermd.org/mailman/listinfo/amber>
>> >> >>>>>> >> >>
>> >> >>>>>> >> > ______________________________**_________________
>> >> >>>>>> >> > AMBER mailing list
>> >> >>>>>> >> > AMBER.ambermd.org
>> >> >>>>>> >> >
>> >> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> >> http://lists.ambermd.org/mailman/listinfo/amber>
>> >> >>>>>> >> >
>> >> >>>>>> >> > __________ Informace od ESET NOD32 Antivirus, verze
>> databaze
>> >> >>>>>> 8385
>> >> >>>>>> >> > (20130528) __________
>> >> >>>>>> >> >
>> >> >>>>>> >> > Tuto zpravu proveril ESET NOD32 Antivirus.
>> >> >>>>>> >> >
>> >> >>>>>> >> > http://www.eset.cz
>> >> >>>>>> >> >
>> >> >>>>>> >> >
>> >> >>>>>> >> >
>> >> >>>>>> >>
>> >> >>>>>> >>
>> >> >>>>>> >> --
>> >> >>>>>> >> Tato zpráva byla vytvořena převratným poštovním klientem
>> >> Opery:
>> >> >>>>>> >> http://www.opera.com/mail/
>> >> >>>>>> >>
>> >> >>>>>> >> ______________________________**_________________
>> >> >>>>>> >> AMBER mailing list
>> >> >>>>>> >> AMBER.ambermd.org
>> >> >>>>>> >>
>> >> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> >> http://lists.ambermd.org/mailman/listinfo/amber>
>> >> >>>>>> >>
>> >> >>>>>> > ______________________________**_________________
>> >> >>>>>> > AMBER mailing list
>> >> >>>>>> > AMBER.ambermd.org
>> >> >>>>>> >
>> >> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> >> http://lists.ambermd.org/mailman/listinfo/amber>
>> >> >>>>>> >
>> >> >>>>>> > __________ Informace od ESET NOD32 Antivirus, verze databaze
>> >> 8386
>> >> >>>>>> > (20130528) __________
>> >> >>>>>> >
>> >> >>>>>> > Tuto zpravu proveril ESET NOD32 Antivirus.
>> >> >>>>>> >
>> >> >>>>>> > http://www.eset.cz
>> >> >>>>>> >
>> >> >>>>>> >
>> >> >>>>>> >
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>>> --
>> >> >>>>>> Tato zpráva byla vytvořena převratným poštovním klientem
>> Opery:
>> >> >>>>>> http://www.opera.com/mail/
>> >> >>>>>>
>> >> >>>>>> ______________________________**_________________
>> >> >>>>>> AMBER mailing list
>> >> >>>>>> AMBER.ambermd.org
>> >> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> >> http://lists.ambermd.org/mailman/listinfo/amber>
>> >> >>>>>>
>> >> >>>>>> ______________________________**_________________
>> >> >>>>> AMBER mailing list
>> >> >>>>> AMBER.ambermd.org
>> >> >>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> >> http://lists.ambermd.org/mailman/listinfo/amber>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> ______________________________**_________________
>> >> >>>>> AMBER mailing list
>> >> >>>>> AMBER.ambermd.org
>> >> >>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> >> http://lists.ambermd.org/mailman/listinfo/amber>
>> >> >>>>>
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>> ______________________________**_________________
>> >> >>>> AMBER mailing list
>> >> >>>> AMBER.ambermd.org
>> >> >>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> >> http://lists.ambermd.org/mailman/listinfo/amber>
>> >> >>>>
>> >> >>>> __________ Informace od ESET NOD32 Antivirus, verze databaze
>> 8386
>> >> >>>> (20130528) __________
>> >> >>>>
>> >> >>>> Tuto zpravu proveril ESET NOD32 Antivirus.
>> >> >>>>
>> >> >>>> http://www.eset.cz
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>
>> >> >>> --
>> >> >>> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
>> >> >>> http://www.opera.com/mail/
>> >> >>> _______________________________________________
>> >> >>> AMBER mailing list
>> >> >>> AMBER.ambermd.org
>> >> >>> http://lists.ambermd.org/mailman/listinfo/amber
>> >> >>>
>> >> >>>
>> >> > _______________________________________________
>> >> > AMBER mailing list
>> >> > AMBER.ambermd.org
>> >> > http://lists.ambermd.org/mailman/listinfo/amber
>> >> >
>> >> > __________ Informace od ESET NOD32 Antivirus, verze databaze 8390
>> >> > (20130529) __________
>> >> >
>> >> > Tuto zpravu proveril ESET NOD32 Antivirus.
>> >> >
>> >> > http://www.eset.cz
>> >> >
>> >> >
>> >> >
>> >>
>> >>
>> >> --
>> >> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
>> >> http://www.opera.com/mail/
>> >>
>> >> _______________________________________________
>> >> AMBER mailing list
>> >> AMBER.ambermd.org
>> >> http://lists.ambermd.org/mailman/listinfo/amber
>> >>
>> > _______________________________________________
>> > AMBER mailing list
>> > AMBER.ambermd.org
>> > http://lists.ambermd.org/mailman/listinfo/amber
>> >
>> > __________ Informace od ESET NOD32 Antivirus, verze databaze 8390
>> > (20130529) __________
>> >
>> > Tuto zpravu proveril ESET NOD32 Antivirus.
>> >
>> > http://www.eset.cz
>> >
>> >
>> >
>>
>>
>> --
>> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
>> http://www.opera.com/mail/
>>
>> _______________________________________________
>> AMBER mailing list
>> AMBER.ambermd.org
>> http://lists.ambermd.org/mailman/listinfo/amber
>>
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
>
> __________ Informace od ESET NOD32 Antivirus, verze databaze 8390
> (20130529) __________
>
> Tuto zpravu proveril ESET NOD32 Antivirus.
>
> http://www.eset.cz
>
>
>
--
Tato zpráva byla vytvořena převratným poštovním klientem Opery:
http://www.opera.com/mail/
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Wed May 29 2013 - 19:00:02 PDT
