Re: [AMBER] GTX Titan was finally released

From: filip fratev <filipfratev.yahoo.com>
Date: Sun, 17 Mar 2013 06:53:32 -0700 (PDT)

Hi Ross,


I am glad to hear that you got
your Titan's. Thanks for your detail comments!
 
I follow your advice and I have
never overclocked any of my GPU's under Linux or used factory overclocked card,
which is in fact impossible if you not hack your Bios. Thus if you are my
future reviewer, please don't reject my papers :))
However, I overclocked my CPU to
4.6Ghz just to have better scale during the 2xGPU's test. Hope that this is not
a problem. I observed about 2ns difference during the Factor IX bench. 
 
I agree with you that the speed
under Linux is correct in terms of  the
provided by Nvidia characteristics. I wanted just to inform the readers and to
point out that for this GPU the base and boost speed have different conception.
Please look this:
http://imageshack.us/photo/my-images/690/titanspped.png/
As you can see, my card is not overclocked
and the reported Base and Boost speeds are 837Mhz and 876Mhz, respectively. However
your GPU works always on 993MHz under load (windows). This is the boost 2.0
speed, which is unfortunately not supported under Linux, according to Nvidia.
In other words the Titan was made to work on 993Mhz speed (default) and this is
the benefit from boost 2.0. I don't now how EVGA Titan S performs but if you
clock the regular Titan card to 1Ghz, your actual speed is 1176Mhz as you can
see from this picture:
http://imageshack.us/photo/my-images/26/titansppedcloked.png/

I hope that this report is
informative.
   
Anyway this card is really
awesome and I personally recommend it to the Amber users.
 
All the best,
Filip


________________________________
 From: Ross Walker <ross.rosswalker.co.uk>
To: filip fratev <filipfratev.yahoo.com>; AMBER Mailing List <amber.ambermd.org>
Sent: Sunday, March 17, 2013 3:08 AM
Subject: Re: [AMBER] GTX Titan was finally released
 
Hi Filip,

Thanks for the feedback on the GTX Titan - these numbers are awesome. I
finally got some of mine own as well now so will be updating the webpage
soon and will look at some tuning options. We've experimented with the DP
mode and found that it slows things down by about 10% or so hence you are
best leaving the Titan in gaming mode (which is the default right now
given the DP CUDA mode is undocumented at present).

A note to everyone though - if you find your Titan running slower than you
expect it probably means some kind soul put it in DP mode for you.

With regards to clock speed this is definitely just a cosmetic issue in
mdout - we are looking at ways to report the clock speed in a better way -
right now you are just getting the base clock reported but the card
 is
ramping up properly. An example

K20X : cores = 2688, SP clock speed = (3.95TF * 1000) / 2688 / 2 = 732MHz
(http://www.nvidia.com/object/tesla-servers.html)

K20X performance for JAC / DHFR NVE = 89.13 NS/day

GTX Titan = 110.65 ns/day

(110.65 / 89.13) * 732 = 908 MHz.

So you are getting the full speed under Linux.

http://www.evga.com/Products/ProductList.aspx?type=0&family=GeForce+TITAN+S
eries+Family&chipset=GTX+TITAN

EVGA GeForce GTX TITAN = 837MHz base, 876 MHz Boost

EVGA GeForce GTX TITAN Superclocked = 876 MHz base, 928MHz Boost

So the back of the envelope calculation of 908MHz is right in the range
and the boost clock is being activated under Linux
 correctly.

Not sure how
 you are getting 1150MHz reported under windows since the
published boost numbers are 928MHz. Are you manually over clocking?

Given you report your CPU speed at 4.6GHz when the I7-3770K is only rated
at 3.4Ghz (3.9 GHz boost) http://ark.intel.com/products/65719/ I would
guess that the answer is yes and you are over clocking your CPU as well.

All I can say is good luck, probably best to include this in the methods
section of any papers you publish.

All the best
Ross


On 3/14/13 9:33 AM, "filip fratev" <filipfratev.yahoo.com> wrote:

>Hi Ross and all,
>
>These are my test results for GTX Titan. Just great gpu! My results
>differ 1-3% from those obtained by Ian. I am not sure why. First
 of all
>Titan is a cold card.  When
 I set the fan speed to only 70-75%, the
>temperature never goes above 60-65C. It is a pity that the card clock is
>only 876Mhz. Under Windows I was not able to heat the card above 74 C and
>the speed was 1150-1170Mhz, i.e. under Windows the single precision speed
>and the boost speed are equal to the so called maximal clock speed. Many
>folks have already hacked their bioses. Anyway..
>
>I use i7 3770K.4.6Ghz, GB GTX-Titan and my RAM is clocked above 2400Mhz.
>OS was Suse 12.1+cuda 5.0.
>
>----------------------------------
>DHFR NVE = 23,558 atoms
>
>1xGTX Titan = 110.65 ns/day
>2xGTX Titan = 125.28 ns/day
>
>--------------------------------------
>DHFR NPT = 23,558 atoms
>
>1xGTX Titan = 85.27 ns/day
>2xGTX Titan = 101.88 ns/day
>
>---------------------------------------
>FactorIX NVE = 90,906
 atoms
>
>1xGTX Titan = 31.55 ns/day
>2xGTX Titan = 38.05 ns/day
>
>----------------------------------
>FactorIX  NPT = 90,906 atoms
>
>1xGTX Titan = 25.85 ns/day
>2xGTX Titan = 32.54 ns/day
>
>----------------------------------------
>Cellulose NVE = 408,609 atoms
>
>1xGTX Titan = 7.50 ns/day
>2xGTX Titan = 8.72 ns/day
>
>----------------------------------------
>Cellulose NPT = 408,609 atoms
>
>1xGTX Titan = 6.31 ns/day
>2xGTX Titan = 7.71 ns/day
>
>-------------------------------------------
>
>All the best,
>Filip
>
>
>
>________________________________
> From: filip fratev <filipfratev.yahoo.com>
>To: AMBER Mailing List <amber.ambermd.org>
>Sent: Wednesday, February 27, 2013 9:21 PM
>Subject: Re: [AMBER] GTX Titan was finally released
>
>Hi Marek,
>
>
>Thanks for sharing your benchmarks! Also thanks to Ross and Scott!
>
>8-10% is not insignificant difference considering that the difference
>between one two GPU's are 14% in JAC NPT. At least for me:)
>
>The EVGA revealed the Titan clock speed for their  superclocked version-
>876Mhz, i.e. noting intriguing.
>
>
>Ross you mentioned:
>>>Firstly you are referring to the double precision clock rate and not
>>>the single precision clock.
>
>What will be the single precision clock?
>
>
>All the best,
>Filip
>
>P.S. The GPU Boost 2.0 is different than
 the CPU boost. Your GPU can be
>under 100% load but it will
 still work on the boost clock (876Mhz, single
>precision?) and under Windows on nearly 1Ghz and this will be changeed
>only if your temperature is above 80C.Thus if one use a water cooling
>under Windows will be able to use Titan on 1Ghz which is around 10-15% +
>in the performance.
>
>
>
>________________________________
>From: Marek Maly <marek.maly.ujep.cz>
>To: AMBER Mailing List <amber.ambermd.org>
>Sent: Wednesday, February 27, 2013 8:23 PM
>Subject: Re: [AMBER] GTX Titan was finally released
>
>Hi Guys,
>here are finally the results of in factory over clocked GTX680
>( EVGA GeForce GTX680 Classified ) in combination with "ASUS P9X79
 PRO"
>motherboard.
>
>As one can see the
 increase from the reference 1006MHz to 1111MHz make
>just
>a small difference in results (reflecting percentually more or less the
>difference
>in frquency ). I did not test it in Boost clock (1176MHz) and I am not
>going to do it, as for the long MD runs this regime seems to me a bit
>dangerous :))
>
>Regarding the reliability of this OC version, I am fully satisfied,
>already
>tested 2 of these in few weeks simulations.
>
>  Best wishes,
>
>          Marek
>
>
>JAC_PRODUCTION_NVE - 23,558 atoms PME
>-------------------------------------
>
>        1 x GTX680: |        ns/day =      80.38  seconds/ns =
>1074.90
>
>JAC_PRODUCTION_NPT - 23,558 atoms PME
>-------------------------------------
>
>   
 Â    1 x GTX680: |        ns/day =      64.59  seconds/ns =
>1337.60
>
>FACTOR_IX_PRODUCTION_NVE - 90,906 atoms PME
>-------------------------------------------
>
>        1 x GTX680: |        ns/day =      20.99  seconds/ns =
>4115.84
>
>FACTOR_IX_PRODUCTION_NPT - 90,906 atoms PME
>-------------------------------------------
>
>        1 x GTX680: |        ns/day =      16.89  seconds/ns =
>5115.66
>
>CELLULOSE_PRODUCTION_NVE - 408,609 atoms PME
>--------------------------------------------
>
>        1 x GTX680: |        ns/day =      4.67  seconds/ns =
>18485.98
>
>CELLULOSE_PRODUCTION_NPT -
 408,609 atoms PME
>--------------------------------------------
>
>        1 x GTX680: |        ns/day =      3.87  seconds/ns =
>22323.57
>
>TRPCAGE_PRODUCTION - 304 atoms GB
>---------------------------------
>
>        1 x GTX680: |        ns/day =    774.84  seconds/ns =
>111.51
>        2 x GTX680: N/A      3 x GTX680: N/A      4 x GTX680: N/A
>MYOGLOBIN_PRODUCTION - 2,492 atoms GB
>-------------------------------------
>
>        1 x GTX680: |        ns/day =    166.44  seconds/ns =
>519.10
>
>NUCLEOSOME_PRODUCTION - 25,095 atoms GB
>---------------------------------------
>
>     
 Â  1 x GTX680: |        ns/day =      2.90  seconds/ns =
>29755.05
>
>
>
>
>
>Dne Tue, 26 Feb 2013 17:14:01 +0100 Scott Le Grand
><varelse2005.gmail.com>
>napsal/-a:
>
>> As an side, go run JAC NVE in SPFP mode...
>>
>> If you get ~75+ ns/day, you're running at 1.05+ GHz...
>>
>> Otherwise, something's up.  And I second what Ross is saying - just sit
>> back and ride Pixel's Law.  In the mid-term, I think I'll get JAC to
>>200+
>> ns/day with a couple GTX Titans once I get the time to optimize GPU to
>> GPU
>> communication...
>>
>>
>>
>> On Mon, Feb 25, 2013 at 5:46 PM, Ross Walker <ross.rosswalker.co.uk>
>> wrote:
>>
>>> Hi Filip,
>>>
>>> I think you are worrying too much hear. Firstly you are referring to
>>>the
>>> double precision clock rate and not the single precision clock. AMBER
>>> stopped relying on the double precision side of things and switched to
>>> fixed point accumulation with the release of the GTX680 and K10. Second
>>> the stock single precision clock will be faster than the K20X so you
>>>can
>>> expect performance to be better than the K20X. It also has more cores
>>> active 'I think', don't have the specs here or internet access to check
>>> right now.
>>>
>>> Thirdly, the boost clock. AMBER pretty much runs the
 entire GPU flat
>>>out
>>> ALL the time. The
 boost clock is only useful, as with CPUs, when you
>>>are
>>> only using a fraction of the cores. In the case of GPUs unless you are
>>> running very small atom counts this is unlikely to happen so even if
>>>the
>>> boost clock was supported it wouldn't do you any good.
>>>
>>> In short, I wouldn't worry about it. Let's just wait and see how it
>>> truly
>>> performs when the "vaporware" actually turns up.
>>>
>>> All the best
>>> Ross
>>>
>>>
>>> On 2/25/13 2:39 PM, "filip fratev" <filipfratev.yahoo.com> wrote:
>>>
>>> >Hi all,
>>> >I received some tests performed. Here is the comparison between
 LuxMax
>>> >results obtained by
 GTX660 under Linux and Windows, respectively:
>>> >
>>> >http://img22.imageshack.us/img22/1279/luxmarkubuntu1204.png
>>> >http://img692.imageshack.us/img692/9647/luxmarkwin7.png
>>> >
>>> >According to these results the GTX660 works at 1071Mhz, thus the Boost
>>> >speed and the results between Linux and Windows are similar.
>>> >
>>> >However, Nvidia answered me that the GTX Titan core speed under Linux
>>> >will be 837MHz and about the boost technology this: "unfortunately no,
>>> >boost 1.0/2.0 are only supported on windows."
>>> >Personally I trust on the above tests:)
>>>
>If they really caped their
 GTX GPU's under Linux to the base clock
>>> >presumably only the BIOS hack option will be possible, which is....:)
>>> >
>>> >Regards,
>>> >Filip
>>> >
>>> >
>>> >
>>> >
>>> >________________________________
>>> > From: Aron Broom <broomsday.gmail.com>
>>> >To: filip fratev <filipfratev.yahoo.com>; AMBER Mailing List
>>> ><amber.ambermd.org>
>>> >Sent: Saturday, February 23, 2013 10:12 PM
>>> >Subject: Re:
 [AMBER] GTX Titan was finally released
>>> >
>>> >Just as another
 note, I checked out the AMBER output from running on a
>>> >GTX570,
>>> >
>>> >|------------------- GPU DEVICE INFO --------------------
>>> >|
>>> >|  CUDA Capable Devices Detected:      1
>>> >|          CUDA Device ID in use:      0
>>> >|                CUDA Device Name: GeForce GTX 570
>>> >|    CUDA Device Global Mem Size:  1279 MB
>>> >| CUDA Device Num Multiprocessors:    15
>>> >|          CUDA Device Core Freq:  1.46 GHz
>>> >|
>>> >|--------------------------------------------------------
>>> >
>>> >So in that case the Core Freq reported is indeed the correct one,
 even
>>> >though the GTX570 has two lower clock speeds it runs at depending on
>>> load
>>> >(810 MHz, and 101 MHz)
>>> >
>>> >I know with the 500 series, the available nVidia tools for linux will
>>> >least
>>> >allow you to set the device to maintain the highest clock speeds
>>> >regardless
>>> >of load.  I have NOT done that in the above case, but if such a thing
>>>
>>> is
>>> >possible for the 600 series, it might be worth looking at.  Sadly the
>>>
>>> tool
>>> >is only easily usable if you have a display connected although if you
>>> >google "Axel Kohlmeyer" and go to his homepage there are some
>>> suggestions
>>> >on installing these tools on a typical server where you can fake
 a
>>> >display.
>>> >
>>> >~Aron
>>> >
>>> >On Sat, Feb 23, 2013 at 2:33 PM, filip fratev <filipfratev.yahoo.com>
>>> >wrote:
>>> >
>>> >> Hi Ross, Aron and all,
>>> >> Thanks for your detail answers!!
>>> >>
>>> >> So, it seems that nobody know whether Nvidia
>>> >> support the boost speed even on GTX680. Moreover, because the core
>>> >>speed is
>>> >> wrongly (I hope) printed as in the case of Amber 12 as well in all
>>> >> benchmark
>>> >> applications, we can see the difference only if compare the GTX680
>>>to
>>> >>K10
>>>
>> (1
>>>
>> GPU) where we can see 37% performance increase (JAC), which can
>>>comes
>>> >>only
>>> >> from the
>>> >> core/memory clock.
>>> >>
>>> >> Ross, please ask Nvidia about these issues.
>>> >> I've already asked them but don't believe that will receive any
>>> adequate
>>> >> answer.
>>> >> I also asked several users but nobody knows and they told me that
>>> Nvidia
>>> >> never
>>> >> said something about their Boost technology under Linux.
>>> >> Thus, at this point I think that we can trust
>>> >> only to your information.
>>> >>
>>> >> Regards,
>>> >> Filip
>>> >>
>>> >>
>>> >>
 ________________________________
>>> >>  From: Ross Walker <ross.rosswalker.co.uk>
>>> >> To: filip fratev <filipfratev.yahoo.com>; AMBER Mailing List <
>>> >> amber.ambermd.org>
>>> >> Sent: Saturday, February 23, 2013 6:45 AM
>>> >> Subject: Re: [AMBER] GTX Titan was finally released
>>> >>
>>> >> Hi Filip,
>>> >>
>>> >>
>>> >> >As you know I plan to purchase few GTX Titans:)
>>> >> >but I am not sure
 actually at what speed they will run: 836, 876
>>>or 
>>>
 993
>>> >> >Mhz?
>>> >> >It seems that by default (80C target) the Titan
>>> >> >runs under Windows only on the maximal core speed (around 1Ghz)
>>>not 
>>> the
>>> >> >boost
>>> >> >one. It goes back to 836 only if the temperature rises above 80C
>>>but
>>> >>with
>>> >> >100%
>>> >> >fan speed this looks almost impossible. At least this is what I saw
>>> >>from
>>> >> >the
>>> >> >reviews.
>>> >>
>>> >> No idea since I am still waiting for NVIDIA to actually send me a
>>> >> development card to try this with. I guess the Titan's will be
>>> vaporware
>>> >> for a while. I am intrigued to know about how the
 clock speed will
>>> work
>>> >> and I am waiting for NVIDIA engineering to get back to me with a
>>> >> definitive answer. Note the Titan can also be run in two modes from
>>> >>what I
>>> >> gather. One with the DP cores turned down and the SP cores clocked
>>>up
>>> >> (Gaming mode) and one where it turns on all the DP cores and clocks
>>>
>>> down
>>> >> the single precision (CUDA mode). Note AMBER was retooled for the
>>> GK104
>>> >> chip to not use double precision anymore. It uses a combination of
>>> >>single
>>> >> and fixed precision which we worked very hard to tune to
>>>match/better
>>> >>the
>>> >> SPDP accuracy. Thus it is entirely possible that one will
 actually
>>> want
>>> >>to
>>> >> run the Titan cards in gaming mode when running AMBER. Of course
>>> this is
>>> >> entirely speculation until I lay my hands on one. The thermal window
>>> >>also
>>> >> has potential issues for 4 GPU boxes but there may end up being a
>>> hack
>>> >>to
>>> >> disable the down clocking and allow temps over 80C. Note most cards
>>>I
>>> >>have
>>> >> (GTX680s) run around 90C right now. SDSC runs it's machine room at
>>> 85F
>>> >>in
>>> >> order to save power - since disks and CPUs don't care if the room
>>>is 
>>> 85F
>>> >> vs 60F. This might be a different story if the GPUs throttle
 based
>>>on
>>> >> temperature but I guess we'll just have to wait and see.
>>> >>
>>> >> >
>>> >> >I was also horrified to see that many GTX680
>>> >> >(and other cards) users complain that under Linux their cards run
>>>at
>>> >>only
>>> >> >about
>>> >> >700Mhz core speed instead of 1Ghz. What is your experience with GTX
>>> >>680?
>>> >> >I was also wondering whether the GTX680 use the
>>> >> >boost clock during the Amber calculations or the just the base one?
>>> >> >
>>> >>
>>> >> I think this is just speculation. When you run AMBER with a GTX680
>>>it
>>> >> prints the following:
>>> >>
>>>
>>
>>> >> |------------------- GPU DEVICE INFO --------------------
>>> >> |
>>> >> |  CUDA Capable Devices Detected:      1
>>> >> |          CUDA Device ID in use:      0
>>> >> |                CUDA Device Name: GeForce GTX 680
>>> >> |    CUDA Device Global Mem Size:  2047 MB
>>> >> | CUDA Device Num Multiprocessors:      8
>>> >> |          CUDA Device Core Freq:  0.71 GHz
>>> >> |
>>> >> |--------------------------------------------------------
>>> >>
>>> >>
>>> >>
>>> >> But this is a query that occurs at the very beginning of a
 run
>>>before
>>> >>any
>>> >> CUDA kernels have been run. I believe that when unloaded the 680 in
>>> >>Linux
>>> >> clocks down to 705MHz to save power. When you stress it hard it
>>> >> automatically clocks up the frequency. I am not sure if there is
>>>way 
>>> to
>>> >> check this though while the card is under load. Certainly the
>>> >>performance
>>> >> we see would be what it is if the clock speed was only 705MHz. I am
>>> >>asking
>>> >> NVIDIA engineering to clarify though.
>>> >>
>>> >> >Finally, what is the performance difference of
>>> >> >pmemdCuda under Linux and Cygwin?
>>> >>
>>> >> Never tried and I very much doubt you'll
 be able to get pmemd.cuda
>>> >> compiled under cygwin. Cygwin emulates things through the cygwin
>>>dll 
>>> and
>>> >> so you'd need a cygwin compatible version of the nvidia compiler I'd
>>> >> expect.
>>> >>
>>> >> Note have a native Windows version of pmemd.cuda but never released
>>>
>>> the
>>> >> binary since the performance is about half that of what it is on
>>> Linux
>>> >>due
>>> >> to a bug in cuda 4.2 under windows that limited performance. cuda 3
>>> >>showed
>>> >> good performance under windows but you can't use that with AMBER
>>>12. 
>>> We
>>> >> haven't had time to get back to looking at this with cuda 5
>>>
>>unfortunately.
>>> >>
>>> >> All the best
>>> >> Ross
>>> >>
>>> >> /\
>>> >> \/
>>> >> |\oss Walker
>>> >>
>>> >> ---------------------------------------------------------
>>> >> |            Assistant Research Professor              |
>>> >> |            San Diego Supercomputer Center            |
>>> >> |            Adjunct Assistant Professor              |
>>> >> |        Dept. of Chemistry and Biochemistry          |
>>> >> |       
 Â  University of California San Diego          |
>>> >> |                    NVIDIA Fellow                    |
>>> >> | http://www.rosswalker.co.uk | http://www.wmd-lab.org  |
>>> >> | Tel: +1 858 822 0854 | EMail:- ross.rosswalker.co.uk  |
>>> >> ---------------------------------------------------------
>>> >>
>>> >> Note: Electronic Mail is not secure, has no guarantee of delivery,
>>> may
>>> >>not
>>> >> be
 read every day, and should not be used for urgent or
 sensitive
>>> >>issues.
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> AMBER mailing list
>>> >> AMBER.ambermd.org
>>> >> http://lists.ambermd.org/mailman/listinfo/amber
>>> >> _______________________________________________
>>> >> AMBER mailing list
>>> >> AMBER.ambermd.org
>>> >> http://lists.ambermd.org/mailman/listinfo/amber
>>> >>
>>> >
>>> >
>>> >
>>> >--
>>> >Aron Broom M.Sc
>>> >PhD Student
>>> >Department of Chemistry
>>> >University of Waterloo
>>> >_______________________________________________
>>> >AMBER mailing list
>>> >AMBER.ambermd.org
>>> >http://lists.ambermd.org/mailman/listinfo/amber
>>>
>_______________________________________________
>>> >AMBER mailing list
>>> >AMBER.ambermd.org
>>> >http://lists.ambermd.org/mailman/listinfo/amber
>>>
>>>
>>>
>>> _______________________________________________
>>> AMBER mailing list
>>> AMBER.ambermd.org
>>> http://lists.ambermd.org/mailman/listinfo/amber
>>>
>> _______________________________________________
>> AMBER mailing list
>> AMBER.ambermd.org
>> http://lists.ambermd.org/mailman/listinfo/amber
>>
>> __________ Informace od ESET NOD32 Antivirus, verze databaze 8053
>> (20130226) __________
>>
>> Tuto zpravu proveril ESET NOD32 Antivirus.
>>
>> http://www.eset.cz
>>
>>
>>
>
>
>--
>Tato zpráva byla vytvořena převratným poštovním klientem Opery:
>http://www.opera.com/mail/
>
>_______________________________________________
>AMBER mailing list
>AMBER.ambermd.org
>http://lists.ambermd.org/mailman/listinfo/amber
>_______________________________________________
>AMBER mailing list
>AMBER.ambermd.org
>http://lists.ambermd.org/mailman/listinfo/amber
>_______________________________________________
>AMBER mailing list
>AMBER.ambermd.org
>http://lists.ambermd.org/mailman/listinfo/amber



_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Sun Mar 17 2013 - 07:00:02 PDT