Thanks!
Gustavo Seabra
Professor Adjunto
Departamento de Química Fundamental
Universidade Federal de Pernambuco
Fone: +55-81-2126-7450
On Thu, Mar 14, 2013 at 2:42 PM, Aron Broom <broomsday.gmail.com> wrote:
> Yeah, the bandwidth limitations across the (I'm assuming PCI 3.0)
> motherboard connections is a bottle-neck for using any more than 1GPU.
> Still those numbers are pretty extreme. What are we talking about in terms
> of CPU equivalence? Even using NAMD or something that scales quite
> linearly on the CPU I think some of those would take >64 cores to match,
> and you aren't getting 64 cores for anything even remotely close to the
> $1000 for the Titan.
>
> ~Aron
>
> On Thu, Mar 14, 2013 at 1:31 PM, Gustavo Seabra <gustavo.seabra.gmail.com
> >wrote:
>
> > Looks like there's really little gain by using 2 GPUs in parallel. Is
> that
> > expected?
> >
> > Gustavo Seabra
> > Professor Adjunto
> > Departamento de Química Fundamental
> > Universidade Federal de Pernambuco
> > Fone: +55-81-2126-7450
> >
> >
> > On Thu, Mar 14, 2013 at 1:33 PM, filip fratev <filipfratev.yahoo.com>
> > wrote:
> >
> > > Hi Ross and all,
> > >
> > > These are my test results for GTX Titan. Just great gpu! My results
> > differ
> > > 1-3% from those obtained by Ian. I am not sure why. First of all Titan
> > is a
> > > cold card. When I set the fan speed to only 70-75%, the temperature
> > never
> > > goes above 60-65C. It is a pity that the card clock is only 876Mhz.
> Under
> > > Windows I was not able to heat the card above 74 C and the speed was
> > > 1150-1170Mhz, i.e. under Windows the single precision speed and the
> boost
> > > speed are equal to the so called maximal clock speed. Many folks have
> > > already hacked their bioses. Anyway..
> > >
> > > I use i7 3770K.4.6Ghz, GB GTX-Titan and my RAM is clocked above
> 2400Mhz.
> > > OS was Suse 12.1+cuda 5.0.
> > >
> > > ----------------------------------
> > > DHFR NVE = 23,558 atoms
> > >
> > > 1xGTX Titan = 110.65 ns/day
> > > 2xGTX Titan = 125.28 ns/day
> > >
> > > --------------------------------------
> > > DHFR NPT = 23,558 atoms
> > >
> > > 1xGTX Titan = 85.27 ns/day
> > > 2xGTX Titan = 101.88 ns/day
> > >
> > > ---------------------------------------
> > > FactorIX NVE = 90,906 atoms
> > >
> > > 1xGTX Titan = 31.55 ns/day
> > > 2xGTX Titan = 38.05 ns/day
> > >
> > > ----------------------------------
> > > FactorIX NPT = 90,906 atoms
> > >
> > > 1xGTX Titan = 25.85 ns/day
> > > 2xGTX Titan = 32.54 ns/day
> > >
> > > ----------------------------------------
> > > Cellulose NVE = 408,609 atoms
> > >
> > > 1xGTX Titan = 7.50 ns/day
> > > 2xGTX Titan = 8.72 ns/day
> > >
> > > ----------------------------------------
> > > Cellulose NPT = 408,609 atoms
> > >
> > > 1xGTX Titan = 6.31 ns/day
> > > 2xGTX Titan = 7.71 ns/day
> > >
> > > -------------------------------------------
> > >
> > > All the best,
> > > Filip
> > >
> > >
> > >
> > > ________________________________
> > > From: filip fratev <filipfratev.yahoo.com>
> > > To: AMBER Mailing List <amber.ambermd.org>
> > > Sent: Wednesday, February 27, 2013 9:21 PM
> > > Subject: Re: [AMBER] GTX Titan was finally released
> > >
> > > Hi Marek,
> > >
> > >
> > > Thanks for sharing your benchmarks! Also thanks to Ross and Scott!
> > >
> > > 8-10% is not insignificant difference considering that the difference
> > > between one two GPU's are 14% in JAC NPT. At least for me:)
> > >
> > > The EVGA revealed the Titan clock speed for their superclocked
> version-
> > > 876Mhz, i.e. noting intriguing.
> > >
> > >
> > > Ross you mentioned:
> > > >>Firstly you are referring to the double precision clock rate and not
> > the
> > > single precision clock.
> > >
> > > What will be the single precision clock?
> > >
> > >
> > > All the best,
> > > Filip
> > >
> > > P.S. The GPU Boost 2.0 is different than the CPU boost. Your GPU can be
> > > under 100% load but it will still work on the boost clock (876Mhz,
> single
> > > precision?) and under Windows on nearly 1Ghz and this will be changeed
> > only
> > > if your temperature is above 80C.Thus if one use a water cooling under
> > > Windows will be able to use Titan on 1Ghz which is around 10-15% + in
> the
> > > performance.
> > >
> > >
> > >
> > > ________________________________
> > > From: Marek Maly <marek.maly.ujep.cz>
> > > To: AMBER Mailing List <amber.ambermd.org>
> > > Sent: Wednesday, February 27, 2013 8:23 PM
> > > Subject: Re: [AMBER] GTX Titan was finally released
> > >
> > > Hi Guys,
> > > here are finally the results of in factory over clocked GTX680
> > > ( EVGA GeForce GTX680 Classified ) in combination with "ASUS P9X79 PRO"
> > > motherboard.
> > >
> > > As one can see the increase from the reference 1006MHz to 1111MHz make
> > just
> > > a small difference in results (reflecting percentually more or less the
> > > difference
> > > in frquency ). I did not test it in Boost clock (1176MHz) and I am not
> > > going to do it, as for the long MD runs this regime seems to me a bit
> > > dangerous :))
> > >
> > > Regarding the reliability of this OC version, I am fully satisfied,
> > already
> > > tested 2 of these in few weeks simulations.
> > >
> > > Best wishes,
> > >
> > > Marek
> > >
> > >
> > > JAC_PRODUCTION_NVE - 23,558 atoms PME
> > > -------------------------------------
> > >
> > > 1 x GTX680: | ns/day = 80.38 seconds/ns =
> > 1074.90
> > >
> > > JAC_PRODUCTION_NPT - 23,558 atoms PME
> > > -------------------------------------
> > >
> > > 1 x GTX680: | ns/day = 64.59 seconds/ns =
> > 1337.60
> > >
> > > FACTOR_IX_PRODUCTION_NVE - 90,906 atoms PME
> > > -------------------------------------------
> > >
> > > 1 x GTX680: | ns/day = 20.99 seconds/ns =
> > 4115.84
> > >
> > > FACTOR_IX_PRODUCTION_NPT - 90,906 atoms PME
> > > -------------------------------------------
> > >
> > > 1 x GTX680: | ns/day = 16.89 seconds/ns =
> > 5115.66
> > >
> > > CELLULOSE_PRODUCTION_NVE - 408,609 atoms PME
> > > --------------------------------------------
> > >
> > > 1 x GTX680: | ns/day = 4.67 seconds/ns =
> > 18485.98
> > >
> > > CELLULOSE_PRODUCTION_NPT - 408,609 atoms PME
> > > --------------------------------------------
> > >
> > > 1 x GTX680: | ns/day = 3.87 seconds/ns =
> > 22323.57
> > >
> > > TRPCAGE_PRODUCTION - 304 atoms GB
> > > ---------------------------------
> > >
> > > 1 x GTX680: | ns/day = 774.84 seconds/ns =
> > 111.51
> > > 2 x GTX680: N/A 3 x GTX680: N/A 4 x GTX680: N/A
> > > MYOGLOBIN_PRODUCTION - 2,492 atoms GB
> > > -------------------------------------
> > >
> > > 1 x GTX680: | ns/day = 166.44 seconds/ns =
> > 519.10
> > >
> > > NUCLEOSOME_PRODUCTION - 25,095 atoms GB
> > > ---------------------------------------
> > >
> > > 1 x GTX680: | ns/day = 2.90 seconds/ns =
> > 29755.05
> > >
> > >
> > >
> > >
> > >
> > > Dne Tue, 26 Feb 2013 17:14:01 +0100 Scott Le Grand <
> > varelse2005.gmail.com
> > > >
> > > napsal/-a:
> > >
> > > > As an side, go run JAC NVE in SPFP mode...
> > > >
> > > > If you get ~75+ ns/day, you're running at 1.05+ GHz...
> > > >
> > > > Otherwise, something's up. And I second what Ross is saying - just
> sit
> > > > back and ride Pixel's Law. In the mid-term, I think I'll get JAC to
> > 200+
> > > > ns/day with a couple GTX Titans once I get the time to optimize GPU
> to
> > > > GPU
> > > > communication...
> > > >
> > > >
> > > >
> > > > On Mon, Feb 25, 2013 at 5:46 PM, Ross Walker <ross.rosswalker.co.uk>
> > > > wrote:
> > > >
> > > >> Hi Filip,
> > > >>
> > > >> I think you are worrying too much hear. Firstly you are referring to
> > the
> > > >> double precision clock rate and not the single precision clock.
> AMBER
> > > >> stopped relying on the double precision side of things and switched
> to
> > > >> fixed point accumulation with the release of the GTX680 and K10.
> > Second
> > > >> the stock single precision clock will be faster than the K20X so you
> > can
> > > >> expect performance to be better than the K20X. It also has more
> cores
> > > >> active 'I think', don't have the specs here or internet access to
> > check
> > > >> right now.
> > > >>
> > > >> Thirdly, the boost clock. AMBER pretty much runs the entire GPU flat
> > out
> > > >> ALL the time. The boost clock is only useful, as with CPUs, when you
> > are
> > > >> only using a fraction of the cores. In the case of GPUs unless you
> are
> > > >> running very small atom counts this is unlikely to happen so even if
> > the
> > > >> boost clock was supported it wouldn't do you any good.
> > > >>
> > > >> In short, I wouldn't worry about it. Let's just wait and see how it
> > > >> truly
> > > >> performs when the "vaporware" actually turns up.
> > > >>
> > > >> All the best
> > > >> Ross
> > > >>
> > > >>
> > > >> On 2/25/13 2:39 PM, "filip fratev" <filipfratev.yahoo.com> wrote:
> > > >>
> > > >> >Hi all,
> > > >> >I received some tests performed. Here is the comparison between
> > LuxMax
> > > >> >results obtained by GTX660 under Linux and Windows, respectively:
> > > >> >
> > > >> >http://img22.imageshack.us/img22/1279/luxmarkubuntu1204.png
> > > >> >http://img692.imageshack.us/img692/9647/luxmarkwin7.png
> > > >> >
> > > >> >According to these results the GTX660 works at 1071Mhz, thus the
> > Boost
> > > >> >speed and the results between Linux and Windows are similar.
> > > >> >
> > > >> >However, Nvidia answered me that the GTX Titan core speed under
> Linux
> > > >> >will be 837MHz and about the boost technology this: "unfortunately
> > no,
> > > >> >boost 1.0/2.0 are only supported on windows."
> > > >> >Personally I trust on the above tests:)
> > > >> >If they really caped their GTX GPU's under Linux to the base clock
> > > >> >presumably only the BIOS hack option will be possible, which
> is....:)
> > > >> >
> > > >> >Regards,
> > > >> >Filip
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >________________________________
> > > >> > From: Aron Broom <broomsday.gmail.com>
> > > >> >To: filip fratev <filipfratev.yahoo.com>; AMBER Mailing List
> > > >> ><amber.ambermd.org>
> > > >> >Sent: Saturday, February 23, 2013 10:12 PM
> > > >> >Subject: Re: [AMBER] GTX Titan was finally released
> > > >> >
> > > >> >Just as another note, I checked out the AMBER output from running
> on
> > a
> > > >> >GTX570,
> > > >> >
> > > >> >|------------------- GPU DEVICE INFO --------------------
> > > >> >|
> > > >> >| CUDA Capable Devices Detected: 1
> > > >> >| CUDA Device ID in use: 0
> > > >> >| CUDA Device Name: GeForce GTX 570
> > > >> >| CUDA Device Global Mem Size: 1279 MB
> > > >> >| CUDA Device Num Multiprocessors: 15
> > > >> >| CUDA Device Core Freq: 1.46 GHz
> > > >> >|
> > > >> >|--------------------------------------------------------
> > > >> >
> > > >> >So in that case the Core Freq reported is indeed the correct one,
> > even
> > > >> >though the GTX570 has two lower clock speeds it runs at depending
> on
> > > >> load
> > > >> >(810 MHz, and 101 MHz)
> > > >> >
> > > >> >I know with the 500 series, the available nVidia tools for linux
> will
> > > >> >least
> > > >> >allow you to set the device to maintain the highest clock speeds
> > > >> >regardless
> > > >> >of load. I have NOT done that in the above case, but if such a
> thing
> > > >> is
> > > >> >possible for the 600 series, it might be worth looking at. Sadly
> the
> > > >> tool
> > > >> >is only easily usable if you have a display connected although if
> you
> > > >> >google "Axel Kohlmeyer" and go to his homepage there are some
> > > >> suggestions
> > > >> >on installing these tools on a typical server where you can fake a
> > > >> >display.
> > > >> >
> > > >> >~Aron
> > > >> >
> > > >> >On Sat, Feb 23, 2013 at 2:33 PM, filip fratev <
> filipfratev.yahoo.com
> > >
> > > >> >wrote:
> > > >> >
> > > >> >> Hi Ross, Aron and all,
> > > >> >> Thanks for your detail answers!!
> > > >> >>
> > > >> >> So, it seems that nobody know whether Nvidia
> > > >> >> support the boost speed even on GTX680. Moreover, because the
> core
> > > >> >>speed is
> > > >> >> wrongly (I hope) printed as in the case of Amber 12 as well in
> all
> > > >> >> benchmark
> > > >> >> applications, we can see the difference only if compare the
> GTX680
> > to
> > > >> >>K10
> > > >> >> (1
> > > >> >> GPU) where we can see 37% performance increase (JAC), which can
> > comes
> > > >> >>only
> > > >> >> from the
> > > >> >> core/memory clock.
> > > >> >>
> > > >> >> Ross, please ask Nvidia about these issues.
> > > >> >> I've already asked them but don't believe that will receive any
> > > >> adequate
> > > >> >> answer.
> > > >> >> I also asked several users but nobody knows and they told me that
> > > >> Nvidia
> > > >> >> never
> > > >> >> said something about their Boost technology under Linux.
> > > >> >> Thus, at this point I think that we can trust
> > > >> >> only to your information.
> > > >> >>
> > > >> >> Regards,
> > > >> >> Filip
> > > >> >>
> > > >> >>
> > > >> >> ________________________________
> > > >> >> From: Ross Walker <ross.rosswalker.co.uk>
> > > >> >> To: filip fratev <filipfratev.yahoo.com>; AMBER Mailing List <
> > > >> >> amber.ambermd.org>
> > > >> >> Sent: Saturday, February 23, 2013 6:45 AM
> > > >> >> Subject: Re: [AMBER] GTX Titan was finally released
> > > >> >>
> > > >> >> Hi Filip,
> > > >> >>
> > > >> >>
> > > >> >> >As you know I plan to purchase few GTX Titans:)
> > > >> >> >but I am not sure actually at what speed they will run: 836, 876
> > or
> > > >> 993
> > > >> >> >Mhz?
> > > >> >> >It seems that by default (80C target) the Titan
> > > >> >> >runs under Windows only on the maximal core speed (around 1Ghz)
> > not
> > > >> the
> > > >> >> >boost
> > > >> >> >one. It goes back to 836 only if the temperature rises above 80C
> > but
> > > >> >>with
> > > >> >> >100%
> > > >> >> >fan speed this looks almost impossible. At least this is what I
> > saw
> > > >> >>from
> > > >> >> >the
> > > >> >> >reviews.
> > > >> >>
> > > >> >> No idea since I am still waiting for NVIDIA to actually send me a
> > > >> >> development card to try this with. I guess the Titan's will be
> > > >> vaporware
> > > >> >> for a while. I am intrigued to know about how the clock speed
> will
> > > >> work
> > > >> >> and I am waiting for NVIDIA engineering to get back to me with a
> > > >> >> definitive answer. Note the Titan can also be run in two modes
> from
> > > >> >>what I
> > > >> >> gather. One with the DP cores turned down and the SP cores
> clocked
> > up
> > > >> >> (Gaming mode) and one where it turns on all the DP cores and
> clocks
> > > >> down
> > > >> >> the single precision (CUDA mode). Note AMBER was retooled for the
> > > >> GK104
> > > >> >> chip to not use double precision anymore. It uses a combination
> of
> > > >> >>single
> > > >> >> and fixed precision which we worked very hard to tune to
> > match/better
> > > >> >>the
> > > >> >> SPDP accuracy. Thus it is entirely possible that one will
> actually
> > > >> want
> > > >> >>to
> > > >> >> run the Titan cards in gaming mode when running AMBER. Of course
> > > >> this is
> > > >> >> entirely speculation until I lay my hands on one. The thermal
> > window
> > > >> >>also
> > > >> >> has potential issues for 4 GPU boxes but there may end up being a
> > > >> hack
> > > >> >>to
> > > >> >> disable the down clocking and allow temps over 80C. Note most
> > cards I
> > > >> >>have
> > > >> >> (GTX680s) run around 90C right now. SDSC runs it's machine room
> at
> > > >> 85F
> > > >> >>in
> > > >> >> order to save power - since disks and CPUs don't care if the room
> > is
> > > >> 85F
> > > >> >> vs 60F. This might be a different story if the GPUs throttle
> based
> > on
> > > >> >> temperature but I guess we'll just have to wait and see.
> > > >> >>
> > > >> >> >
> > > >> >> >I was also horrified to see that many GTX680
> > > >> >> >(and other cards) users complain that under Linux their cards
> run
> > at
> > > >> >>only
> > > >> >> >about
> > > >> >> >700Mhz core speed instead of 1Ghz. What is your experience with
> > GTX
> > > >> >>680?
> > > >> >> >I was also wondering whether the GTX680 use the
> > > >> >> >boost clock during the Amber calculations or the just the base
> > one?
> > > >> >> >
> > > >> >>
> > > >> >> I think this is just speculation. When you run AMBER with a
> GTX680
> > it
> > > >> >> prints the following:
> > > >> >>
> > > >> >>
> > > >> >> |------------------- GPU DEVICE INFO --------------------
> > > >> >> |
> > > >> >> | CUDA Capable Devices Detected: 1
> > > >> >> | CUDA Device ID in use: 0
> > > >> >> | CUDA Device Name: GeForce GTX 680
> > > >> >> | CUDA Device Global Mem Size: 2047 MB
> > > >> >> | CUDA Device Num Multiprocessors: 8
> > > >> >> | CUDA Device Core Freq: 0.71 GHz
> > > >> >> |
> > > >> >> |--------------------------------------------------------
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> But this is a query that occurs at the very beginning of a run
> > before
> > > >> >>any
> > > >> >> CUDA kernels have been run. I believe that when unloaded the 680
> in
> > > >> >>Linux
> > > >> >> clocks down to 705MHz to save power. When you stress it hard it
> > > >> >> automatically clocks up the frequency. I am not sure if there is
> > way
> > > >> to
> > > >> >> check this though while the card is under load. Certainly the
> > > >> >>performance
> > > >> >> we see would be what it is if the clock speed was only 705MHz. I
> am
> > > >> >>asking
> > > >> >> NVIDIA engineering to clarify though.
> > > >> >>
> > > >> >> >Finally, what is the performance difference of
> > > >> >> >pmemdCuda under Linux and Cygwin?
> > > >> >>
> > > >> >> Never tried and I very much doubt you'll be able to get
> pmemd.cuda
> > > >> >> compiled under cygwin. Cygwin emulates things through the cygwin
> > dll
> > > >> and
> > > >> >> so you'd need a cygwin compatible version of the nvidia compiler
> > I'd
> > > >> >> expect.
> > > >> >>
> > > >> >> Note have a native Windows version of pmemd.cuda but never
> released
> > > >> the
> > > >> >> binary since the performance is about half that of what it is on
> > > >> Linux
> > > >> >>due
> > > >> >> to a bug in cuda 4.2 under windows that limited performance.
> cuda 3
> > > >> >>showed
> > > >> >> good performance under windows but you can't use that with AMBER
> > 12.
> > > >> We
> > > >> >> haven't had time to get back to looking at this with cuda 5
> > > >> >>unfortunately.
> > > >> >>
> > > >> >> All the best
> > > >> >> Ross
> > > >> >>
> > > >> >> /\
> > > >> >> \/
> > > >> >> |\oss Walker
> > > >> >>
> > > >> >> ---------------------------------------------------------
> > > >> >> | Assistant Research Professor |
> > > >> >> | San Diego Supercomputer Center |
> > > >> >> | Adjunct Assistant Professor |
> > > >> >> | Dept. of Chemistry and Biochemistry |
> > > >> >> | University of California San Diego |
> > > >> >> | NVIDIA Fellow |
> > > >> >> | http://www.rosswalker.co.uk | http://www.wmd-lab.org |
> > > >> >> | Tel: +1 858 822 0854 | EMail:- ross.rosswalker.co.uk |
> > > >> >> ---------------------------------------------------------
> > > >> >>
> > > >> >> Note: Electronic Mail is not secure, has no guarantee of
> delivery,
> > > >> may
> > > >> >>not
> > > >> >> be read every day, and should not be used for urgent or sensitive
> > > >> >>issues.
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> _______________________________________________
> > > >> >> AMBER mailing list
> > > >> >> AMBER.ambermd.org
> > > >> >> http://lists.ambermd.org/mailman/listinfo/amber
> > > >> >> _______________________________________________
> > > >> >> AMBER mailing list
> > > >> >> AMBER.ambermd.org
> > > >> >> http://lists.ambermd.org/mailman/listinfo/amber
> > > >> >>
> > > >> >
> > > >> >
> > > >> >
> > > >> >--
> > > >> >Aron Broom M.Sc
> > > >> >PhD Student
> > > >> >Department of Chemistry
> > > >> >University of Waterloo
> > > >> >_______________________________________________
> > > >> >AMBER mailing list
> > > >> >AMBER.ambermd.org
> > > >> >http://lists.ambermd.org/mailman/listinfo/amber
> > > >> >_______________________________________________
> > > >> >AMBER mailing list
> > > >> >AMBER.ambermd.org
> > > >> >http://lists.ambermd.org/mailman/listinfo/amber
> > > >>
> > > >>
> > > >>
> > > >> _______________________________________________
> > > >> AMBER mailing list
> > > >> AMBER.ambermd.org
> > > >> http://lists.ambermd.org/mailman/listinfo/amber
> > > >>
> > > > _______________________________________________
> > > > AMBER mailing list
> > > > AMBER.ambermd.org
> > > > http://lists.ambermd.org/mailman/listinfo/amber
> > > >
> > > > __________ Informace od ESET NOD32 Antivirus, verze databaze 8053
> > > > (20130226) __________
> > > >
> > > > Tuto zpravu proveril ESET NOD32 Antivirus.
> > > >
> > > > http://www.eset.cz
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Tato zpráva byla vytvořena převratným poštovním klientem Opery:
> > > http://www.opera.com/mail/
> > >
> > > _______________________________________________
> > > AMBER mailing list
> > > AMBER.ambermd.org
> > > http://lists.ambermd.org/mailman/listinfo/amber
> > > _______________________________________________
> > > AMBER mailing list
> > > AMBER.ambermd.org
> > > http://lists.ambermd.org/mailman/listinfo/amber
> > > _______________________________________________
> > > AMBER mailing list
> > > AMBER.ambermd.org
> > > http://lists.ambermd.org/mailman/listinfo/amber
> > >
> > _______________________________________________
> > AMBER mailing list
> > AMBER.ambermd.org
> > http://lists.ambermd.org/mailman/listinfo/amber
> >
>
>
>
> --
> Aron Broom M.Sc
> PhD Student
> Department of Chemistry
> University of Waterloo
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
>
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Thu Mar 14 2013 - 11:00:05 PDT
