Probably needs an hour or two of load-balancing work on my part to boost
the scaling 20-50%...
The P2P would then be free beer on top of that. How well does Nucleosome
GB scale?
On Thu, Mar 14, 2013 at 11:54 AM, Ross Walker <ross.rosswalker.co.uk> wrote:
> For now it is expected yes. We hope to have a peer to peer implementation
> (instead of MPI) at some point likely for the next release of AMBER that
> will improve things when running within a single node.
>
> All the best
> Ross
>
>
> On 3/14/13 10:31 AM, "Gustavo Seabra" <gustavo.seabra.gmail.com> wrote:
>
> >Looks like there's really little gain by using 2 GPUs in parallel. Is that
> >expected?
> >
> >Gustavo Seabra
> >Professor Adjunto
> >Departamento de Química Fundamental
> >Universidade Federal de Pernambuco
> >Fone: +55-81-2126-7450
> >
> >
> >On Thu, Mar 14, 2013 at 1:33 PM, filip fratev <filipfratev.yahoo.com>
> >wrote:
> >
> >> Hi Ross and all,
> >>
> >> These are my test results for GTX Titan. Just great gpu! My results
> >>differ
> >> 1-3% from those obtained by Ian. I am not sure why. First of all Titan
> >>is a
> >> cold card. When I set the fan speed to only 70-75%, the temperature
> >>never
> >> goes above 60-65C. It is a pity that the card clock is only 876Mhz.
> >>Under
> >> Windows I was not able to heat the card above 74 C and the speed was
> >> 1150-1170Mhz, i.e. under Windows the single precision speed and the
> >>boost
> >> speed are equal to the so called maximal clock speed. Many folks have
> >> already hacked their bioses. Anyway..
> >>
> >> I use i7 3770K.4.6Ghz, GB GTX-Titan and my RAM is clocked above
> 2400Mhz.
> >> OS was Suse 12.1+cuda 5.0.
> >>
> >> ----------------------------------
> >> DHFR NVE = 23,558 atoms
> >>
> >> 1xGTX Titan = 110.65 ns/day
> >> 2xGTX Titan = 125.28 ns/day
> >>
> >> --------------------------------------
> >> DHFR NPT = 23,558 atoms
> >>
> >> 1xGTX Titan = 85.27 ns/day
> >> 2xGTX Titan = 101.88 ns/day
> >>
> >> ---------------------------------------
> >> FactorIX NVE = 90,906 atoms
> >>
> >> 1xGTX Titan = 31.55 ns/day
> >> 2xGTX Titan = 38.05 ns/day
> >>
> >> ----------------------------------
> >> FactorIX NPT = 90,906 atoms
> >>
> >> 1xGTX Titan = 25.85 ns/day
> >> 2xGTX Titan = 32.54 ns/day
> >>
> >> ----------------------------------------
> >> Cellulose NVE = 408,609 atoms
> >>
> >> 1xGTX Titan = 7.50 ns/day
> >> 2xGTX Titan = 8.72 ns/day
> >>
> >> ----------------------------------------
> >> Cellulose NPT = 408,609 atoms
> >>
> >> 1xGTX Titan = 6.31 ns/day
> >> 2xGTX Titan = 7.71 ns/day
> >>
> >> -------------------------------------------
> >>
> >> All the best,
> >> Filip
> >>
> >>
> >>
> >> ________________________________
> >> From: filip fratev <filipfratev.yahoo.com>
> >> To: AMBER Mailing List <amber.ambermd.org>
> >> Sent: Wednesday, February 27, 2013 9:21 PM
> >> Subject: Re: [AMBER] GTX Titan was finally released
> >>
> >> Hi Marek,
> >>
> >>
> >> Thanks for sharing your benchmarks! Also thanks to Ross and Scott!
> >>
> >> 8-10% is not insignificant difference considering that the difference
> >> between one two GPU's are 14% in JAC NPT. At least for me:)
> >>
> >> The EVGA revealed the Titan clock speed for their superclocked version-
> >> 876Mhz, i.e. noting intriguing.
> >>
> >>
> >> Ross you mentioned:
> >> >>Firstly you are referring to the double precision clock rate and not
> >>the
> >> single precision clock.
> >>
> >> What will be the single precision clock?
> >>
> >>
> >> All the best,
> >> Filip
> >>
> >> P.S. The GPU Boost 2.0 is different than the CPU boost. Your GPU can be
> >> under 100% load but it will still work on the boost clock (876Mhz,
> >>single
> >> precision?) and under Windows on nearly 1Ghz and this will be changeed
> >>only
> >> if your temperature is above 80C.Thus if one use a water cooling under
> >> Windows will be able to use Titan on 1Ghz which is around 10-15% + in
> >>the
> >> performance.
> >>
> >>
> >>
> >> ________________________________
> >> From: Marek Maly <marek.maly.ujep.cz>
> >> To: AMBER Mailing List <amber.ambermd.org>
> >> Sent: Wednesday, February 27, 2013 8:23 PM
> >> Subject: Re: [AMBER] GTX Titan was finally released
> >>
> >> Hi Guys,
> >> here are finally the results of in factory over clocked GTX680
> >> ( EVGA GeForce GTX680 Classified ) in combination with "ASUS P9X79 PRO"
> >> motherboard.
> >>
> >> As one can see the increase from the reference 1006MHz to 1111MHz make
> >>just
> >> a small difference in results (reflecting percentually more or less the
> >> difference
> >> in frquency ). I did not test it in Boost clock (1176MHz) and I am not
> >> going to do it, as for the long MD runs this regime seems to me a bit
> >> dangerous :))
> >>
> >> Regarding the reliability of this OC version, I am fully satisfied,
> >>already
> >> tested 2 of these in few weeks simulations.
> >>
> >> Best wishes,
> >>
> >> Marek
> >>
> >>
> >> JAC_PRODUCTION_NVE - 23,558 atoms PME
> >> -------------------------------------
> >>
> >> 1 x GTX680: | ns/day = 80.38 seconds/ns =
> >>1074.90
> >>
> >> JAC_PRODUCTION_NPT - 23,558 atoms PME
> >> -------------------------------------
> >>
> >> 1 x GTX680: | ns/day = 64.59 seconds/ns =
> >>1337.60
> >>
> >> FACTOR_IX_PRODUCTION_NVE - 90,906 atoms PME
> >> -------------------------------------------
> >>
> >> 1 x GTX680: | ns/day = 20.99 seconds/ns =
> >>4115.84
> >>
> >> FACTOR_IX_PRODUCTION_NPT - 90,906 atoms PME
> >> -------------------------------------------
> >>
> >> 1 x GTX680: | ns/day = 16.89 seconds/ns =
> >>5115.66
> >>
> >> CELLULOSE_PRODUCTION_NVE - 408,609 atoms PME
> >> --------------------------------------------
> >>
> >> 1 x GTX680: | ns/day = 4.67 seconds/ns =
> >>18485.98
> >>
> >> CELLULOSE_PRODUCTION_NPT - 408,609 atoms PME
> >> --------------------------------------------
> >>
> >> 1 x GTX680: | ns/day = 3.87 seconds/ns =
> >>22323.57
> >>
> >> TRPCAGE_PRODUCTION - 304 atoms GB
> >> ---------------------------------
> >>
> >> 1 x GTX680: | ns/day = 774.84 seconds/ns =
> >>111.51
> >> 2 x GTX680: N/A 3 x GTX680: N/A 4 x GTX680: N/A
> >> MYOGLOBIN_PRODUCTION - 2,492 atoms GB
> >> -------------------------------------
> >>
> >> 1 x GTX680: | ns/day = 166.44 seconds/ns =
> >>519.10
> >>
> >> NUCLEOSOME_PRODUCTION - 25,095 atoms GB
> >> ---------------------------------------
> >>
> >> 1 x GTX680: | ns/day = 2.90 seconds/ns =
> >>29755.05
> >>
> >>
> >>
> >>
> >>
> >> Dne Tue, 26 Feb 2013 17:14:01 +0100 Scott Le Grand
> >><varelse2005.gmail.com
> >> >
> >> napsal/-a:
> >>
> >> > As an side, go run JAC NVE in SPFP mode...
> >> >
> >> > If you get ~75+ ns/day, you're running at 1.05+ GHz...
> >> >
> >> > Otherwise, something's up. And I second what Ross is saying - just
> >>sit
> >> > back and ride Pixel's Law. In the mid-term, I think I'll get JAC to
> >>200+
> >> > ns/day with a couple GTX Titans once I get the time to optimize GPU to
> >> > GPU
> >> > communication...
> >> >
> >> >
> >> >
> >> > On Mon, Feb 25, 2013 at 5:46 PM, Ross Walker <ross.rosswalker.co.uk>
> >> > wrote:
> >> >
> >> >> Hi Filip,
> >> >>
> >> >> I think you are worrying too much hear. Firstly you are referring to
> >>the
> >> >> double precision clock rate and not the single precision clock. AMBER
> >> >> stopped relying on the double precision side of things and switched
> >>to
> >> >> fixed point accumulation with the release of the GTX680 and K10.
> >>Second
> >> >> the stock single precision clock will be faster than the K20X so you
> >>can
> >> >> expect performance to be better than the K20X. It also has more cores
> >> >> active 'I think', don't have the specs here or internet access to
> >>check
> >> >> right now.
> >> >>
> >> >> Thirdly, the boost clock. AMBER pretty much runs the entire GPU flat
> >>out
> >> >> ALL the time. The boost clock is only useful, as with CPUs, when you
> >>are
> >> >> only using a fraction of the cores. In the case of GPUs unless you
> >>are
> >> >> running very small atom counts this is unlikely to happen so even if
> >>the
> >> >> boost clock was supported it wouldn't do you any good.
> >> >>
> >> >> In short, I wouldn't worry about it. Let's just wait and see how it
> >> >> truly
> >> >> performs when the "vaporware" actually turns up.
> >> >>
> >> >> All the best
> >> >> Ross
> >> >>
> >> >>
> >> >> On 2/25/13 2:39 PM, "filip fratev" <filipfratev.yahoo.com> wrote:
> >> >>
> >> >> >Hi all,
> >> >> >I received some tests performed. Here is the comparison between
> >>LuxMax
> >> >> >results obtained by GTX660 under Linux and Windows, respectively:
> >> >> >
> >> >> >http://img22.imageshack.us/img22/1279/luxmarkubuntu1204.png
> >> >> >http://img692.imageshack.us/img692/9647/luxmarkwin7.png
> >> >> >
> >> >> >According to these results the GTX660 works at 1071Mhz, thus the
> >>Boost
> >> >> >speed and the results between Linux and Windows are similar.
> >> >> >
> >> >> >However, Nvidia answered me that the GTX Titan core speed under
> >>Linux
> >> >> >will be 837MHz and about the boost technology this: "unfortunately
> >>no,
> >> >> >boost 1.0/2.0 are only supported on windows."
> >> >> >Personally I trust on the above tests:)
> >> >> >If they really caped their GTX GPU's under Linux to the base clock
> >> >> >presumably only the BIOS hack option will be possible, which
> >>is....:)
> >> >> >
> >> >> >Regards,
> >> >> >Filip
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >________________________________
> >> >> > From: Aron Broom <broomsday.gmail.com>
> >> >> >To: filip fratev <filipfratev.yahoo.com>; AMBER Mailing List
> >> >> ><amber.ambermd.org>
> >> >> >Sent: Saturday, February 23, 2013 10:12 PM
> >> >> >Subject: Re: [AMBER] GTX Titan was finally released
> >> >> >
> >> >> >Just as another note, I checked out the AMBER output from running
> >>on a
> >> >> >GTX570,
> >> >> >
> >> >> >|------------------- GPU DEVICE INFO --------------------
> >> >> >|
> >> >> >| CUDA Capable Devices Detected: 1
> >> >> >| CUDA Device ID in use: 0
> >> >> >| CUDA Device Name: GeForce GTX 570
> >> >> >| CUDA Device Global Mem Size: 1279 MB
> >> >> >| CUDA Device Num Multiprocessors: 15
> >> >> >| CUDA Device Core Freq: 1.46 GHz
> >> >> >|
> >> >> >|--------------------------------------------------------
> >> >> >
> >> >> >So in that case the Core Freq reported is indeed the correct one,
> >>even
> >> >> >though the GTX570 has two lower clock speeds it runs at depending on
> >> >> load
> >> >> >(810 MHz, and 101 MHz)
> >> >> >
> >> >> >I know with the 500 series, the available nVidia tools for linux
> >>will
> >> >> >least
> >> >> >allow you to set the device to maintain the highest clock speeds
> >> >> >regardless
> >> >> >of load. I have NOT done that in the above case, but if such a
> >>thing
> >> >> is
> >> >> >possible for the 600 series, it might be worth looking at. Sadly
> >>the
> >> >> tool
> >> >> >is only easily usable if you have a display connected although if
> >>you
> >> >> >google "Axel Kohlmeyer" and go to his homepage there are some
> >> >> suggestions
> >> >> >on installing these tools on a typical server where you can fake a
> >> >> >display.
> >> >> >
> >> >> >~Aron
> >> >> >
> >> >> >On Sat, Feb 23, 2013 at 2:33 PM, filip fratev
> >><filipfratev.yahoo.com>
> >> >> >wrote:
> >> >> >
> >> >> >> Hi Ross, Aron and all,
> >> >> >> Thanks for your detail answers!!
> >> >> >>
> >> >> >> So, it seems that nobody know whether Nvidia
> >> >> >> support the boost speed even on GTX680. Moreover, because the core
> >> >> >>speed is
> >> >> >> wrongly (I hope) printed as in the case of Amber 12 as well in all
> >> >> >> benchmark
> >> >> >> applications, we can see the difference only if compare the
> >>GTX680 to
> >> >> >>K10
> >> >> >> (1
> >> >> >> GPU) where we can see 37% performance increase (JAC), which can
> >>comes
> >> >> >>only
> >> >> >> from the
> >> >> >> core/memory clock.
> >> >> >>
> >> >> >> Ross, please ask Nvidia about these issues.
> >> >> >> I've already asked them but don't believe that will receive any
> >> >> adequate
> >> >> >> answer.
> >> >> >> I also asked several users but nobody knows and they told me that
> >> >> Nvidia
> >> >> >> never
> >> >> >> said something about their Boost technology under Linux.
> >> >> >> Thus, at this point I think that we can trust
> >> >> >> only to your information.
> >> >> >>
> >> >> >> Regards,
> >> >> >> Filip
> >> >> >>
> >> >> >>
> >> >> >> ________________________________
> >> >> >> From: Ross Walker <ross.rosswalker.co.uk>
> >> >> >> To: filip fratev <filipfratev.yahoo.com>; AMBER Mailing List <
> >> >> >> amber.ambermd.org>
> >> >> >> Sent: Saturday, February 23, 2013 6:45 AM
> >> >> >> Subject: Re: [AMBER] GTX Titan was finally released
> >> >> >>
> >> >> >> Hi Filip,
> >> >> >>
> >> >> >>
> >> >> >> >As you know I plan to purchase few GTX Titans:)
> >> >> >> >but I am not sure actually at what speed they will run: 836, 876
> >>or
> >> >> 993
> >> >> >> >Mhz?
> >> >> >> >It seems that by default (80C target) the Titan
> >> >> >> >runs under Windows only on the maximal core speed (around 1Ghz)
> >>not
> >> >> the
> >> >> >> >boost
> >> >> >> >one. It goes back to 836 only if the temperature rises above 80C
> >>but
> >> >> >>with
> >> >> >> >100%
> >> >> >> >fan speed this looks almost impossible. At least this is what I
> >>saw
> >> >> >>from
> >> >> >> >the
> >> >> >> >reviews.
> >> >> >>
> >> >> >> No idea since I am still waiting for NVIDIA to actually send me a
> >> >> >> development card to try this with. I guess the Titan's will be
> >> >> vaporware
> >> >> >> for a while. I am intrigued to know about how the clock speed will
> >> >> work
> >> >> >> and I am waiting for NVIDIA engineering to get back to me with a
> >> >> >> definitive answer. Note the Titan can also be run in two modes
> >>from
> >> >> >>what I
> >> >> >> gather. One with the DP cores turned down and the SP cores
> >>clocked up
> >> >> >> (Gaming mode) and one where it turns on all the DP cores and
> >>clocks
> >> >> down
> >> >> >> the single precision (CUDA mode). Note AMBER was retooled for the
> >> >> GK104
> >> >> >> chip to not use double precision anymore. It uses a combination of
> >> >> >>single
> >> >> >> and fixed precision which we worked very hard to tune to
> >>match/better
> >> >> >>the
> >> >> >> SPDP accuracy. Thus it is entirely possible that one will actually
> >> >> want
> >> >> >>to
> >> >> >> run the Titan cards in gaming mode when running AMBER. Of course
> >> >> this is
> >> >> >> entirely speculation until I lay my hands on one. The thermal
> >>window
> >> >> >>also
> >> >> >> has potential issues for 4 GPU boxes but there may end up being a
> >> >> hack
> >> >> >>to
> >> >> >> disable the down clocking and allow temps over 80C. Note most
> >>cards I
> >> >> >>have
> >> >> >> (GTX680s) run around 90C right now. SDSC runs it's machine room at
> >> >> 85F
> >> >> >>in
> >> >> >> order to save power - since disks and CPUs don't care if the room
> >>is
> >> >> 85F
> >> >> >> vs 60F. This might be a different story if the GPUs throttle
> >>based on
> >> >> >> temperature but I guess we'll just have to wait and see.
> >> >> >>
> >> >> >> >
> >> >> >> >I was also horrified to see that many GTX680
> >> >> >> >(and other cards) users complain that under Linux their cards
> >>run at
> >> >> >>only
> >> >> >> >about
> >> >> >> >700Mhz core speed instead of 1Ghz. What is your experience with
> >>GTX
> >> >> >>680?
> >> >> >> >I was also wondering whether the GTX680 use the
> >> >> >> >boost clock during the Amber calculations or the just the base
> >>one?
> >> >> >> >
> >> >> >>
> >> >> >> I think this is just speculation. When you run AMBER with a
> >>GTX680 it
> >> >> >> prints the following:
> >> >> >>
> >> >> >>
> >> >> >> |------------------- GPU DEVICE INFO --------------------
> >> >> >> |
> >> >> >> | CUDA Capable Devices Detected: 1
> >> >> >> | CUDA Device ID in use: 0
> >> >> >> | CUDA Device Name: GeForce GTX 680
> >> >> >> | CUDA Device Global Mem Size: 2047 MB
> >> >> >> | CUDA Device Num Multiprocessors: 8
> >> >> >> | CUDA Device Core Freq: 0.71 GHz
> >> >> >> |
> >> >> >> |--------------------------------------------------------
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> But this is a query that occurs at the very beginning of a run
> >>before
> >> >> >>any
> >> >> >> CUDA kernels have been run. I believe that when unloaded the 680
> >>in
> >> >> >>Linux
> >> >> >> clocks down to 705MHz to save power. When you stress it hard it
> >> >> >> automatically clocks up the frequency. I am not sure if there is
> >>way
> >> >> to
> >> >> >> check this though while the card is under load. Certainly the
> >> >> >>performance
> >> >> >> we see would be what it is if the clock speed was only 705MHz. I
> >>am
> >> >> >>asking
> >> >> >> NVIDIA engineering to clarify though.
> >> >> >>
> >> >> >> >Finally, what is the performance difference of
> >> >> >> >pmemdCuda under Linux and Cygwin?
> >> >> >>
> >> >> >> Never tried and I very much doubt you'll be able to get pmemd.cuda
> >> >> >> compiled under cygwin. Cygwin emulates things through the cygwin
> >>dll
> >> >> and
> >> >> >> so you'd need a cygwin compatible version of the nvidia compiler
> >>I'd
> >> >> >> expect.
> >> >> >>
> >> >> >> Note have a native Windows version of pmemd.cuda but never
> >>released
> >> >> the
> >> >> >> binary since the performance is about half that of what it is on
> >> >> Linux
> >> >> >>due
> >> >> >> to a bug in cuda 4.2 under windows that limited performance. cuda
> >>3
> >> >> >>showed
> >> >> >> good performance under windows but you can't use that with AMBER
> >>12.
> >> >> We
> >> >> >> haven't had time to get back to looking at this with cuda 5
> >> >> >>unfortunately.
> >> >> >>
> >> >> >> All the best
> >> >> >> Ross
> >> >> >>
> >> >> >> /\
> >> >> >> \/
> >> >> >> |\oss Walker
> >> >> >>
> >> >> >> ---------------------------------------------------------
> >> >> >> | Assistant Research Professor |
> >> >> >> | San Diego Supercomputer Center |
> >> >> >> | Adjunct Assistant Professor |
> >> >> >> | Dept. of Chemistry and Biochemistry |
> >> >> >> | University of California San Diego |
> >> >> >> | NVIDIA Fellow |
> >> >> >> | http://www.rosswalker.co.uk | http://www.wmd-lab.org |
> >> >> >> | Tel: +1 858 822 0854 | EMail:- ross.rosswalker.co.uk |
> >> >> >> ---------------------------------------------------------
> >> >> >>
> >> >> >> Note: Electronic Mail is not secure, has no guarantee of delivery,
> >> >> may
> >> >> >>not
> >> >> >> be read every day, and should not be used for urgent or sensitive
> >> >> >>issues.
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> _______________________________________________
> >> >> >> AMBER mailing list
> >> >> >> AMBER.ambermd.org
> >> >> >> http://lists.ambermd.org/mailman/listinfo/amber
> >> >> >> _______________________________________________
> >> >> >> AMBER mailing list
> >> >> >> AMBER.ambermd.org
> >> >> >> http://lists.ambermd.org/mailman/listinfo/amber
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> >--
> >> >> >Aron Broom M.Sc
> >> >> >PhD Student
> >> >> >Department of Chemistry
> >> >> >University of Waterloo
> >> >> >_______________________________________________
> >> >> >AMBER mailing list
> >> >> >AMBER.ambermd.org
> >> >> >http://lists.ambermd.org/mailman/listinfo/amber
> >> >> >_______________________________________________
> >> >> >AMBER mailing list
> >> >> >AMBER.ambermd.org
> >> >> >http://lists.ambermd.org/mailman/listinfo/amber
> >> >>
> >> >>
> >> >>
> >> >> _______________________________________________
> >> >> AMBER mailing list
> >> >> AMBER.ambermd.org
> >> >> http://lists.ambermd.org/mailman/listinfo/amber
> >> >>
> >> > _______________________________________________
> >> > AMBER mailing list
> >> > AMBER.ambermd.org
> >> > http://lists.ambermd.org/mailman/listinfo/amber
> >> >
> >> > __________ Informace od ESET NOD32 Antivirus, verze databaze 8053
> >> > (20130226) __________
> >> >
> >> > Tuto zpravu proveril ESET NOD32 Antivirus.
> >> >
> >> > http://www.eset.cz
> >> >
> >> >
> >> >
> >>
> >>
> >> --
> >> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
> >> http://www.opera.com/mail/
> >>
> >> _______________________________________________
> >> AMBER mailing list
> >> AMBER.ambermd.org
> >> http://lists.ambermd.org/mailman/listinfo/amber
> >> _______________________________________________
> >> AMBER mailing list
> >> AMBER.ambermd.org
> >> http://lists.ambermd.org/mailman/listinfo/amber
> >> _______________________________________________
> >> AMBER mailing list
> >> AMBER.ambermd.org
> >> http://lists.ambermd.org/mailman/listinfo/amber
> >>
> >_______________________________________________
> >AMBER mailing list
> >AMBER.ambermd.org
> >http://lists.ambermd.org/mailman/listinfo/amber
>
>
>
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
>
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Thu Mar 14 2013 - 12:30:02 PDT
