Re: [AMBER] 4x GTX 980

From: Ross Walker <ross.rosswalker.co.uk>
Date: Sun, 23 Nov 2014 18:02:57 -0800

Yes VERY dangerous. - Consider the X for experimental and assume that
right now results using it will NOT be publishable quality. It needs quite
a bit of work and testing to make it feasible and to quantify the
precision and performance.



On 11/23/14, 5:32 PM, "Parker de Waal" <Parker.deWaal.vai.org> wrote:

>Hah, fair enough.
>
>However that still begs the question: is it `dangerous` to use SPXP
>results? Can I trust anything out of this for now?
>
>Best,
>Parker
>________________________________________
>From: Scott Le Grand [varelse2005.gmail.com]
>Sent: Sunday, November 23, 2014 8:26 PM
>To: AMBER Mailing List
>Subject: Re: [AMBER] 4x GTX 980
>
>eXtreme!
>eXperimental!
>eXplosions!
>etc...
>
>Take your pick. I created it to replace the useless SPSP mode to see if I
>could relax precision further than SPFP does and still maintain NVE
>stability. It was a work in progress until GM204 made it suddenly
>relevant...
>
>I hope to get to it during Christmas break...
>
>
>On Sun, Nov 23, 2014 at 4:51 PM, Parker de Waal <Parker.deWaal.vai.org>
>wrote:
>
>> Thanks for the quick reply, these benches are much better!
>>
>> However that leads me to another question: I can't find any information
>>on
>> SPXP... is it `dangerous` to use SPXP results? Also what is SPXP? single
>> precision eXtreme point?
>>
>> Parker
>> ________________________________________
>> From: Scott Le Grand [varelse2005.gmail.com]
>> Sent: Sunday, November 23, 2014 7:32 PM
>> To: AMBER Mailing List
>> Subject: Re: [AMBER] 4x GTX 980
>>
>> Also you need to compile to SPXP (not the default SPFP)...
>>
>> We're working on this... This is a real curveball they threw us this
>> round. The worst-case scenario is stabilizing SPXP and leaving SPFP as
>> is...
>>
>>
>> On Sun, Nov 23, 2014 at 4:31 PM, Scott Le Grand <varelse2005.gmail.com>
>> wrote:
>>
>> > Turn off SLI...
>> >
>> >
>> > On Sun, Nov 23, 2014 at 4:22 PM, Parker de Waal
>><Parker.deWaal.vai.org>
>> > wrote:
>> >
>> >> Hi Scott,
>> >>
>> >> After putting together the machine and running the benchmark suite
>>I've
>> >> noticed that my performance is significantly worse than your numbers.
>> >> Specifically, while running 2x cards in SLI mode the FactorIX-NVE I'm
>> only
>> >> getting:
>> >>
>> >> | Average timings for last 1000 steps:
>> >> | Elapsed(s) = 5.19 Per Step(ms) = 5.19
>> >> | ns/day = 33.28 seconds/ns = 2596.11
>> >>
>> >> Interestingly I get the almost the same number when running with a
>> single
>> >> gpu.
>> >>
>> >> I'm wondering if I set this up correctly as I've never built an SLI
>> >> machine before. However, I have noticed that having SLI on ($
>> >> nvidia-xconfig --sli=on) disables gpup2p (according the gpup2p
>>checker).
>> >>
>> >> Are there settings you are supposed to specify to get that kind of
>> >> performance?
>> >>
>> >> Best,
>> >> Parker
>> >> ________________________________________
>> >> From: Scott Le Grand [varelse2005.gmail.com]
>> >> Sent: Monday, November 17, 2014 5:42 PM
>> >> To: AMBER Mailing List
>> >> Subject: Re: [AMBER] 4x GTX 980
>> >>
>> >> GTX 980 is well on its way to provide blowout AMBER numbers despite
>>the
>> >> claims of its hopeful competitors (Really, will they ever learn?
>> >> Nah...)...
>> >>
>> >> It didn't do so out the door because the performance a of key HW
>> >> instruction was crippled relative to Kepler (llrintf).
>> >>
>> >> In the near-term future, either NVIDIA will address the above or we
>>will
>> >> adopt a workaround to stabilize SPXP and make it the preferred
>> operational
>> >> mode for GTX9xx cards. In fact, if you're using a thermostat, I
>>*think*
>> >> you can run JAC NVE in SPXP today at ~185 ns/day out of the box.
>>This
>> is
>> >> of course the *correct* mode for directly comparing performance of
>> >> unstable
>> >> crappy MD code, but at least our crappy unstable mode is
>>deterministic.
>> >>
>> >> Scott
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Sun, Nov 16, 2014 at 8:16 PM, James Maier <jimbo.maier.gmail.com>
>> >> wrote:
>> >>
>> >> > **Please note that all experiences I've cited are for GTX 780s.
>> >> >
>> >> > On Sun, Nov 16, 2014 at 11:13 PM, James Maier
>><jimbo.maier.gmail.com>
>> >> > wrote:
>> >> >
>> >> > > Hi Parker,
>> >> > >
>> >> > > For the case, you can't go wrong with a HAF or almost anything
>>with
>> a
>> >> fan
>> >> > > on the side. We went with the "Thermaltake Overseer" for our
>> cluster;
>> >> > it's
>> >> > > a bit cheaper than the HAF and has a built-in SATA dock. It's a
>> pretty
>> >> > neat
>> >> > > case.
>> >> > > Not that it matters, but the blue lights look exquisite against
>>the
>> >> green
>> >> > > GTX logo.
>> >> > >
>> >> > >
>> >>
>>
>>http://scanmail.trustwave.com/?c=129&d=2Iny1NljvAaXtmtNQj3lbHdUQKJ_mBi8he
>>YVTSrY4g&u=http%3a%2f%2fwww%2enewegg%2ecom%2fProduct%2fProduct%2easpx%3fI
>>tem%3dN82E16811133194
>> >> > >
>> >> > > We actually moved the fan that's pre-installed on the front to
>>the
>> >> side,
>> >> > > and put this "bgears b-Blaster" fan in the front instead:
>> >> > >
>> >> > >
>> >>
>>
>>http://scanmail.trustwave.com/?c=129&d=2Iny1NljvAaXtmtNQj3lbHdUQKJ_mBi8he
>>IQSSjU4g&u=http%3a%2f%2fwww%2enewegg%2ecom%2fProduct%2fProduct%2easpx%3fi
>>tem%3dN82E16835132022
>> >> > >
>> >> > > 120 mm, 103 CFM, not too expensive. We've not had any problems
>>with
>> >> them
>> >> > > so far after about 9 months.
>> >> > >
>> >> > > As for the GPU, we've found the reference blower design to be
>>more
>> >> > > effective when trying to squeeze four GPUs right next to each
>> other. I
>> >> > > believe the blower style can generate higher (negative) pressure
>>or
>> >> > > something, because the ACX, while great on its own, seems to do a
>> >> > terrible
>> >> > > job when something's directly occluding it. Our ACX GPUs were
>>also
>> >> higher
>> >> > > clocked though, so that's an important caveat to anything I wish
>>I
>> >> could
>> >> > > tell you more definitively.
>> >> > >
>> >> > > But don't take my word for it; consult some BitCoin mining
>> >> forums---they
>> >> > > seem to know all about this stuff. I believe a forum like that is
>> >> where I
>> >> > > first read something about the reference style being better for
>>>2
>> >> GPUs,
>> >> > > which seems to match our experience.
>> >> > >
>> >> > > As Scott has cautioned, "superclocked is a 'super' way to get
>> >> inaccurate
>> >> > > results." Sticking with a normal clockspeed will likely save
>> >> headaches,
>> >> > > esp. as the 980s are a new design.
>> >> > >
>> >> > > The other concern is power; you need a power supply with a rail
>>with
>> >> > > enough wattage for all cards or with separate rails that all can
>> >> drive a
>> >> > > card. I'm not sure if there is a single PSU that can
>>"technically"
>> (on
>> >> > > paper, following all stats to the letter) drive four GTX780s
>> (require
>> >> 42
>> >> > > Amps each, I've only been able to find +12v with 133.3 Amps, plus
>> >> about
>> >> > 5 V
>> >> > > through each PCI slot). If someone knows of an adequate PSU,
>>please
>> >> > correct
>> >> > > me.
>> >> > >
>> >> > > In that regard, the GTX 980s look promising due to their lower
>>power
>> >> > draw.
>> >> > > There are only 2 issues I can think of atm:
>> >> > >
>> >> > > (1) 980s are "provisionally" supported by AMBER. This may not be
>>an
>> >> > issue,
>> >> > > and according to
>> >>
>>
>>http://scanmail.trustwave.com/?c=129&d=2Iny1NljvAaXtmtNQj3lbHdUQKJ_mBi8hb
>>QZSHiFtQ&u=http%3a%2f%2fambermd%2eorg%2fgpus
>> >> there's an optimization coming.
>> >> > > (2) 980s are immature and thus the silicon is not likely as
>>stable
>> as,
>> >> > > say, a 780 or a Titan.
>> >> > >
>> >> > > I'm guessing Ross, Scott, or anyone who has gotten their hands on
>> some
>> >> > > 980s can chime in more about the above points.
>> >> > >
>> >> > > HTH,
>> >> > >
>> >> > > James
>> >> > >
>> >> > > On Sun, Nov 16, 2014 at 9:28 PM, Parker de Waal <
>> >> Parker.deWaal.vai.org>
>> >> > > wrote:
>> >> > >
>> >> > >> Hi Everyone,
>> >> > >>
>> >> > >> I apologize for the off topic post, however I'm currently
>>building
>> a
>> >> 4x
>> >> > >> GTX 980 machine and was wondering if anyone had experience or
>> >> insight on
>> >> > >> picking a proper case/fans. I'm worried that 4 cards will lead
>>to
>> >> > >> overheating in most traditional cases.
>> >> > >>
>> >> > >> Additionally I'm wondering about the different fans on the GTX
>>980
>> >> > cards,
>> >> > >> specifically the reference fan blower vs. the ACX 2.0 fans on
>>the
>> >> EVGA
>> >> > >> cards.
>> >> > >>
>> >> > >> Any insight would be extremely valuable.
>> >> > >>
>> >> > >> Best,
>> >> > >> Parker
>> >> > >> _______________________________________________
>> >> > >> AMBER mailing list
>> >> > >> AMBER.ambermd.org
>> >> > >>
>> >>
>>
>>http://scanmail.trustwave.com/?c=129&d=2Iny1NljvAaXtmtNQj3lbHdUQKJ_mBi8he
>>URTiWG5Q&u=http%3a%2f%2flists%2eambermd%2eorg%2fmailman%2flistinfo%2fambe
>>r
>> >> > >>
>> >> > >
>> >> > >
>> >> > _______________________________________________
>> >> > AMBER mailing list
>> >> > AMBER.ambermd.org
>> >> >
>> >>
>>
>>http://scanmail.trustwave.com/?c=129&d=2Iny1NljvAaXtmtNQj3lbHdUQKJ_mBi8he
>>URTiWG5Q&u=http%3a%2f%2flists%2eambermd%2eorg%2fmailman%2flistinfo%2fambe
>>r
>> >> >
>> >> _______________________________________________
>> >> AMBER mailing list
>> >> AMBER.ambermd.org
>> >>
>> >>
>>
>>http://scanmail.trustwave.com/?c=129&d=2Iny1NljvAaXtmtNQj3lbHdUQKJ_mBi8he
>>URTiWG5Q&u=http%3a%2f%2flists%2eambermd%2eorg%2fmailman%2flistinfo%2fambe
>>r
>> >> _______________________________________________
>> >> AMBER mailing list
>> >> AMBER.ambermd.org
>> >>
>>
>>http://scanmail.trustwave.com/?c=129&d=2Iny1NljvAaXtmtNQj3lbHdUQKJ_mBi8he
>>URTiWG5Q&u=http%3a%2f%2flists%2eambermd%2eorg%2fmailman%2flistinfo%2fambe
>>r
>> >>
>> >
>> >
>> _______________________________________________
>> AMBER mailing list
>> AMBER.ambermd.org
>>
>>
>>http://scanmail.trustwave.com/?c=129&d=2Iny1NljvAaXtmtNQj3lbHdUQKJ_mBi8he
>>URTiWG5Q&u=http%3a%2f%2flists%2eambermd%2eorg%2fmailman%2flistinfo%2fambe
>>r
>>
>> _______________________________________________
>> AMBER mailing list
>> AMBER.ambermd.org
>>
>>http://scanmail.trustwave.com/?c=129&d=2Iny1NljvAaXtmtNQj3lbHdUQKJ_mBi8he
>>URTiWG5Q&u=http%3a%2f%2flists%2eambermd%2eorg%2fmailman%2flistinfo%2fambe
>>r
>>
>_______________________________________________
>AMBER mailing list
>AMBER.ambermd.org
>http://scanmail.trustwave.com/?c=129&d=2Iny1NljvAaXtmtNQj3lbHdUQKJ_mBi8heU
>RTiWG5Q&u=http%3a%2f%2flists%2eambermd%2eorg%2fmailman%2flistinfo%2famber
>_______________________________________________
>AMBER mailing list
>AMBER.ambermd.org
>http://lists.ambermd.org/mailman/listinfo/amber



_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Sun Nov 23 2014 - 18:30:02 PST
Custom Search