Ps. please read
http://ambermd.org/gpus/ in it's entirety - it has a lot
of information on there about how to run effectively on single and
multiple gpus.
On 11/23/14, 6:01 PM, "Ross Walker" <ross.rosswalker.co.uk> wrote:
>No no no no...
>
>Nobody use SPXP right now please!!! - This is for DEVELOPERS only!
>
>Compile to SPFP as default - your performance should match the GTX780
>numbers.
>
>SLI has nothing to do with AMBER multi-GPU performance. - You should turn
>off X11 and run the GPU Peer to Peer check and select pairs of GPUs that
>can communicate at FULL speed using peer to peer - Note which motherboard
>do you have? No single socket motherboards (except the CirraScale P2P
>breakout system) will get you x16 speed across 4 GPUs - so putting 4 GPUs
>in such a system pretty much limits you to multiple single GPU runs.
>
>All the best
>Ross
>
>
>
>On 11/23/14, 4:32 PM, "Scott Le Grand" <varelse2005.gmail.com> wrote:
>
>>Also you need to compile to SPXP (not the default SPFP)...
>>
>>We're working on this... This is a real curveball they threw us this
>>round. The worst-case scenario is stabilizing SPXP and leaving SPFP as
>>is...
>>
>>
>>On Sun, Nov 23, 2014 at 4:31 PM, Scott Le Grand <varelse2005.gmail.com>
>>wrote:
>>
>>> Turn off SLI...
>>>
>>>
>>> On Sun, Nov 23, 2014 at 4:22 PM, Parker de Waal <Parker.deWaal.vai.org>
>>> wrote:
>>>
>>>> Hi Scott,
>>>>
>>>> After putting together the machine and running the benchmark suite
>>>>I've
>>>> noticed that my performance is significantly worse than your numbers.
>>>> Specifically, while running 2x cards in SLI mode the FactorIX-NVE I'm
>>>>only
>>>> getting:
>>>>
>>>> | Average timings for last 1000 steps:
>>>> | Elapsed(s) = 5.19 Per Step(ms) = 5.19
>>>> | ns/day = 33.28 seconds/ns = 2596.11
>>>>
>>>> Interestingly I get the almost the same number when running with a
>>>>single
>>>> gpu.
>>>>
>>>> I'm wondering if I set this up correctly as I've never built an SLI
>>>> machine before. However, I have noticed that having SLI on ($
>>>> nvidia-xconfig --sli=on) disables gpup2p (according the gpup2p
>>>>checker).
>>>>
>>>> Are there settings you are supposed to specify to get that kind of
>>>> performance?
>>>>
>>>> Best,
>>>> Parker
>>>> ________________________________________
>>>> From: Scott Le Grand [varelse2005.gmail.com]
>>>> Sent: Monday, November 17, 2014 5:42 PM
>>>> To: AMBER Mailing List
>>>> Subject: Re: [AMBER] 4x GTX 980
>>>>
>>>> GTX 980 is well on its way to provide blowout AMBER numbers despite
>>>>the
>>>> claims of its hopeful competitors (Really, will they ever learn?
>>>> Nah...)...
>>>>
>>>> It didn't do so out the door because the performance a of key HW
>>>> instruction was crippled relative to Kepler (llrintf).
>>>>
>>>> In the near-term future, either NVIDIA will address the above or we
>>>>will
>>>> adopt a workaround to stabilize SPXP and make it the preferred
>>>>operational
>>>> mode for GTX9xx cards. In fact, if you're using a thermostat, I
>>>>*think*
>>>> you can run JAC NVE in SPXP today at ~185 ns/day out of the box.
>>>>This
>>>>is
>>>> of course the *correct* mode for directly comparing performance of
>>>> unstable
>>>> crappy MD code, but at least our crappy unstable mode is
>>>>deterministic.
>>>>
>>>> Scott
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Sun, Nov 16, 2014 at 8:16 PM, James Maier <jimbo.maier.gmail.com>
>>>> wrote:
>>>>
>>>> > **Please note that all experiences I've cited are for GTX 780s.
>>>> >
>>>> > On Sun, Nov 16, 2014 at 11:13 PM, James Maier
>>>><jimbo.maier.gmail.com>
>>>> > wrote:
>>>> >
>>>> > > Hi Parker,
>>>> > >
>>>> > > For the case, you can't go wrong with a HAF or almost anything
>>>>with a
>>>> fan
>>>> > > on the side. We went with the "Thermaltake Overseer" for our
>>>>cluster;
>>>> > it's
>>>> > > a bit cheaper than the HAF and has a built-in SATA dock. It's a
>>>>pretty
>>>> > neat
>>>> > > case.
>>>> > > Not that it matters, but the blue lights look exquisite against
>>>>the
>>>> green
>>>> > > GTX logo.
>>>> > >
>>>> > >
>>>>
>>>>http://scanmail.trustwave.com/?c=129&d=jvrq1G8mSq5JFVaMS-53aR6yoV3GwD4h
>>>>a
>>>>Wghp9jzIA&u=http%3a%2f%2fwww%2enewegg%2ecom%2fProduct%2fProduct%2easpx%
>>>>3
>>>>fItem%3dN82E16811133194
>>>> > >
>>>> > > We actually moved the fan that's pre-installed on the front to the
>>>> side,
>>>> > > and put this "bgears b-Blaster" fan in the front instead:
>>>> > >
>>>> > >
>>>>
>>>>http://scanmail.trustwave.com/?c=129&d=jvrq1G8mSq5JFVaMS-53aR6yoV3GwD4h
>>>>a
>>>>Wwko9r_IA&u=http%3a%2f%2fwww%2enewegg%2ecom%2fProduct%2fProduct%2easpx%
>>>>3
>>>>fitem%3dN82E16835132022
>>>> > >
>>>> > > 120 mm, 103 CFM, not too expensive. We've not had any problems
>>>>with
>>>> them
>>>> > > so far after about 9 months.
>>>> > >
>>>> > > As for the GPU, we've found the reference blower design to be more
>>>> > > effective when trying to squeeze four GPUs right next to each
>>>>other. I
>>>> > > believe the blower style can generate higher (negative) pressure
>>>>or
>>>> > > something, because the ACX, while great on its own, seems to do a
>>>> > terrible
>>>> > > job when something's directly occluding it. Our ACX GPUs were also
>>>> higher
>>>> > > clocked though, so that's an important caveat to anything I wish I
>>>> could
>>>> > > tell you more definitively.
>>>> > >
>>>> > > But don't take my word for it; consult some BitCoin mining
>>>> forums---they
>>>> > > seem to know all about this stuff. I believe a forum like that is
>>>> where I
>>>> > > first read something about the reference style being better for >2
>>>> GPUs,
>>>> > > which seems to match our experience.
>>>> > >
>>>> > > As Scott has cautioned, "superclocked is a 'super' way to get
>>>> inaccurate
>>>> > > results." Sticking with a normal clockspeed will likely save
>>>> headaches,
>>>> > > esp. as the 980s are a new design.
>>>> > >
>>>> > > The other concern is power; you need a power supply with a rail
>>>>with
>>>> > > enough wattage for all cards or with separate rails that all can
>>>> drive a
>>>> > > card. I'm not sure if there is a single PSU that can
>>>>"technically"
>>>>(on
>>>> > > paper, following all stats to the letter) drive four GTX780s
>>>>(require
>>>> 42
>>>> > > Amps each, I've only been able to find +12v with 133.3 Amps, plus
>>>> about
>>>> > 5 V
>>>> > > through each PCI slot). If someone knows of an adequate PSU,
>>>>please
>>>> > correct
>>>> > > me.
>>>> > >
>>>> > > In that regard, the GTX 980s look promising due to their lower
>>>>power
>>>> > draw.
>>>> > > There are only 2 issues I can think of atm:
>>>> > >
>>>> > > (1) 980s are "provisionally" supported by AMBER. This may not be
>>>>an
>>>> > issue,
>>>> > > and according to
>>>>
>>>>http://scanmail.trustwave.com/?c=129&d=j_rq1PpOS9l3jkIDZU511Jlwn3ZzLHUR
>>>>s
>>>>H2mJkZJLg&u=http%3a%2f%2fambermd%2eorg%2fgpus
>>>> there's an optimization coming.
>>>> > > (2) 980s are immature and thus the silicon is not likely as
>>>>stable
>>>>as,
>>>> > > say, a 780 or a Titan.
>>>> > >
>>>> > > I'm guessing Ross, Scott, or anyone who has gotten their hands on
>>>>some
>>>> > > 980s can chime in more about the above points.
>>>> > >
>>>> > > HTH,
>>>> > >
>>>> > > James
>>>> > >
>>>> > > On Sun, Nov 16, 2014 at 9:28 PM, Parker de Waal <
>>>> Parker.deWaal.vai.org>
>>>> > > wrote:
>>>> > >
>>>> > >> Hi Everyone,
>>>> > >>
>>>> > >> I apologize for the off topic post, however I'm currently
>>>>building a
>>>> 4x
>>>> > >> GTX 980 machine and was wondering if anyone had experience or
>>>> insight on
>>>> > >> picking a proper case/fans. I'm worried that 4 cards will lead to
>>>> > >> overheating in most traditional cases.
>>>> > >>
>>>> > >> Additionally I'm wondering about the different fans on the GTX
>>>>980
>>>> > cards,
>>>> > >> specifically the reference fan blower vs. the ACX 2.0 fans on the
>>>> EVGA
>>>> > >> cards.
>>>> > >>
>>>> > >> Any insight would be extremely valuable.
>>>> > >>
>>>> > >> Best,
>>>> > >> Parker
>>>> > >> _______________________________________________
>>>> > >> AMBER mailing list
>>>> > >> AMBER.ambermd.org
>>>> > >>
>>>>
>>>>http://scanmail.trustwave.com/?c=129&d=j_rq1PpOS9l3jkIDZU511Jlwn3ZzLHUR
>>>>s
>>>>CyuIBtKfg&u=http%3a%2f%2flists%2eambermd%2eorg%2fmailman%2flistinfo%2fa
>>>>m
>>>>ber
>>>> > >>
>>>> > >
>>>> > >
>>>> > _______________________________________________
>>>> > AMBER mailing list
>>>> > AMBER.ambermd.org
>>>> >
>>>>
>>>>http://scanmail.trustwave.com/?c=129&d=j_rq1PpOS9l3jkIDZU511Jlwn3ZzLHUR
>>>>s
>>>>CyuIBtKfg&u=http%3a%2f%2flists%2eambermd%2eorg%2fmailman%2flistinfo%2fa
>>>>m
>>>>ber
>>>> >
>>>> _______________________________________________
>>>> AMBER mailing list
>>>> AMBER.ambermd.org
>>>>
>>>>
>>>>http://scanmail.trustwave.com/?c=129&d=j_rq1PpOS9l3jkIDZU511Jlwn3ZzLHUR
>>>>s
>>>>CyuIBtKfg&u=http%3a%2f%2flists%2eambermd%2eorg%2fmailman%2flistinfo%2fa
>>>>m
>>>>ber
>>>> _______________________________________________
>>>> AMBER mailing list
>>>> AMBER.ambermd.org
>>>> http://lists.ambermd.org/mailman/listinfo/amber
>>>>
>>>
>>>
>>_______________________________________________
>>AMBER mailing list
>>AMBER.ambermd.org
>>http://lists.ambermd.org/mailman/listinfo/amber
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Sun Nov 23 2014 - 18:30:03 PST
