No no no no...
Nobody use SPXP right now please!!! - This is for DEVELOPERS only!
Compile to SPFP as default - your performance should match the GTX780
numbers.
SLI has nothing to do with AMBER multi-GPU performance. - You should turn
off X11 and run the GPU Peer to Peer check and select pairs of GPUs that
can communicate at FULL speed using peer to peer - Note which motherboard
do you have? No single socket motherboards (except the CirraScale P2P
breakout system) will get you x16 speed across 4 GPUs - so putting 4 GPUs
in such a system pretty much limits you to multiple single GPU runs.
All the best
Ross
On 11/23/14, 4:32 PM, "Scott Le Grand" <varelse2005.gmail.com> wrote:
>Also you need to compile to SPXP (not the default SPFP)...
>
>We're working on this... This is a real curveball they threw us this
>round. The worst-case scenario is stabilizing SPXP and leaving SPFP as
>is...
>
>
>On Sun, Nov 23, 2014 at 4:31 PM, Scott Le Grand <varelse2005.gmail.com>
>wrote:
>
>> Turn off SLI...
>>
>>
>> On Sun, Nov 23, 2014 at 4:22 PM, Parker de Waal <Parker.deWaal.vai.org>
>> wrote:
>>
>>> Hi Scott,
>>>
>>> After putting together the machine and running the benchmark suite I've
>>> noticed that my performance is significantly worse than your numbers.
>>> Specifically, while running 2x cards in SLI mode the FactorIX-NVE I'm
>>>only
>>> getting:
>>>
>>> | Average timings for last 1000 steps:
>>> | Elapsed(s) = 5.19 Per Step(ms) = 5.19
>>> | ns/day = 33.28 seconds/ns = 2596.11
>>>
>>> Interestingly I get the almost the same number when running with a
>>>single
>>> gpu.
>>>
>>> I'm wondering if I set this up correctly as I've never built an SLI
>>> machine before. However, I have noticed that having SLI on ($
>>> nvidia-xconfig --sli=on) disables gpup2p (according the gpup2p
>>>checker).
>>>
>>> Are there settings you are supposed to specify to get that kind of
>>> performance?
>>>
>>> Best,
>>> Parker
>>> ________________________________________
>>> From: Scott Le Grand [varelse2005.gmail.com]
>>> Sent: Monday, November 17, 2014 5:42 PM
>>> To: AMBER Mailing List
>>> Subject: Re: [AMBER] 4x GTX 980
>>>
>>> GTX 980 is well on its way to provide blowout AMBER numbers despite the
>>> claims of its hopeful competitors (Really, will they ever learn?
>>> Nah...)...
>>>
>>> It didn't do so out the door because the performance a of key HW
>>> instruction was crippled relative to Kepler (llrintf).
>>>
>>> In the near-term future, either NVIDIA will address the above or we
>>>will
>>> adopt a workaround to stabilize SPXP and make it the preferred
>>>operational
>>> mode for GTX9xx cards. In fact, if you're using a thermostat, I
>>>*think*
>>> you can run JAC NVE in SPXP today at ~185 ns/day out of the box. This
>>>is
>>> of course the *correct* mode for directly comparing performance of
>>> unstable
>>> crappy MD code, but at least our crappy unstable mode is deterministic.
>>>
>>> Scott
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Nov 16, 2014 at 8:16 PM, James Maier <jimbo.maier.gmail.com>
>>> wrote:
>>>
>>> > **Please note that all experiences I've cited are for GTX 780s.
>>> >
>>> > On Sun, Nov 16, 2014 at 11:13 PM, James Maier <jimbo.maier.gmail.com>
>>> > wrote:
>>> >
>>> > > Hi Parker,
>>> > >
>>> > > For the case, you can't go wrong with a HAF or almost anything
>>>with a
>>> fan
>>> > > on the side. We went with the "Thermaltake Overseer" for our
>>>cluster;
>>> > it's
>>> > > a bit cheaper than the HAF and has a built-in SATA dock. It's a
>>>pretty
>>> > neat
>>> > > case.
>>> > > Not that it matters, but the blue lights look exquisite against the
>>> green
>>> > > GTX logo.
>>> > >
>>> > >
>>>
>>>http://scanmail.trustwave.com/?c=129&d=jvrq1G8mSq5JFVaMS-53aR6yoV3GwD4ha
>>>Wghp9jzIA&u=http%3a%2f%2fwww%2enewegg%2ecom%2fProduct%2fProduct%2easpx%3
>>>fItem%3dN82E16811133194
>>> > >
>>> > > We actually moved the fan that's pre-installed on the front to the
>>> side,
>>> > > and put this "bgears b-Blaster" fan in the front instead:
>>> > >
>>> > >
>>>
>>>http://scanmail.trustwave.com/?c=129&d=jvrq1G8mSq5JFVaMS-53aR6yoV3GwD4ha
>>>Wwko9r_IA&u=http%3a%2f%2fwww%2enewegg%2ecom%2fProduct%2fProduct%2easpx%3
>>>fitem%3dN82E16835132022
>>> > >
>>> > > 120 mm, 103 CFM, not too expensive. We've not had any problems with
>>> them
>>> > > so far after about 9 months.
>>> > >
>>> > > As for the GPU, we've found the reference blower design to be more
>>> > > effective when trying to squeeze four GPUs right next to each
>>>other. I
>>> > > believe the blower style can generate higher (negative) pressure or
>>> > > something, because the ACX, while great on its own, seems to do a
>>> > terrible
>>> > > job when something's directly occluding it. Our ACX GPUs were also
>>> higher
>>> > > clocked though, so that's an important caveat to anything I wish I
>>> could
>>> > > tell you more definitively.
>>> > >
>>> > > But don't take my word for it; consult some BitCoin mining
>>> forums---they
>>> > > seem to know all about this stuff. I believe a forum like that is
>>> where I
>>> > > first read something about the reference style being better for >2
>>> GPUs,
>>> > > which seems to match our experience.
>>> > >
>>> > > As Scott has cautioned, "superclocked is a 'super' way to get
>>> inaccurate
>>> > > results." Sticking with a normal clockspeed will likely save
>>> headaches,
>>> > > esp. as the 980s are a new design.
>>> > >
>>> > > The other concern is power; you need a power supply with a rail
>>>with
>>> > > enough wattage for all cards or with separate rails that all can
>>> drive a
>>> > > card. I'm not sure if there is a single PSU that can "technically"
>>>(on
>>> > > paper, following all stats to the letter) drive four GTX780s
>>>(require
>>> 42
>>> > > Amps each, I've only been able to find +12v with 133.3 Amps, plus
>>> about
>>> > 5 V
>>> > > through each PCI slot). If someone knows of an adequate PSU, please
>>> > correct
>>> > > me.
>>> > >
>>> > > In that regard, the GTX 980s look promising due to their lower
>>>power
>>> > draw.
>>> > > There are only 2 issues I can think of atm:
>>> > >
>>> > > (1) 980s are "provisionally" supported by AMBER. This may not be an
>>> > issue,
>>> > > and according to
>>>
>>>http://scanmail.trustwave.com/?c=129&d=j_rq1PpOS9l3jkIDZU511Jlwn3ZzLHURs
>>>H2mJkZJLg&u=http%3a%2f%2fambermd%2eorg%2fgpus
>>> there's an optimization coming.
>>> > > (2) 980s are immature and thus the silicon is not likely as stable
>>>as,
>>> > > say, a 780 or a Titan.
>>> > >
>>> > > I'm guessing Ross, Scott, or anyone who has gotten their hands on
>>>some
>>> > > 980s can chime in more about the above points.
>>> > >
>>> > > HTH,
>>> > >
>>> > > James
>>> > >
>>> > > On Sun, Nov 16, 2014 at 9:28 PM, Parker de Waal <
>>> Parker.deWaal.vai.org>
>>> > > wrote:
>>> > >
>>> > >> Hi Everyone,
>>> > >>
>>> > >> I apologize for the off topic post, however I'm currently
>>>building a
>>> 4x
>>> > >> GTX 980 machine and was wondering if anyone had experience or
>>> insight on
>>> > >> picking a proper case/fans. I'm worried that 4 cards will lead to
>>> > >> overheating in most traditional cases.
>>> > >>
>>> > >> Additionally I'm wondering about the different fans on the GTX 980
>>> > cards,
>>> > >> specifically the reference fan blower vs. the ACX 2.0 fans on the
>>> EVGA
>>> > >> cards.
>>> > >>
>>> > >> Any insight would be extremely valuable.
>>> > >>
>>> > >> Best,
>>> > >> Parker
>>> > >> _______________________________________________
>>> > >> AMBER mailing list
>>> > >> AMBER.ambermd.org
>>> > >>
>>>
>>>http://scanmail.trustwave.com/?c=129&d=j_rq1PpOS9l3jkIDZU511Jlwn3ZzLHURs
>>>CyuIBtKfg&u=http%3a%2f%2flists%2eambermd%2eorg%2fmailman%2flistinfo%2fam
>>>ber
>>> > >>
>>> > >
>>> > >
>>> > _______________________________________________
>>> > AMBER mailing list
>>> > AMBER.ambermd.org
>>> >
>>>
>>>http://scanmail.trustwave.com/?c=129&d=j_rq1PpOS9l3jkIDZU511Jlwn3ZzLHURs
>>>CyuIBtKfg&u=http%3a%2f%2flists%2eambermd%2eorg%2fmailman%2flistinfo%2fam
>>>ber
>>> >
>>> _______________________________________________
>>> AMBER mailing list
>>> AMBER.ambermd.org
>>>
>>>
>>>http://scanmail.trustwave.com/?c=129&d=j_rq1PpOS9l3jkIDZU511Jlwn3ZzLHURs
>>>CyuIBtKfg&u=http%3a%2f%2flists%2eambermd%2eorg%2fmailman%2flistinfo%2fam
>>>ber
>>> _______________________________________________
>>> AMBER mailing list
>>> AMBER.ambermd.org
>>> http://lists.ambermd.org/mailman/listinfo/amber
>>>
>>
>>
>_______________________________________________
>AMBER mailing list
>AMBER.ambermd.org
>http://lists.ambermd.org/mailman/listinfo/amber
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Sun Nov 23 2014 - 18:30:02 PST
