Hi ET,
As the code stands right now there is NO advantage for having the pled
chip there. The plan is to re-write the code to use peer-to-peer
communication between GPUs instead of MPI which is what is currently used.
This should have an advantage and should not be disadvantaged by having 2
x 16 connected together with a PCI-E switch to a single x16 pci-E channel.
However, how well that will actually work in practice will have to wait
until the code is done.
All the best
Ross
On 6/22/13 10:36 PM, "ET" <sketchfoot.gmail.com> wrote:
>Just to clarify:
>
>There is only small to medium improvement in using a plex chip versus
>x8/x8/x8/x8 ?
>
>
>On 22 June 2013 21:31, Scott Le Grand <varelse2005.gmail.com> wrote:
>
>> For a fat node, the plex chip may be enough to drive GPUDirect/P2P.
>>I've
>> more or less given up on MPI 2.0 so that's the only step forward from
>> here. Currently verifying this with nvidia though before proceeding
>> On Jun 22, 2013 1:27 PM, "ET" <sketchfoot.gmail.com> wrote:
>>
>> > Thanks for that Ross, I hadn't consider that.Some more food for
>>thought.
>> >
>> >
>> > On 22 June 2013 21:12, Ross Walker <rosscwalker.gmail.com> wrote:
>> >
>> > > You need to be looking at DUAL socket motherboards if you want 4-way
>> x16
>> > > support. A single CPU does not have enough bandwidth (pins) to
>>support
>> > true
>> > > 4 by x16 - they cheat using pxii switches.
>> > >
>> > > All the Exxact machines on the Amber website are dual socket by quad
>> slot
>> > > x16. Of course this makes them more expensive than the budget single
>> > socket
>> > > systems you can build for sub $1000 (exc GPU cost).
>> > >
>> > > All the best
>> > > Ross
>> > >
>> > >
>> > >
>> > > On Jun 22, 2013, at 12:45, ET <sketchfoot.gmail.com> wrote:
>> > >
>> > > > Looks like the Asus P9X79-E WS is for you then Scott! :) Haven't
>> seen
>> > > many
>> > > > (if any!) boards with that anount of bandwidth so far.
>> > > >
>> > > > br,
>> > > > g
>> > > >
>> > > >
>> > > > On 22 June 2013 19:31, Scott Le Grand <varelse2005.gmail.com>
>>wrote:
>> > > >
>> > > >> It may be overkill now but I'm planning to revisit the Multi GPU
>> code
>> > in
>> > > >> the near future and that's why I need a motherboard that can
>>really
>> > take
>> > > >> advantage of it.
>> > > >> On Jun 22, 2013 10:12 AM, "ET" <sketchfoot.gmail.com> wrote:
>> > > >>
>> > > >>> Hi,
>> > > >>>
>> > > >>> Thanks v much for you specs Divi. :) I've been debating with
>>myself
>> > as
>> > > my
>> > > >>> board as it looks good and has a very nice spec. From what I've
>> read,
>> > > the
>> > > >>> only problems with it is the higher than average power draw.
>> > > >>>
>> > > >>> .Scott. I believe the board runs in x8/x8/x8/x8 for a 4 GPU
>>config
>> -
>> > so
>> > > >>> effectively PCI2 2.0 x16 rate. Would this present any
>>problems, if
>> > you
>> > > >>> were running the serial GPU code, From what I read on the AMBER
>>GPU
>> > > >>> hardware page, this is more important for the parallel GPU code?
>> > > Though,
>> > > >> I
>> > > >>> imagine having 4x serial ruins going simultaneously would also
>>tax
>> > the
>> > > >> GPU
>> > > >>> to CPU interface, though how much I'm not sure.
>> > > >>>
>> > > >>> Apparently, if you are going intel, you can only acheive PCIe
>>3.0
>> > using
>> > > >> at
>> > > >>> least a Sandy Bridge-E or ivy bridge CPU in a socket 155. Please
>> > > correct
>> > > >> me
>> > > >>> if I have understood this incorrectly though.
>> > > >>>
>> > > >>> http://www.enthusiastpc.net/articles/00003/3.aspx
>> > > >>>
>> > > >>>
>> > > >>> A socket 2011 proposition would be the Asus P9X79-E WES which
>>has
>> 2x
>> > > PLX
>> > > >>> PEX 8747 chips so can run at x16/x16/x16/x16 with four GPUs
>> > > >>>
>> > > >>> https://www.asus.com/Motherboards/P9X79E_WS/#specifications
>> > > >>>
>> > > >>> However, I'm unsure whether this is overkill for running 4xGPUs
>> doing
>> > > >> AMBER
>> > > >>> serial code.
>> > > >>>
>> > > >>> What do you guys think?
>> > > >>>
>> > > >>> br,
>> > > >>> g
>> > > >>>
>> > > >>>
>> > > >>>
>> > > >>> On 22 June 2013 16:15, Scott Le Grand <varelse2005.gmail.com>
>> wrote:
>> > > >>>
>> > > >>>> Does this MB support full p2p at 16x PCIE Gen 3 speeds between
>> all 4
>> > > >>> GPUs?
>> > > >>>> On Jun 21, 2013 4:09 PM, "Divi/GMAIL" <dvenkatlu.gmail.com>
>> wrote:
>> > > >>>>
>> > > >>>>>
>> > > >>>>> ET:
>> > > >>>>> I am using GA-Z77X-UP7 that has PLX chipset and supports 3rd
>> Gen
>> > > >>>> LGA1155
>> > > >>>>> socket. Bought together with 2 TITANS sometime in March.
>> > > >>>>> It has been running pretty stable 24/7 since then. I
>>thought of
>> > > >>> buying
>> > > >>>>> two more titans later to fill all four slots. With so much
>>mess
>> > > >> going
>> > > >>> on
>> > > >>>>> with TITANS, I put off that plan until the dust settles. You
>> might
>> > > >>> want
>> > > >>>> to
>> > > >>>>> check new 4th GEN cpus and supporting motherboards as the
>> Hardware
>> > > >> keep
>> > > >>>>> changing pretty rapidly these days.
>> > > >>>>>
>> > > >>>>> I have i5-processor with 16 GB ram and 256 GB SSD. All four
>> > PCI-E
>> > > >>>> lanes
>> > > >>>>> are X-16. It also has native X-16 link directly "hardwired" to
>> > > >>> CPU-lanes
>> > > >>>>> that will bypass PLX chipset, in case if you run single GPU.
>>This
>> > > >> might
>> > > >>>>> reduce a bit of latency but not much. I get 35ns/day on
>>FIX/NVE
>> > > >>> benchmark
>> > > >>>>> bypassing PLX chipset, but get about 34ns/day using PLX
>>chipset
>> (on
>> > > >>> TITAN
>> > > >>>>> of
>> > > >>>>> course!!). Not a deal breaker..
>> > > >>>>>
>> > > >>>>> Link below:
>> > > >>>>>
>> > > >>>>>
>>http://www.gigabyte.com/products/product-page.aspx?pid=4334#ov
>> > > >>>>>
>> > > >>>>> HTH
>> > > >>>>> Divi
>> > > >>>>>
>> > > >>>>> -----Original Message-----
>> > > >>>>> From: ET
>> > > >>>>> Sent: Thursday, June 20, 2013 8:18 PM
>> > > >>>>> To: AMBER Mailing List
>> > > >>>>> Subject: [AMBER] Anyone running machines with Quad GPU setups
>> > > >>>>>
>> > > >>>>> Hi all,
>> > > >>>>>
>> > > >>>>> I was looking at getting a new mobo to run a quad GPU system.
>>I
>> was
>> > > >>>>> wondering if anyone has done this. If you could post the
>>model &
>> > make
>> > > >>> of:
>> > > >>>>>
>> > > >>>>> 1) motherboard
>> > > >>>>> 2) CPU
>> > > >>>>> 3) RAM
>> > > >>>>> 4) Case
>> > > >>>>> 5) The aggregate estimate of ns in simulation you have run on
>> your
>> > > >>> setup
>> > > >>>>> without issue,
>> > > >>>>>
>> > > >>>>> I would be much obliged! :)
>> > > >>>>>
>> > > >>>>> br,
>> > > >>>>> g
>> > > >>>>> _______________________________________________
>> > > >>>>> AMBER mailing list
>> > > >>>>> AMBER.ambermd.org
>> > > >>>>> http://lists.ambermd.org/mailman/listinfo/amber
>> > > >>>>>
>> > > >>>>>
>> > > >>>>> _______________________________________________
>> > > >>>>> AMBER mailing list
>> > > >>>>> AMBER.ambermd.org
>> > > >>>>> http://lists.ambermd.org/mailman/listinfo/amber
>> > > >>>>>
>> > > >>>> _______________________________________________
>> > > >>>> AMBER mailing list
>> > > >>>> AMBER.ambermd.org
>> > > >>>> http://lists.ambermd.org/mailman/listinfo/amber
>> > > >>>>
>> > > >>> _______________________________________________
>> > > >>> AMBER mailing list
>> > > >>> AMBER.ambermd.org
>> > > >>> http://lists.ambermd.org/mailman/listinfo/amber
>> > > >>>
>> > > >> _______________________________________________
>> > > >> AMBER mailing list
>> > > >> AMBER.ambermd.org
>> > > >> http://lists.ambermd.org/mailman/listinfo/amber
>> > > >>
>> > > > _______________________________________________
>> > > > AMBER mailing list
>> > > > AMBER.ambermd.org
>> > > > http://lists.ambermd.org/mailman/listinfo/amber
>> > >
>> > > _______________________________________________
>> > > AMBER mailing list
>> > > AMBER.ambermd.org
>> > > http://lists.ambermd.org/mailman/listinfo/amber
>> > >
>> > _______________________________________________
>> > AMBER mailing list
>> > AMBER.ambermd.org
>> > http://lists.ambermd.org/mailman/listinfo/amber
>> >
>> _______________________________________________
>> AMBER mailing list
>> AMBER.ambermd.org
>> http://lists.ambermd.org/mailman/listinfo/amber
>>
>_______________________________________________
>AMBER mailing list
>AMBER.ambermd.org
>http://lists.ambermd.org/mailman/listinfo/amber
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Sun Jun 23 2013 - 17:30:02 PDT
