Which is to say if MVAPICH, Mellanox, and NVIDIA do the right thing(tm),
the new P2P pathway will seamlessly become the MPI pathway using P2P
intra-node and RDMA inter-node. If they can't, then I would stick to fat
nodes and decent interconnect between them for REMD runs.
On Tue, Oct 8, 2013 at 11:13 AM, Ross Walker <ross.rosswalker.co.uk> wrote:
> We **are** really doing much work for infiniband --> We **aren't** really
> doing much work for infiniband
>
>
>
>
>
> On 10/8/13 11:02 AM, "Ross Walker" <ross.rosswalker.co.uk> wrote:
>
> >Dear Peter,
> >
> >This optimization will be for INTRAnode improvements - using peer to peer
> >- essentially the code will test if the two GPUs can communicate via peer
> >to peer and if it can it will use that otherwise it will full back on
> >generic MPI. We are really doing much work for infiniband since it
> >massively increases the cost of machines for marginal gain. Right now you
> >need GPUs to be on the same IOH channel in order to communicate. For most
> >motherboards this means if you put 4 GPUs in them you can run them in two
> >specific pairs of two. We are waiting on motherboards (and or multi-GPU
> >systems) that have PIIX switches that allow all 4 GPUs to communicate via
> >RDMA. As far as I know none of these systems 'publicly' exist yet. If you
> >are considering a very large purchase you might want to arrange an NDA
> >with one or two major vendors and motherboard manufactures to see if you
> >can get timeframes out of them.
> >
> >For now the 4 GPU systems, such as those being sold by exxact can run 2 x
> >2 GPU runs using peer to peer, where the 2 gpus used for each job are
> >connected to the same CPU socket.
> >
> >All the best
> >Ross
> >
> >
> >
> >On 10/8/13 10:55 AM, "peter.stauffert.boehringer-ingelheim.com"
> ><peter.stauffert.boehringer-ingelheim.com> wrote:
> >
> >>Hi all,
> >>
> >>currently multi GPU Amber runs do not scale very well.
> >>
> >>But as Ross Walker wrote last Friday, this will be significantly improved
> >>in the next Amber version.
> >>
> >>Which kind of multi GPU runs will be improved, intranode multi GPU runs
> >>or internode GPU runs?
> >>
> >>We are on the way to configure a new HPC environment with GPUs and the
> >>questions is whether we should configure many nodes with only one or two
> >>GPUs or is it better to install many GPUs (4 to 8) in only a few servers?
> >>
> >>There is the development of GPU Direct RDMA with mvapich2 by Nvidia,
> >>Mellanox, and OSU (Prof. D.K.Panda) to improve the internode GPU-GPU
> >>communication with the recommendation to install GPU and InfiniBand
> >>adapter on the same I/O hub (so only few GPUs should b installed in a
> >>server), but there also many improvements for the intranode GPU-GPU
> >>communications on the way.
> >>
> >>What are your recommendations to configure GPU servers to run multi GPU
> >>amber runs with the next version of Amber?
> >>
> >>Thanks
> >>
> >>Peter
> >>
> >>Dr. Peter Stauffert
> >>Boehringer Ingelheim Pharma GmbH & Co. KG
> >>mailto:peter.stauffert.boehringer-ingelheim.com
> >>
> >>_______________________________________________
> >>AMBER mailing list
> >>AMBER.ambermd.org
> >>http://lists.ambermd.org/mailman/listinfo/amber
> >
> >
> >
> >_______________________________________________
> >AMBER mailing list
> >AMBER.ambermd.org
> >http://lists.ambermd.org/mailman/listinfo/amber
>
>
>
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
>
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Tue Oct 08 2013 - 11:30:11 PDT
