Dear Jason:
thank you for this awesome information! I did take a look at that benchmark
page first, but it doesn't contain the word "nvlink" at all.... is there a
synonym for nvlink or something? Sorry if this is obvious, but I've had
very little experience with nvlink, only on a dgx which has faster GPUs
than any other nodes I have access to so I am not sure if its the GPUs or
the nvlink or something else (besides shared PCIe bus, which you explained
very nicely) in the dgx that is helping the scaling.
Thank you again,
Chris.
On Tue, Mar 28, 2017 at 8:09 PM, Jason Swails <jason.swails.gmail.com>
wrote:
> On Tue, Mar 28, 2017 at 8:38 PM, Chris Neale <candrewn.gmail.com> wrote:
>
> > Dear Users:
> >
> > I have some questions about hardware for a cpu/gpu purchase and I would
> > appreciate some insight. Basically, I don't really understand the ins and
> > outs of peer-to-peer GPU communication and I need some help.
> >
>
> Peer-to-peer (P2P) communicates data directly from one GPU memory bank to
> another through a shared PCIe bus. Thus, the bottleneck here is the PCIe
> bandwidth. Without P2P, GPUs share memory by having to go through system
> memory, so it has to go through the PCIe bus, to the CPU (possibly to RAM
> if it doesn't all fit in cache), back through the PCIe bus to the other
> GPU.
>
> This is *much* slower than P2P communication and as GPUs get faster, this
> bottleneck becomes more and more pronounced. For modern cards, you lose a
> *lot* of efficiency when you parallelize without the P2P shortcut (because
> the GPUs are getting faster much more quickly than the bottleneck). In
> fact, the benchmarks for Amber don't even quote multi-GPU runs without P2P
> anymore: http://ambermd.org/gpus/benchmarks.htm
>
> 1. What hardware specifications are required to enable peer-to-peer GPU
> > communication for pmemd?
> >
>
> Your motherboard needs to support P2P. I believe there are some
> motherboards today that support 4-way P2P, but most common is to have boxes
> that have 4 GPUs with two separate PCIe buses. So you can do two separate
> 2-way P2P runs.
>
> Basically what you need to look for is P2P support in the motherboard and
> make sure that the GPUs you hook up are hooked up on the slots on the
> shared bus.
>
>
> > 2. Given 2 or 4 Pascal P100's or GTX-1080's how much is gpu-to-gpu nvlink
> > (without possible gpu-to-cpu nvlink) going to affect performance of
> > multi-gpu runs? -- Note that I realize P100's are excessivly expensive
> per
> > flop compared to GTX-1080's.
> >
>
> It looks like the benchmark page may give you some clue:
> http://ambermd.org/gpus/benchmarks.htm
>
> HTH,
> Jason
>
> --
> Jason M. Swails
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
>
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Tue Mar 28 2017 - 19:30:02 PDT
