On Tue, Mar 28, 2017 at 8:38 PM, Chris Neale <candrewn.gmail.com> wrote:
> Dear Users:
>
> I have some questions about hardware for a cpu/gpu purchase and I would
> appreciate some insight. Basically, I don't really understand the ins and
> outs of peer-to-peer GPU communication and I need some help.
>
Peer-to-peer (P2P) communicates data directly from one GPU memory bank to
another through a shared PCIe bus. Thus, the bottleneck here is the PCIe
bandwidth. Without P2P, GPUs share memory by having to go through system
memory, so it has to go through the PCIe bus, to the CPU (possibly to RAM
if it doesn't all fit in cache), back through the PCIe bus to the other GPU.
This is *much* slower than P2P communication and as GPUs get faster, this
bottleneck becomes more and more pronounced. For modern cards, you lose a
*lot* of efficiency when you parallelize without the P2P shortcut (because
the GPUs are getting faster much more quickly than the bottleneck). In
fact, the benchmarks for Amber don't even quote multi-GPU runs without P2P
anymore: http://ambermd.org/gpus/benchmarks.htm
1. What hardware specifications are required to enable peer-to-peer GPU
> communication for pmemd?
>
Your motherboard needs to support P2P. I believe there are some
motherboards today that support 4-way P2P, but most common is to have boxes
that have 4 GPUs with two separate PCIe buses. So you can do two separate
2-way P2P runs.
Basically what you need to look for is P2P support in the motherboard and
make sure that the GPUs you hook up are hooked up on the slots on the
shared bus.
> 2. Given 2 or 4 Pascal P100's or GTX-1080's how much is gpu-to-gpu nvlink
> (without possible gpu-to-cpu nvlink) going to affect performance of
> multi-gpu runs? -- Note that I realize P100's are excessivly expensive per
> flop compared to GTX-1080's.
>
It looks like the benchmark page may give you some clue:
http://ambermd.org/gpus/benchmarks.htm
HTH,
Jason
--
Jason M. Swails
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Tue Mar 28 2017 - 19:30:02 PDT
