Hi Chris,
While AMBER will run 8 way P2P you won't see performance improvement for PME runs. Only large scale GB runs such as the 20K atom nucleosome test will scale to 8 GPUs. PME is blocked waiting on a decent FFT library from NVIDIA that can scale to multiple GPUs for the typical size of FFT grids AMBER uses. The only advantage you would get from going to an 8 way P2P system is the flexibility it would give you in scheduling 2 way P2P runs - you could use any 2 GPUs rather than having to make sure they were on the same PCI-E domain, or better on the same PLX switch. Note even things like the 10 and 20 P2P systems (cocolink etc) use PLX switches so although the P2P domain spans all GPUs there are still bottlenecks between banks of them. See:
http://exxactcorp.com/blog/exploring-the-complexities-of-pcie-connectivity-and-peer-to-peer-communication/ <
http://exxactcorp.com/blog/exploring-the-complexities-of-pcie-connectivity-and-peer-to-peer-communication/>
My advice would be to stick to the existing 8 way boxes and run 8 x 1 GPU or 4 x 2 GPU runs (4 way P2P is possible but not very efficient - better to run 2 x 2 GPU runs). Note REMD runs will happily run across 8 GPUs if you have at least 8 replicas.
All the best
Ross
> On Feb 6, 2017, at 15:31, Chris Neale <candrewn.gmail.com> wrote:
>
> Dear Users:
>
> We would like to build a node with 8 Pascal GTX-1080 GPUs capable of
> running peer-to-peer for a single job (pmemd) utilizing all 8 GPUs. I see
> an Exxact stystem linked on the amber homepage:
> https://exxactcorp.com/index.php/solution/solu_detail/207 but I am not
> entirely clear whether "Supports up to 8x Double Width GPUs" implies 8-way
> peer-to-peer GPU communication.
>
> We have already one node with 8 GPUs but it is arranged as 2x4 such that a
> single job can only use 4 GPUs efficiently and I am not sure if there are
> yet motherboards that can handle 8-way GPU runs (or if they are yet
> somewhat economical).
>
> Also, were such a machine available, will the same pmemd code that runs
> well on 4 GPUs also run well on 8 GPUs or would code changes be required?
>
> Thank you for your help,
> Chris.
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Mon Feb 06 2017 - 13:00:04 PST
