Thank you Ross.
I have a 40K atom system running with charmm cutoffs (fswitch 1.0-1.2 nm)
and HMR that scales tolerably well to 4 GPUs (see below) on an NVIDIA dgx1,
so I was hoping that things would be OK to 8 GPS (and of course we don't
need a lot of the special DL features on the dgx1). These are the P1000
GPUS, so I don't know if that makes a difference. I don't at the moment
have a machine where I can test 4x GTX 1080's as the GTX node that I have
is 2x2.
1-GPU: 190 ns/day
2-GPU: 250 ns/day (67 % efficiency)
4-GPU: 350 ns/day (46 % efficiency)
For a system with 250K atoms, the efficiency is about the same (71% to 2
GPUs and 54% to 4 GPUs).
Thank you again for your help,
Chris.
On Mon, Feb 6, 2017 at 1:56 PM, Ross Walker <ross.rosswalker.co.uk> wrote:
> Hi Chris,
>
> While AMBER will run 8 way P2P you won't see performance improvement for
> PME runs. Only large scale GB runs such as the 20K atom nucleosome test
> will scale to 8 GPUs. PME is blocked waiting on a decent FFT library from
> NVIDIA that can scale to multiple GPUs for the typical size of FFT grids
> AMBER uses. The only advantage you would get from going to an 8 way P2P
> system is the flexibility it would give you in scheduling 2 way P2P runs -
> you could use any 2 GPUs rather than having to make sure they were on the
> same PCI-E domain, or better on the same PLX switch. Note even things like
> the 10 and 20 P2P systems (cocolink etc) use PLX switches so although the
> P2P domain spans all GPUs there are still bottlenecks between banks of
> them. See:
>
> http://exxactcorp.com/blog/exploring-the-complexities-of-
> pcie-connectivity-and-peer-to-peer-communication/ <
> http://exxactcorp.com/blog/exploring-the-complexities-of-
> pcie-connectivity-and-peer-to-peer-communication/>
>
> My advice would be to stick to the existing 8 way boxes and run 8 x 1 GPU
> or 4 x 2 GPU runs (4 way P2P is possible but not very efficient - better to
> run 2 x 2 GPU runs). Note REMD runs will happily run across 8 GPUs if you
> have at least 8 replicas.
>
> All the best
> Ross
>
> > On Feb 6, 2017, at 15:31, Chris Neale <candrewn.gmail.com> wrote:
> >
> > Dear Users:
> >
> > We would like to build a node with 8 Pascal GTX-1080 GPUs capable of
> > running peer-to-peer for a single job (pmemd) utilizing all 8 GPUs. I see
> > an Exxact stystem linked on the amber homepage:
> > https://exxactcorp.com/index.php/solution/solu_detail/207 but I am not
> > entirely clear whether "Supports up to 8x Double Width GPUs" implies
> 8-way
> > peer-to-peer GPU communication.
> >
> > We have already one node with 8 GPUs but it is arranged as 2x4 such that
> a
> > single job can only use 4 GPUs efficiently and I am not sure if there are
> > yet motherboards that can handle 8-way GPU runs (or if they are yet
> > somewhat economical).
> >
> > Also, were such a machine available, will the same pmemd code that runs
> > well on 4 GPUs also run well on 8 GPUs or would code changes be required?
> >
> > Thank you for your help,
> > Chris.
> > _______________________________________________
> > AMBER mailing list
> > AMBER.ambermd.org
> > http://lists.ambermd.org/mailman/listinfo/amber
>
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
>
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Mon Feb 06 2017 - 13:30:03 PST
