Hi Matt,
Unfortunately the way AMBER does P2P does not work with the GPUs in process exclusive mode. I would suggest going to the sysadmin and having them modify the queuing system to allow either the user to specify the mode in the job submission script or have some logic in there to set the mode depending if the user is requesting more than 1 GPU or if they add some specific request - e.g. P2P or some other keyword in the job submission script. We used to do this at SDSC and it worked great. It can go in the prolog of the queuing system which runs as root and thus can issue nvidia-smi -c XXX commands. Eva Hocks had this working well at SDSC with SGE, your sysadmin may want to try contacting her and see if she'll share the config scripts.
All the best
Ross
> On Jan 26, 2017, at 17:12, Matthew Harrigan <harrigan.stanford.edu> wrote:
>
> Hi all,
>
>
> I'm trying to run a multi-gpu simulation. Our cluster's GPUs are set to process exclusive mode. Per http://archive.ambermd.org/201405/0145.html it is not surprising that this fails with the message
>
>
> cudaDeviceEnablePeerAccess all CUDA-capable devices are busy or unavailable
> cudaDeviceEnablePeerAccess all CUDA-capable devices are busy or unavailable
>
>
> However, our cluster administrator doesn't want to set the compute mode to default *but also* points out that p2p works in general in process exclusive mode. Specifically, if I run p2pBandwidthLatencyTest from the CUDA SDK samples, it shows p2p is enabled
>
>
> $ ./p2pBandwidthLatencyTest
> [P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
> Device: 0, Tesla K80, pciBusID: 93, pciDeviceID: 0, pciDomainID:0
> Device: 1, Tesla K80, pciBusID: 94, pciDeviceID: 0, pciDomainID:0
> Device=0 CAN Access Peer Device=1
> Device=1 CAN Access Peer Device=0
> <snip>
>
>
> Is there some way amber can be modified to use p2p in process exclusive mode?
>
>
> Matt
>
>
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Fri Jan 27 2017 - 05:00:03 PST
