Hi Siwakorn,
Yes when running the GPU MPI code it has to create an inter-GPU process for the communication. This is prohibited when running the GPUs in exclusive mode hence the error.
Asking why your sysadmin prefers the GPUs to be in exclusive mode is a reasonable question - they just need to properly configure a queuing system that treats GPUs like a consumable. It's pretty easy to do. Assuming though that they don't want to do that you are stuck running single GPU runs I'm afraid. That said the acceleration from multi-GPU is minimal these days with AMBER, unless running very large GB simulations, so there is not much to gain. You are better off running two independent simulations from different starting conditions to improve your overall sampling.
All the best
Ross
> On May 10, 2023, at 04:44, Siwakorn Sukharom via AMBER <amber.ambermd.org> wrote:
>
> When I run pmemd.cuda.MPI on gpu that set the compute mode to exclusive process, i got this error
>
>
> cudaDeviceEnablePeerAccess CUDA-capable device(s) is/are busy or unavailable
> cudaDeviceEnablePeerAccess CUDA-capable device(s) is/are busy or unavailable
>
> but it can run fine on default mode. However, the problem is the system admin prefer to set gpu mode to exclusive process and ask me if i can make amber to run on exclusive process or not
>
> ________________________________
>
> Disclaimer:
>
> This e-mail and any files transmitted with it may contain confidential and proprietary information of the National Science and Technology Development Agency (NSTDA), Thailand. They are intended solely for the use of the addressed individuals or entities. If you are not the intended recipient, you are required to immediately delete this e-mail and its contents from your system. Any disclosure, distribution, or action based upon the contents of this e-mail is strictly prohibited. Any views or opinions presented in this e-mail are solely those of the sender and do not necessarily represent those of NSTDA. NSTDA does not accept any responsibility for the content of this message or the consequences of any actions taken on the basis of the information provided. NSTDA accepts no liability for any damage caused by any virus or malware which may be inserted in this e-mail during transmission.
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> https://urldefense.com/v3/__http://lists.ambermd.org/mailman/listinfo/amber__;!!Mih3wA!CfD4I2YaOnpj8kUEX_3CRfma5NKESyOii2JQYGF4uiADJD3Bq30JZLesgE55hAP1CKKau3jJelk$
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Wed May 10 2023 - 05:00:02 PDT
