Just like Jerrano wrote, the numbering of the device listed by nvidia-smi does not seem to correspond with the number in export CUDA_VISIBLE_DEVICES.
For example, processes 1 and 2 were listed now as running and process 0 was not running.
When I typed setenv CUDA_VISIBLE_DEVICES 0, I was told that all the GPUS were busy.
When I typed setenv CUDA_VISIBLE_DEVICES 1, then I was able to run pmemd.cuda on process 0.
I am not sure why this is, but I currently have pmemd.cuda running on three different GPUS.
JYI, I prevented pmemd.cuda from running on more than one GPU by logging in as root and executing the following commands:
nvidia-smi -pm 1
nvidia-smi -c 3
This sets Persistence and Compute Exclusive Modes as written on the ambermd.org/gpus page. The problem is this setting is lost on a reboot, so one should add this settings to the system's startup scripts.
Best, Kristin
-----Original Message-----
From: Jerrano Bowleg [mailto:jlb1692.msstate.edu]
Sent: Friday, June 16, 2017 4:04 PM
To: AMBER Mailing List <amber.ambermd.org>
Subject: Re: [AMBER] Specific GPU for running pmemd.cuda
Hello, Kristin,
I've also had similar problems when using export CUDA_VISIBLE_DEVICS=1. It seems the gpu id you specified does not match the one in your system. I noticed for instance that when I specified device 0, it corresponded to gpu 1. The number matching is off.
I did not find a solution to this problem.
On Fri, Jun 16, 2017 at 2:48 PM, Krantzman, Kristin D <KrantzmanK.cofc.edu>
wrote:
> [image: Boxbe]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.boxbe.com_ov
> erview&d=DwICAg&c=7MSSWy9Bs2yocjNQzurxOQ&r=rDr-VDoXCKpr5ouXXYSZkHF6q5g
> U19w8m7W8WZccZ7U&m=AB7QIDOW2yGL0jH_ichMEt561EuTTuZTC1dL-U7Qmh8&s=RCh6Y
> Yl_ZlYA0ZP53YY55QqgQW3cnkkc0wwZN9T27_Q&e= > This message is eligible
> for Automatic Cleanup! (KrantzmanK.cofc.edu) Add cleanup rule
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.boxbe.com_po
> pup-3Furl-3Dhttps-253A-252F-252Fwww.boxbe.com-252Fcleanup-253Fkey-253D
> jp-25252F1VxEJUlMvI10P4AvIhnaNjWZT-25252BCcmNQrUNu7W2Gw-25253D-2526tok
> en-253D-25252Bk-25252BXVkLivG19eMrWyIlANAm5HdfA-25252FeYIsY-25252F828t
> -25252Biv73UX23rwR6pjgCcrKf2iVnD1FDbQvFRvTo-25252FT57-25252BxPI3rlU6Ud
> 9AZGrLQnM-25252BN40hjgE1GVO1IYzzasA3-25252B0DXMnw7Pm8jlQI-25252FYU-252
> 53D-26tc-5Fserial-3D30936656575-26tc-5Frand-3D786610339-26utm-5Fsource
> -3Dstf-26utm-5Fmedium-3Demail-26utm-5Fcampaign-3DANNO-5FCLEANUP-5FADD-
> 26utm-5Fcontent-3D001&d=DwICAg&c=7MSSWy9Bs2yocjNQzurxOQ&r=rDr-VDoXCKpr
> 5ouXXYSZkHF6q5gU19w8m7W8WZccZ7U&m=AB7QIDOW2yGL0jH_ichMEt561EuTTuZTC1dL
> -U7Qmh8&s=tNr7i7IWSqs105wnx54NpzqFCG4MnXs8v4xXEF8kRJQ&e= >
> | More info
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__blog.boxbe.com_ge
> neral_boxbe-2Dautomatic-2Dcleanup-3Ftc-5Fserial-3D30936656575-26tc-5Fr
> and-3D786610339-26utm-5Fsource-3Dstf-26utm-5Fmedium-3Demail-26utm-5Fca
> mpaign-3DANNO-5FCLEANUP-5FADD-26utm-5Fcontent-3D001&d=DwICAg&c=7MSSWy9
> Bs2yocjNQzurxOQ&r=rDr-VDoXCKpr5ouXXYSZkHF6q5gU19w8m7W8WZccZ7U&m=AB7QID
> OW2yGL0jH_ichMEt561EuTTuZTC1dL-U7Qmh8&s=BhjubwcQbDGabIiuksLtyNb_0STKae
> 3DWndc14XzeTo&e= >
>
> Greetings Amber Users,
> I am trying to use pmemd.CUDA on different GPU processors using the
> export CUDA_VISIBLE_DEVICES command.
> However, when I type export CUDA_VISIBLE_DEVICES="1" and then try to
> execute pmemd.cuda, I get the error that all CUDA-capable devices are
> busy are unavailable. When I type nvidia-smi, it seems that only one
> of the three GPUS is in use. Does anyone have experience with solving
> this problem?
> Thank you in advance.
> Best, Kristin
> Kristin D. Krantzman
> Professor, Department of Chemistry and Biochemistry College of
> Charleston Charleston, SC 29424
> (843) 953-3378
>
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.ambermd.org_
> mailman_listinfo_amber&d=DwICAg&c=7MSSWy9Bs2yocjNQzurxOQ&r=rDr-VDoXCKp
> r5ouXXYSZkHF6q5gU19w8m7W8WZccZ7U&m=AB7QIDOW2yGL0jH_ichMEt561EuTTuZTC1d
> L-U7Qmh8&s=Y2-hzSQQqsejPvf63DIc2eGFGmSWb3o3F5mqhdmtScM&e=
>
>
--
Jerrano L. Bowleg
PhD Student
Department of Chemistry
Gwaltney Lab- Hand Lab 1126
Mississippi State University
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.ambermd.org_mailman_listinfo_amber&d=DwICAg&c=7MSSWy9Bs2yocjNQzurxOQ&r=rDr-VDoXCKpr5ouXXYSZkHF6q5gU19w8m7W8WZccZ7U&m=AB7QIDOW2yGL0jH_ichMEt561EuTTuZTC1dL-U7Qmh8&s=Y2-hzSQQqsejPvf63DIc2eGFGmSWb3o3F5mqhdmtScM&e=
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Fri Jun 16 2017 - 15:30:02 PDT
