Hi Jan-Philip and Ross,
Thanks for the response. I suspect the performance drop I saw is most
likely due to running 2 jobs on the same card because I didn't specify
CUDA_VISIBLE_DEVICES in the first place. And, the same CUDA device ID was
reported in the output files.
However, setting CUDA_VISIBLE_DEVICES in the job script doesn't seem to
affect the device ID reported in mdout. With the variable set to 0 and 1
individually, CUDA DEVICE ID 0 is reported in both mdouts, even though
nvidia-smi seems to suggest both GPUs are being used.
Any thoughts on this? Thanks.
+------------------------------------------------------+
| NVIDIA-SMI 4.304.54 Driver Version: 304.54 |
|-------------------------------+----------------------+----------------------+
| GPU Name | Bus-Id Disp. | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla M2090 | 0000:19:00.0 Off | 0 |
| N/A N/A P0 151W / 225W | 4% 217MB / 5375MB | 99% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla M2090 | 0000:1A:00.0 Off | 0 |
| N/A N/A P0 175W / 225W | 6% 296MB / 5375MB | 99% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 12104 /home/gc603449/APPLICATIONS/amber12/bin/pmemd.cuda 204MB |
| 1 12108 /home/gc603449/APPLICATIONS/amber12/bin/pmemd.cuda 283MB |
+-----------------------------------------------------------------------------+
Best regards,
Guanglei
On Tue, Jan 28, 2014 at 12:03 PM, Ross Walker <ross.rosswalker.co.uk> wrote:
> Hi Guanglei,
>
> I suspect you are running both calculations on the same GPU. Either:
>
> 1) run 'nvidia-smi -c 3' to put the cards in process exclusive mode (you
> will need to be root to do this). This will then allow pmemd.cuda to
> automatically select GPUs based on their utilization.
>
> 2) set CUDA_VISIBLE_DEVICES. So for the first run you would do 'export
> CUDA_VISIBLE_DEVICES=0; $AMBERHOME/bin/pmemd.cuda -O ...' and for the
> second run 'export CUDA_VISIBLE_DEVICES=1; $AMBERHOME/bin/pmemd.cuda -O
> ...'
>
> Beyond that the other possibility is that you are still running the
> multi-gpu version of the code on a single GPU. In which case, ditch the
> mpirun and change the executable to pmemd.cuda and NOT pmemd.cuda.MPI.
>
> A side note for those that are interested. The next version of AMBER
> (AMBER 14) will include peer to peer support for multi-GPUs in pmemd. This
> provides MUCH better multi-GPU scaling as long as the GPUs are on the same
> PCI-E bus - typically one can get two GPUs on a single bus (architecture
> is in the pipeline that will allow 4). It also means things like GTX690
> and other dual-GPU cards will provide excellent scaling. Stay tuned - more
> details of how to enable this and supported hardware will be added to the
> AMBER GPU Website (http://ambermd.org/gpus/) as we approach release.
>
> All the best
> Ross
>
>
> On 1/28/14, 8:37 AM, "Guanglei Cui" <amber.mail.archive.gmail.com> wrote:
>
> >Dear AMBER users,
> >
> >I am doing some benchmark on a node with two M2090 cards. For my test
> >system (~26K atoms and NVT), I'll get 36.7ns/day (pmemd.cuda on 1 GPU) and
> >43.5ns/day (pmemd.cuda.MPI on both GPUs). So it makes sense to run two
> >separate simulations, 1 on each GPU. From what I read, amber12 GPU code
> >should perform almost equally well in such situations. However, I observe
> >a
> >performance drop (almost 50%). I have limited experience with the code. I
> >wonder if someone could give me some hints as to what might be causing the
> >performance degradation. I don't have a lot details on the hardware specs
> >of the node, but I can ask if certain factors are more important.
> >
> >Thanks in advance!
> >
> >Best regards,
> >--
> >Guanglei Cui
> >_______________________________________________
> >AMBER mailing list
> >AMBER.ambermd.org
> >http://lists.ambermd.org/mailman/listinfo/amber
>
>
>
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
>
--
Guanglei Cui
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Tue Jan 28 2014 - 11:30:02 PST
