>>
>> Yes if one effectively wants to run essentially replica exchange
>> simulations then yes but one still has to have the queuing system allocate
>> full nodes so sharing nodes is not possible which is what the original
>> poster of this thread was implying.
>>
>
> Yeah, sharing a node across multiple users or job types is tricky. Amber
> doesn't help with the general case either, and rightly so. The job
> scheduler / user has to be involved in getting the locality organised
AMBER on CPU won't help in this case but AMBER GPU will but in order for it to work it needs the GPUs setting to process exclusive mode.
nvidia-smi -c 3
In this mode pmemd.cuda can detect if a GPU is in use and will cycle through the GPUs in a node until it finds a free one. If one only ever plans to run single GPU jobs with AMBER this is the mode i'd recommend setting the GPUs to (in /etc/rc.local at boot) and then one doesn't need to worry about CUDA_VISIBLE_DEVICES. The down side is it prevents P2P communication between GPUs with AMBER so one can't run multi-gpu jobs if throughput is a priority. It also doesn't play well with other codes that share GPUs amongst threads so in a mixed use cluster one has to leave the GPUs in default and rely on the user or scheduler to hand out unused GPUs.
>> See the following:
>>
>> http://ambermd.org/gpus/#Running <http://ambermd.org/gpus/#Running>
>>
>> in summary
>>
>> export CUDA_VISIBLE_DEVICES=0
>> $AMBERHOME/bin/pmemd.cuda -O -i mdin.1 -o mdout.1 ...
>>
>> export CUDA_VISIBLE_DEVICES=1
>> $AMBERHOME/bin/pmemd.cuda -O -i mdin.2 -o mdout.2 ...
>>
>> etc.
>>
>> Most GPU aware queues should set CUDA_VISIBLE_DEVICES for you depending on
>> the specific GPU(s) you are allocated.
>>
>
> Ok, cool. For reference, equivalent gromacs runs on a node with 4N
> hyperthreads and 2 GPUs are achieved with
>
> source $GMXHOME/bin/GMXRC
>
> gmx mdrun -nt 2N -pin on -pinoffset 0 -gpu_id 0 -deffnm first
>
> gmx mdrun -nt 2N -pin on -pinoffset N -gpu_id 1 -deffnm second
>
> Or you can use CUDA_VISIBLE_DEVICES to handle that aspect.
>
> Other examples at
> http://manual.gromacs.org/documentation/2016.3/user-guide/mdrun-performance.html <http://manual.gromacs.org/documentation/2016.3/user-guide/mdrun-performance.html>
This is very helpful, thanks.
All the best
Ross
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Tue Jul 25 2017 - 19:00:03 PDT
