mpirun -np # specifies the number of threads that you want to start. You
only want to start threads on GPUs, so # should be the number of GPUs that
you have.
How to make sure that you actually lock those 4 threads to the 4 different
GPUs is another issue altogether, and that is system dependent. I would
suggest that you consult your MPI implementation's documentation in order to
find out how to compose a hostfile that will tell your MPI wrapper where to
put which threads. If everything runs on the same node I *think* you should
be fine... It prints in the output file which thread is assigned to which
GPU, though, so that should help.
Also note previous posts that indicate there may be issues with > 8 GPUs on
a single node. Furthermore, I don't think that each GPU can take advantage
of 16 PCIe lanes at the same time (so you can't have 12 PCIex16 interfaces
running simultaneously), so I would expect you to take a huge performance
hit if you tried running pmemd.cuda.MPI on 12 GPUs all on the same node.
This is especially exacerbated by the fact that the load-balancing is
static, so those GPUs not communicating at optimum speeds won't be given
less work to do as a result. So in fact, the simulation will run only as
fast as the slowest GPU.
As for the infiniband question, that's handled at the MPI level as well. If
you installed an MPI that takes advantage of infiniband (i.e. mvapich,
mvapich2, etc.), then it should automatically take advantage of that
existing hardware. Note that there are different levels of infiniband (SDR,
DDR, QDR). As I understand it, you really should have QDR or better in
order to really take advantage of multiple GPUs.
Hope this helps,
Jason
On Tue, Mar 15, 2011 at 5:04 PM, Haipeng Wei <haipeng.wei.gmail.com> wrote:
> Hi Ross,
>
> Thanks a lot for your reply. Could you please give me further help on the
> following questions? I read the website you told me, but I am still not
> clear.
>
> 1. How should I write the command line if I have 1 node with 12 gpus?
> mpirun -np 12? does it mean 12 nodes or 1 nodes with 12 gpus? do I need to
> assign one by one or the test will detect all gpus
> 2, If I have multi-node, each node has 2 gpus or more, do I need to
> recompile with IB? or how could I use IB?
>
> Thanks
>
> Haipeng
>
>
> On Tue, Mar 15, 2011 at 3:13 PM, Ross Walker <ross.rosswalker.co.uk>
> wrote:
>
> > Hi Haipeng,
> >
> > Please see the following webpage: http://ambermd.org/gpus/#Running
> >
> > All the best
> > Ross
> >
> > > -----Original Message-----
> > > From: Haipeng Wei [mailto:haipeng.wei.gmail.com]
> > > Sent: Tuesday, March 15, 2011 3:09 PM
> > > To: amber.ambermd.org
> > > Subject: [AMBER] how to run amber on mutli-gpu/muti-nodes
> > >
> > > Hi,
> > >
> > > Could you please give me some help on how to run amber on
> > > multi-gpu/multi-nodes?
> > >
> > > For example, what the command will be if I want to run on 4 nodes(each
> > > node
> > > has two GPUs).
> > >
> > > Thanks a lot
> > > _______________________________________________
> > > AMBER mailing list
> > > AMBER.ambermd.org
> > > http://lists.ambermd.org/mailman/listinfo/amber
> >
> >
> > _______________________________________________
> > AMBER mailing list
> > AMBER.ambermd.org
> > http://lists.ambermd.org/mailman/listinfo/amber
> >
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
>
--
Jason M. Swails
Quantum Theory Project,
University of Florida
Ph.D. Candidate
352-392-4032
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Tue Mar 15 2011 - 17:30:04 PDT
