> Hi Ross,
> Is it possible to run pmemd.cuda.MPI with only 3 cards?
Not with the current release version. You have to use a power of 2 GPUs. This is being addressed for the next release and then you will be able to use any number of GPUs. For the moment though this code is not publically available I am afraid.
> I tried to do that but received this error:
> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
> APPLICATION TERMINATED WITH THE EXIT STRING: Hangup (signal 1)
>
> The problem is that I observed today that the 4th GPU of the second
> GTX590 in fact worse the results or give only up to 5% contribution.
> For instance, the scaling between the docking program Hex and Amber is
> almost the same -1.48x between 2GPU's of GTX590. When I used the 3th
> GPUs in Hex the scale is fine 1.87x compared to 1GPU (Also similar to
> ACEMD results -1 to 3 cards gives nearly 2x speed up). However, the 4th
> GPUs worse the calculations in Hex and I suppose also in Amber will be
> the same. The results in NAMD2.8b1 are similar and the 4th GPU gives
> only 5% contribution.
> Thus if I am able to use only 3GPUs in Amber this will give me
> significant advantage with the current drivers.
I actually think the GTX590 is doing a remarkable job here. What is amazing is the scaling you see for 2 GPUs on the same card. Considering they are sharing a single interface this is very good. I don't think you will get much better than that. Your best bet is to try running two x 2 GPU runs, making sure you put the threads on the correct GPUs. You might be able to run 2 jobs at once without too much slow down.
Unfortunately scaling between the GPUs is always going to suck and there really is no magic bullet fix for this. It is just the laws of physics getting in the way unfortunately and I suspect one will have to wait for PCI-E Gen 3 to make an appearance before things get any better.
Sorry I can't help much more than that but I would try to smile because a single GTX590 with 2 GPUs is doing a lot better than I would have suspected.
> I'd like also to ask you about your opinion what could be the reason
> for such bad/terrible scaling? I mean 1.3-4x is somehow ok, but
1.3x to 1.4x is awesome scaling in my opinion giving the woefully slow interconnect between the GPUs. They are sharing a single PCI-E 16x channel so even seeing some scaling is amazing.
> 1.1x..Only driver problem or I am missing something?
Where is the 1.1x coming from? On 4 GPUs? I would give up trying to use all 4 and just run 2 jobs by 2 GPUs each. Note unless you have a dual core board with some kick butt X series processors in there I doubt both your PCI-E slots are running at full speed anyway. Note normally one would have 4 x x16 slots with 4 GPUs in. Here you have 2 x x16 slots with 4 GPUs in. So each GPU is on an 8x slot. Then when you try to run all 4 in parallel all hell breaks loose. 3 GPUs probably 'just' works because one of those GPUs gets an x16 slot to itself.
All the best
Ross
/\
\/
|\oss Walker
---------------------------------------------------------
| Assistant Research Professor |
| San Diego Supercomputer Center |
| Adjunct Assistant Professor |
| Dept. of Chemistry and Biochemistry |
| University of California San Diego |
| NVIDIA Fellow |
|
http://www.rosswalker.co.uk |
http://www.wmd-lab.org/ |
| Tel: +1 858 822 0854 | EMail:- ross.rosswalker.co.uk |
---------------------------------------------------------
Note: Electronic Mail is not secure, has no guarantee of delivery, may not be read every day, and should not be used for urgent or sensitive issues.
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Sat Apr 09 2011 - 22:00:02 PDT
