Hi Nils,
We've seen issues with MPI in the latest version of Amber 12 - especially
with jobs hanging after a random amount of time. This tends to affect
multi-node jobs more than single node jobs but it is still possible single
node jobs would give issues. We are trying to figure out what the problem
is but time to work on this without distraction is pretty rare at the
moment so it might be a couple of months before it gets looked at properly.
For now I would just stick with using 1 GPU per run and running 3 runs
from different random seeds in the same box.
All the best
Ross
On 9/23/13 3:12 AM, "Nils Oberhauser" <Nils.Oberhauser.unige.ch> wrote:
>Hello,
>
>I am having some trouble running parallel CUDA jobs on 2 and 3 GPUs.
>I'm observing the following problems following the lipid 11 tutorial on
>http://ambermd.org/tutorials/advanced/tutorial16/An_Amber_Lipid_Force_Fiel
>d_
>Tutorial.html :
>To be sure I did not do anything wrong in the setup, I downloaded the
>example files from the tutorial page as input.
>During the production phase the calculations break up after a short time,
>returning:
> "Nonbond cells need to be recalculated, restart simulation from
>previous checkpoint with a higher value for skinnb."
>I restart the calculation, as was suggested by previous posts in the
>mailing
>list. For a while the calculations seem to be running smoothly, but after
>a
>certain time period (sometimes minutes, sometimes hours) there is no more
>write access to the trajectory or any other output file. The
>pmemd.cuda.MPI
>processes keep running forever without any output or error message until
>the
>process is killed manually. Since the jobs just keep running without any
>output, I'm not sure how to further debug this problem...
>When I use only one GPU or 10 parallel CPU cores however, the job finishes
>normally.
>
>The machine I'm using has 3 GeForce GTX 680 Cards and two Intel Xeon
>E5-2620
>(2 GHz, 6 cores each) CPUs, running CentOS 64bit (kernel:
>2.6.32-358.14.1.el6.x86_64) and the latest updates of amber12.
>
>Here is the most confusing part:
>We have another machine with identical hardware but a slightly different
>linux kernel (2.6.32-358.11.1.el6.x86_64) and a not up-to-date version of
>amber12. Since there are currently jobs running on the machine, I cannot
>upgrade the kernel or amber to see if it makes any difference.
>However when I run the same job from the tutorial as described above on
>this
>machine, I don't get any errors. The jobs run and terminate without any
>problems on 3 GPUs. Even the "Nonbond cells.." error never occurs (maybe
>because of the older amber version?).
>I ran the jobs several times on both machines, to be quite sure that the
>different outcomes are not due to the non-deterministic algorithms of the
>software, but it is always the same results.
>
>So, before I go through all the trouble of installing an older kernel to
>the
>first machine, I'm asking here for help. Can it at all be the kernel?
>Maybe some has a different suggestion to debug this problem.
>
>Thanks in advance and best regards,
>Nils
>
>
>
>
>_______________________________________________
>AMBER mailing list
>AMBER.ambermd.org
>http://lists.ambermd.org/mailman/listinfo/amber
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Mon Sep 23 2013 - 11:00:03 PDT
