Thanks, I'll do that right away. But let me just ask a potentially stupid
question first: is it a bad idea to have the executables and input files on
a shared nfs volume as well, or is that fine?
Also, what I forgot to mention in my previous email, I've noticed that with
4 and more nodes, when performance starts dropping extremely, my cluster
management software reports about 70% of the overall CPU usage as "System
CPU", and only the remaining 30% or so as "User CPU". With 2 and less nodes,
however, when performance still scales nicely, virtually all of the CPU
usage is classified as "User CPU". On the other hand, the network load
always stays way below the theoretical capacity of 1Gbps (~0.4Gbps
regardless if cpu count). Does this still support the notion of an NFS or
general gigabit issue?
Nikola
-----Original Message-----
From: owner-amber.scripps.edu [mailto:owner-amber.scripps.edu] On Behalf Of
Robert Duke
Sent: Wednesday, June 06, 2007 2:03 PM
To: amber.scripps.edu
Subject: Re: AMBER: problem running parallel jobs
Nikola -
Yes, a shared nfs volume is a very bad place to write files. Very, very
bad. Write to a local volume if at all possible. An nfs-shared volume can
stall the master process for SECONDS on an attempted write, totally hosing
performance across all your processes. Regarding pmemd performance, you
should not expect too much from gigabit ethernet, but please run the
factor_ix and jac benchmarks that ship with the source code to get results
that can be compared to results on other systems. The only number I would
suggest changing is nstlim, and as the cpu count goes above say 8, I would
start bumping that value up to between 2000 and 5000. With pmemd, all sorts
of loadbalancing goes on and you won't get optimal times for runs of less
than 1000 steps - the code is still doing a ton of loadbalance adjusting,
especially at higher cpu count.
Regards - Bob Duke
----- Original Message -----
From: "Nikola Trbovic" <nt2146.columbia.edu>
To: <amber.scripps.edu>
Sent: Wednesday, June 06, 2007 1:31 PM
Subject: RE: AMBER: problem running parallel jobs
> Thanks a lot Robert! That solved it! No more network errors!
>
> Now I've performed a few pmemd benchmarks (explicit water, ~16000 atoms)
> over night and obtained terrible performance. First of all let me repeat
> that I'm running on a gigabit cluster with four cores per node. Here are
> the
> benchmark results:
>
> Cores Nodes Time
> 4 1 16113
> 8 2 10251
> 16 4 24143
> 32 8 48138
>
> The odd thing is that while a 2-node job still achieves a 1.6-fold speedup
> over a single node job, a 4-node job achieves no speedup at all but
> instead
> takes more than twice as long as a 2-node job, and an 8-node job four
> times
> as long. So above 2 nodes performance scales linearly with the number of
> nodes! I've read the recent note on pushing the limits with gigabit and
> multiple cores, but I haven't seen any benchmarks reporting such an
> extreme
> drop in performance. I will run new benchmarks after increasing the
> network
> buffers and checking my switch settings, but I still wanted to make sure
> that this type of performance scaling is not perhaps indicative of
> remaining
> problems with my network drivers, mpich2 installation or amber
> installation.
> I am using NFS on the cluster, and the trajectories were being saved
> through
> NFS on the head node. From the latest note on gigabit parallel computing
> it
> sounds like that is a really bad idea. Could it explain the observed
> scaling?
>
> Thanks again to Robert, and in advance for any thoughts about the scaling
> issue,
>
> Nikola
>
> -----Original Message-----
> From: owner-amber.scripps.edu [mailto:owner-amber.scripps.edu] On Behalf
> Of
> Robert Konecny
> Sent: Tuesday, June 05, 2007 5:35 PM
> To: amber.scripps.edu
> Subject: Re: AMBER: problem running parallel jobs
>
> Hi Nikola,
>
> try to disable the tcp segmentation offload on your eth0:
>
> /usr/sbin/ethtool -K eth0 tso off
>
> some versions of the tg3 driver choke on heavier traffic.
>
> robert
>
>
>
> On Tue, Jun 05, 2007 at 04:43:02PM -0400, Nikola Trbovic wrote:
>> Dear all,
>>
>> I'm having problems running pmemd and sander with mpi on more than 2
>> nodes over gigabit ethernet. Shortly after starting the job, one of the
>> nodes (which one is random) reports a network error associated with the
>> tg3 driver:
>>
>> tg3: eth0: transmit timed out, resetting
>> tg3: tg3_stop_block timed out, ofs=2c00 enable_bit=2
>> ...
>>
>> This node then disappears from the network for a couple of minutes and
>> the job stalls, although it doesn't terminate.
>>
>> Running 4 processes on one node, or even 8 on two nodes works fine,
>> however. I've tried using mpich2 and mpich, with fftw and without - it
>> made no difference. I'm compiling pmemd with ifort on RHEL 4. I know
>> this all indicates that it is not a problem with amber, but instead with
>> my OS/tg3 driver. But I was wondering if anybody had experienced the
>> same previously and could give advice on how to fix it.
>>
>> Thanks a lot in advance,
>> Nikola Trbovic
>>
>> -----------------------------------------------------------------------
>> The AMBER Mail Reflector
>> To post, send mail to amber.scripps.edu
>> To unsubscribe, send "unsubscribe amber" to majordomo.scripps.edu
> -----------------------------------------------------------------------
> The AMBER Mail Reflector
> To post, send mail to amber.scripps.edu
> To unsubscribe, send "unsubscribe amber" to majordomo.scripps.edu
>
> -----------------------------------------------------------------------
> The AMBER Mail Reflector
> To post, send mail to amber.scripps.edu
> To unsubscribe, send "unsubscribe amber" to majordomo.scripps.edu
>
-----------------------------------------------------------------------
The AMBER Mail Reflector
To post, send mail to amber.scripps.edu
To unsubscribe, send "unsubscribe amber" to majordomo.scripps.edu
-----------------------------------------------------------------------
The AMBER Mail Reflector
To post, send mail to amber.scripps.edu
To unsubscribe, send "unsubscribe amber" to majordomo.scripps.edu
Received on Sun Jun 10 2007 - 06:07:14 PDT
