So that's like, what? < 23000 atoms.
That's likely what is limiting your scaling.
Things that affect scaling:
1) Atom count - more atoms will scale better
2) FFT dimensions - larger NFFT values will scale better
3) cut off - larger cutoffs will run slower but scale better
4) ntt=3 - the langevin thermostat hurts scaling - berendsen and anderson
scale better.
5) NPT - constant pressure runs don't scale as well due to the all to all
in virial.
You are probably just at the limit for such a small system. Note scaling
for DHFR NPT tops out on Gordon at 64 cores of E5-2670 GPUs [at
44.2ns/day] (8 core sandybridge with decent memory bandwidth) - note this
is dual channel QDR IB.
What is the IB you have on your system? - is it dual channel QDR or single
channel FDR? (or something worse?)
Specifically which xeon chips are these? Are they the models prior to
Sandybridge? - The memory bandwidth pre-sandybridge was notoriously bad
and so one typically saw very poor performance if all the cores in a node
were used. Try using just 8 cores per node and see if that helps.
All the best
Ross
On 9/25/13 9:20 AM, "George Tzotzos" <gtzotzos.me.com> wrote:
>Hi Ross,
>
>Using a 140 residue protein in thermostat (~ 4,000 HOHs).
>
>Cheers
>
>George
>
>On Sep 25, 2013, at 12:53 PM, Ross Walker <ross.rosswalker.co.uk> wrote:
>
>> How many atoms is the system you are trying to simulate?
>>
>> And what settings are you using? - Thermostat, barostat etc.
>>
>> All the best
>> Ross
>>
>>
>> On 9/25/13 8:26 AM, "George Tzotzos" <gtzotzos.me.com> wrote:
>>
>>> Jason,
>>>
>>> Many thanks. We have Infiniband running on the cluster. Is there
>>>another
>>> diagnostic to achieve better scaling?
>>>
>>> Any suggestion will be much appreciated.
>>>
>>> Regards
>>>
>>> George
>>>
>>>
>>> On Sep 25, 2013, at 8:52 AM, Jason Swails <jason.swails.gmail.com>
>>>wrote:
>>>
>>>> On Tue, Sep 24, 2013 at 3:32 PM, George Tzotzos <gtzotzos.me.com>
>>>>wrote:
>>>>
>>>>> Hi everybody,
>>>>>
>>>>> I'm trying to run Amber on a cluster of the following specs
>>>>>
>>>>> SGI Specs ¡© SGI ICE X
>>>>> OS - SUSE Linux Enterprise Server 11 SP2
>>>>> Kernel Version: 3.0.38-0.5
>>>>> 2x6-Core Intel Xeon
>>>>>
>>>>> 16 blades 12 cores each
>>>>>
>>>>> Environment
>>>>> export AMBERHOME=/bio/georgios/MD/amber12
>>>>> export
>>>>>
>>>>>
>>>>>LD_LIBRARY_PATH=/opt/rpm_share/lib/lib64:/bio/george/amber12/lib:/bio/
>>>>>ge
>>>>> orge/MD/amber12/AmberTools/lib
>>>>>
>>>>> Command line
>>>>> mpirun -np 48 pmemd.MPI -O -i prod.in -o prod_12ns.out -p
>>>>> 2erb_bis_solv.prmtop -c prod_10ns.rst -r prod_12ns.rst -x
>>>>> prod_12ns.mdcrd
>>>>>
>>>>> Question
>>>>>
>>>>> No advantage in increasing the number of nodes beyond -np 24. The
>>>>> performance is reduced the more cores engaged. In fact it is similar
>>>>>or
>>>>> worse to that on a OSX 2 x 3.06 GHz 6-Core Intel Xeon
>>>>>
>>>>> I'd be very grateful for any suggestions on what may be wrong
>>>>>
>>>>
>>>> Your interconnect between nodes may be too slow. Also, the more cores
>>>> you
>>>> have on a single node, the more bandwidth you need between nodes to
>>>> avoid a
>>>> slow-down (this is why some supercomputers give faster timings for
>>>>Amber
>>>> when you do not utilize the whole node). This does not paint the
>>>>whole
>>>> picture (the topology of the inter-node connections also matters to
>>>>some
>>>> extent), but it's probably the most important part.
>>>>
>>>> You really need Infiniband (QDR is typical, I think) to see good
>>>>scaling
>>>> across nodes with Amber.
>>>>
>>>> HTH,
>>>> Jason
>>>>
>>>> --
>>>> Jason M. Swails
>>>> BioMaPS,
>>>> Rutgers University
>>>> Postdoctoral Researcher
>>>> _______________________________________________
>>>> AMBER mailing list
>>>> AMBER.ambermd.org
>>>> http://lists.ambermd.org/mailman/listinfo/amber
>>>
>>>
>>> _______________________________________________
>>> AMBER mailing list
>>> AMBER.ambermd.org
>>> http://lists.ambermd.org/mailman/listinfo/amber
>>
>>
>>
>> _______________________________________________
>> AMBER mailing list
>> AMBER.ambermd.org
>> http://lists.ambermd.org/mailman/listinfo/amber
>
>
>_______________________________________________
>AMBER mailing list
>AMBER.ambermd.org
>http://lists.ambermd.org/mailman/listinfo/amber
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Wed Sep 25 2013 - 10:00:03 PDT
