the serial sander runs at 0.6ns/day....
may it be a problem with mpi?
On Mon, Aug 29, 2011 at 7:13 PM, Bruno Rodrigues <bbrodrigues.gmail.com>wrote:
> I've found that on the former cluster, the logfile prints out the FFT slab
> distribution, and now it's FFT block distribution.
> Does it mean that something has substantially changed on the way FFT
> distributes the the blocks?
>
>
> On Mon, Aug 29, 2011 at 6:05 PM, Bruno Rodrigues <bbrodrigues.gmail.com>wrote:
>
>> it's InfiniBand, at 40 Gbps.
>>
>>
>> On Mon, Aug 29, 2011 at 5:59 PM, Jason Swails <jason.swails.gmail.com>wrote:
>>
>>> What kind of interconnect does your cluster have?
>>>
>>> On Mon, Aug 29, 2011 at 4:54 PM, Bruno Rodrigues <bbrodrigues.gmail.com
>>> >wrote:
>>>
>>> > After the changes you suggested, I got this information on the output
>>> >
>>> > | Dynamic Memory, Types Used:
>>> > | Reals 688690
>>> > | Integers 595564
>>> >
>>> > | Nonbonded Pairs Initial Allocation: 146264
>>> >
>>> > | Running AMBER/MPI version on 64 nodes
>>> >
>>> > and still a performance of 02.ns/day.
>>> >
>>> > There is now a log file that didn't appear before, with the following
>>> > information:
>>> >
>>> >
>>> > Initial FFT Block Distribution Based on Workload Estimate:
>>> >
>>> > FFT blocks assigned to 12 tasks
>>> >
>>> > First FFT Block Distribution Based on Actual Workload:
>>> >
>>> > FFT blocks assigned to 56 tasks
>>> >
>>> > Image Distribution at run step 344:
>>> >
>>> > Count of images assigned to each task:
>>> > 340 437 412 335 542 572 542
>>> 516
>>> > 291 256 99 1 0 0 0
>>> 0
>>> > 0 0 0 0 0 0 230
>>> 184
>>> > 1 0 244 352 436 6 82
>>> 219
>>> > 23 1 2 64 137 283 173
>>> 59
>>> > 290 133 233 81 253 198 341
>>> 173
>>> > 280 330 367 267 157 117 407
>>> 125
>>> > 361 374 533 455 606 646 1003
>>> 905
>>> >
>>> >
>>> > What does it mean?
>>> >
>>> >
>>> >
>>> >
>>> > On Mon, Aug 29, 2011 at 5:34 PM, Jason Swails <jason.swails.gmail.com
>>> > >wrote:
>>> >
>>> > > Every PBS system is set up differently, so it's impossible for us to
>>> tell
>>> > > what may be happening for sure. However, I suspect that you're not
>>> > getting
>>> > > 64 CPUs like you think you are.
>>> > >
>>> > > On Mon, Aug 29, 2011 at 4:05 PM, Bruno Rodrigues <
>>> bbrodrigues.gmail.com
>>> > > >wrote:
>>> > >
>>> > > > Dear All,
>>> > > >
>>> > > > I'm trying to run parallel Amber 11 on a cluster with PBS. I've
>>> checked
>>> > > the
>>> > > > parallel installation and it's quite fine (the log file attached).
>>> > > >
>>> > > > However, the performance is always between 0.1 and 0.5 ns/day, no
>>> > matter
>>> > > > the
>>> > > > number of processors I choose. Is there something missing in my
>>> script?
>>> > > >
>>> > > > Here are the changes I made on my configure (for the parallel
>>> version):
>>> > > > mpicc --> icc -lmpi
>>> > > > mpif90 --> ifort -lmpi
>>> > > >
>>> > > > This generated the correct config.h needed for the fortran
>>> compiler.
>>> > > >
>>> > > > However, the problem persists with gnu installing, so I guess it
>>> has
>>> > > > nothing
>>> > > > to do with the installation, but it's pretty much a submission
>>> problem.
>>> > > > Here
>>> > > > is an example of my job:
>>> > > >
>>> > > > #!/bin/bash
>>> > > > #
>>> > > > #################################################
>>> > > > # THIS JOB IS TO EQUILIBRATE THE SYSTEM AT 300K #
>>> > > > # TO BE USED IN FUTURE SIMULATIONS. IT STARTS #
>>> > > > # FROM THE EQUILIBRATION ON CHACOBO, WHERE 1ns #
>>> > > > # WAS PERFORMED AFTER THE DNA WAS RELEASED. #
>>> > > > #################################################
>>> > > > #
>>> > > > #PBS -S /bin/sh
>>> > > > #
>>> > > > # Nome do job
>>> > > > #PBS -N prod_slow
>>> > > > #
>>> > > > #Erro na saida padrao
>>> > > > #PBS -j oe
>>> > > > #
>>> > > > # Chamada do ambiente paralelo e numero de slots
>>> > > > #PBS -l select=64:ncpus=1
>>> > > > #PBS -l walltime=200:00:00
>>> > > >
>>> > > > #
>>> > > > cd $PBS_O_WORKDIR
>>> > > >
>>> > > > export sander=/home/u/bbr/bin/amber11/bin/pmemd.MPI
>>> > > >
>>> > >
>>> > > In here, add the line
>>> > >
>>> > > CPUS=`cat $PBS_NODEFILE | wc -l`
>>> > >
>>> > >
>>> > > >
>>> > > > l=heat20
>>> > > > f=prod01
>>> > > > mpiexec -n 64 $sander -O -i $PWD/$f.in -o $PWD/$f.out -inf
>>> $PWD/$f.inf
>>> > \
>>> > > > -c $PWD/1D20_wat_tip3pf.$l -ref $PWD/1D20_wat_tip3pf.$l -r
>>> > > > $PWD/1D20_wat_tip3pf.$f \
>>> > > > -p $PWD/1D20_wat_tip3pf.top -x $PWD/1D20_wat_tip3pf$f.x -e
>>> > > > $PWD/1D20_wat_tip3pf$f.ene
>>> > > >
>>> > >
>>> > > change the beginning to "mpiexec -n $CPUS" instead of "mpiexec -n
>>> 64".
>>> > > pmemd.MPI should report how many processors are being used, which
>>> should
>>> > > help you make sure that you're at least allocating all the processors
>>> you
>>> > > want to be. You could also consider passing mpiexec the PBS_NODEFILE
>>> if
>>> > > you
>>> > > find out how to pass your mpiexec a hostfile or nodefile or something
>>> > (this
>>> > > makes sure that each thread is bound to the proper processor).
>>> > >
>>> > > HTH,
>>> > > Jason
>>> > >
>>> > > --
>>> > > Jason M. Swails
>>> > > Quantum Theory Project,
>>> > > University of Florida
>>> > > Ph.D. Candidate
>>> > > 352-392-4032
>>> > > _______________________________________________
>>> > > AMBER mailing list
>>> > > AMBER.ambermd.org
>>> > > http://lists.ambermd.org/mailman/listinfo/amber
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > --
>>> > Bruno Barbosa Rodrigues
>>> > PhD Student - Physics Department
>>> > Universidade Federal de Minas Gerais - UFMG
>>> > Belo Horizonte - Brazil
>>> > _______________________________________________
>>> > AMBER mailing list
>>> > AMBER.ambermd.org
>>> > http://lists.ambermd.org/mailman/listinfo/amber
>>> >
>>>
>>>
>>>
>>> --
>>> Jason M. Swails
>>> Quantum Theory Project,
>>> University of Florida
>>> Ph.D. Candidate
>>> 352-392-4032
>>> _______________________________________________
>>> AMBER mailing list
>>> AMBER.ambermd.org
>>> http://lists.ambermd.org/mailman/listinfo/amber
>>>
>>
>>
>>
>> --
>> --
>> Bruno Barbosa Rodrigues
>> PhD Student - Physics Department
>> Universidade Federal de Minas Gerais - UFMG
>> Belo Horizonte - Brazil
>>
>
>
>
> --
> --
> Bruno Barbosa Rodrigues
> PhD Student - Physics Department
> Universidade Federal de Minas Gerais - UFMG
> Belo Horizonte - Brazil
>
--
--
Bruno Barbosa Rodrigues
PhD Student - Physics Department
Universidade Federal de Minas Gerais - UFMG
Belo Horizonte - Brazil
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Mon Aug 29 2011 - 21:00:03 PDT
