What kind of interconnect does your cluster have?
On Mon, Aug 29, 2011 at 4:54 PM, Bruno Rodrigues <bbrodrigues.gmail.com>wrote:
> After the changes you suggested, I got this information on the output
>
> | Dynamic Memory, Types Used:
> | Reals 688690
> | Integers 595564
>
> | Nonbonded Pairs Initial Allocation: 146264
>
> | Running AMBER/MPI version on 64 nodes
>
> and still a performance of 02.ns/day.
>
> There is now a log file that didn't appear before, with the following
> information:
>
>
> Initial FFT Block Distribution Based on Workload Estimate:
>
> FFT blocks assigned to 12 tasks
>
> First FFT Block Distribution Based on Actual Workload:
>
> FFT blocks assigned to 56 tasks
>
> Image Distribution at run step 344:
>
> Count of images assigned to each task:
> 340 437 412 335 542 572 542 516
> 291 256 99 1 0 0 0 0
> 0 0 0 0 0 0 230 184
> 1 0 244 352 436 6 82 219
> 23 1 2 64 137 283 173 59
> 290 133 233 81 253 198 341 173
> 280 330 367 267 157 117 407 125
> 361 374 533 455 606 646 1003 905
>
>
> What does it mean?
>
>
>
>
> On Mon, Aug 29, 2011 at 5:34 PM, Jason Swails <jason.swails.gmail.com
> >wrote:
>
> > Every PBS system is set up differently, so it's impossible for us to tell
> > what may be happening for sure. However, I suspect that you're not
> getting
> > 64 CPUs like you think you are.
> >
> > On Mon, Aug 29, 2011 at 4:05 PM, Bruno Rodrigues <bbrodrigues.gmail.com
> > >wrote:
> >
> > > Dear All,
> > >
> > > I'm trying to run parallel Amber 11 on a cluster with PBS. I've checked
> > the
> > > parallel installation and it's quite fine (the log file attached).
> > >
> > > However, the performance is always between 0.1 and 0.5 ns/day, no
> matter
> > > the
> > > number of processors I choose. Is there something missing in my script?
> > >
> > > Here are the changes I made on my configure (for the parallel version):
> > > mpicc --> icc -lmpi
> > > mpif90 --> ifort -lmpi
> > >
> > > This generated the correct config.h needed for the fortran compiler.
> > >
> > > However, the problem persists with gnu installing, so I guess it has
> > > nothing
> > > to do with the installation, but it's pretty much a submission problem.
> > > Here
> > > is an example of my job:
> > >
> > > #!/bin/bash
> > > #
> > > #################################################
> > > # THIS JOB IS TO EQUILIBRATE THE SYSTEM AT 300K #
> > > # TO BE USED IN FUTURE SIMULATIONS. IT STARTS #
> > > # FROM THE EQUILIBRATION ON CHACOBO, WHERE 1ns #
> > > # WAS PERFORMED AFTER THE DNA WAS RELEASED. #
> > > #################################################
> > > #
> > > #PBS -S /bin/sh
> > > #
> > > # Nome do job
> > > #PBS -N prod_slow
> > > #
> > > #Erro na saida padrao
> > > #PBS -j oe
> > > #
> > > # Chamada do ambiente paralelo e numero de slots
> > > #PBS -l select=64:ncpus=1
> > > #PBS -l walltime=200:00:00
> > >
> > > #
> > > cd $PBS_O_WORKDIR
> > >
> > > export sander=/home/u/bbr/bin/amber11/bin/pmemd.MPI
> > >
> >
> > In here, add the line
> >
> > CPUS=`cat $PBS_NODEFILE | wc -l`
> >
> >
> > >
> > > l=heat20
> > > f=prod01
> > > mpiexec -n 64 $sander -O -i $PWD/$f.in -o $PWD/$f.out -inf $PWD/$f.inf
> \
> > > -c $PWD/1D20_wat_tip3pf.$l -ref $PWD/1D20_wat_tip3pf.$l -r
> > > $PWD/1D20_wat_tip3pf.$f \
> > > -p $PWD/1D20_wat_tip3pf.top -x $PWD/1D20_wat_tip3pf$f.x -e
> > > $PWD/1D20_wat_tip3pf$f.ene
> > >
> >
> > change the beginning to "mpiexec -n $CPUS" instead of "mpiexec -n 64".
> > pmemd.MPI should report how many processors are being used, which should
> > help you make sure that you're at least allocating all the processors you
> > want to be. You could also consider passing mpiexec the PBS_NODEFILE if
> > you
> > find out how to pass your mpiexec a hostfile or nodefile or something
> (this
> > makes sure that each thread is bound to the proper processor).
> >
> > HTH,
> > Jason
> >
> > --
> > Jason M. Swails
> > Quantum Theory Project,
> > University of Florida
> > Ph.D. Candidate
> > 352-392-4032
> > _______________________________________________
> > AMBER mailing list
> > AMBER.ambermd.org
> > http://lists.ambermd.org/mailman/listinfo/amber
> >
>
>
>
> --
> --
> Bruno Barbosa Rodrigues
> PhD Student - Physics Department
> Universidade Federal de Minas Gerais - UFMG
> Belo Horizonte - Brazil
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
>
--
Jason M. Swails
Quantum Theory Project,
University of Florida
Ph.D. Candidate
352-392-4032
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Mon Aug 29 2011 - 14:30:02 PDT
