it's InfiniBand, at 40 Gbps.
On Mon, Aug 29, 2011 at 5:59 PM, Jason Swails <jason.swails.gmail.com>wrote:
> What kind of interconnect does your cluster have?
>
> On Mon, Aug 29, 2011 at 4:54 PM, Bruno Rodrigues <bbrodrigues.gmail.com
> >wrote:
>
> > After the changes you suggested, I got this information on the output
> >
> > | Dynamic Memory, Types Used:
> > | Reals 688690
> > | Integers 595564
> >
> > | Nonbonded Pairs Initial Allocation: 146264
> >
> > | Running AMBER/MPI version on 64 nodes
> >
> > and still a performance of 02.ns/day.
> >
> > There is now a log file that didn't appear before, with the following
> > information:
> >
> >
> > Initial FFT Block Distribution Based on Workload Estimate:
> >
> > FFT blocks assigned to 12 tasks
> >
> > First FFT Block Distribution Based on Actual Workload:
> >
> > FFT blocks assigned to 56 tasks
> >
> > Image Distribution at run step 344:
> >
> > Count of images assigned to each task:
> > 340 437 412 335 542 572 542
> 516
> > 291 256 99 1 0 0 0
> 0
> > 0 0 0 0 0 0 230
> 184
> > 1 0 244 352 436 6 82
> 219
> > 23 1 2 64 137 283 173
> 59
> > 290 133 233 81 253 198 341
> 173
> > 280 330 367 267 157 117 407
> 125
> > 361 374 533 455 606 646 1003
> 905
> >
> >
> > What does it mean?
> >
> >
> >
> >
> > On Mon, Aug 29, 2011 at 5:34 PM, Jason Swails <jason.swails.gmail.com
> > >wrote:
> >
> > > Every PBS system is set up differently, so it's impossible for us to
> tell
> > > what may be happening for sure. However, I suspect that you're not
> > getting
> > > 64 CPUs like you think you are.
> > >
> > > On Mon, Aug 29, 2011 at 4:05 PM, Bruno Rodrigues <
> bbrodrigues.gmail.com
> > > >wrote:
> > >
> > > > Dear All,
> > > >
> > > > I'm trying to run parallel Amber 11 on a cluster with PBS. I've
> checked
> > > the
> > > > parallel installation and it's quite fine (the log file attached).
> > > >
> > > > However, the performance is always between 0.1 and 0.5 ns/day, no
> > matter
> > > > the
> > > > number of processors I choose. Is there something missing in my
> script?
> > > >
> > > > Here are the changes I made on my configure (for the parallel
> version):
> > > > mpicc --> icc -lmpi
> > > > mpif90 --> ifort -lmpi
> > > >
> > > > This generated the correct config.h needed for the fortran compiler.
> > > >
> > > > However, the problem persists with gnu installing, so I guess it has
> > > > nothing
> > > > to do with the installation, but it's pretty much a submission
> problem.
> > > > Here
> > > > is an example of my job:
> > > >
> > > > #!/bin/bash
> > > > #
> > > > #################################################
> > > > # THIS JOB IS TO EQUILIBRATE THE SYSTEM AT 300K #
> > > > # TO BE USED IN FUTURE SIMULATIONS. IT STARTS #
> > > > # FROM THE EQUILIBRATION ON CHACOBO, WHERE 1ns #
> > > > # WAS PERFORMED AFTER THE DNA WAS RELEASED. #
> > > > #################################################
> > > > #
> > > > #PBS -S /bin/sh
> > > > #
> > > > # Nome do job
> > > > #PBS -N prod_slow
> > > > #
> > > > #Erro na saida padrao
> > > > #PBS -j oe
> > > > #
> > > > # Chamada do ambiente paralelo e numero de slots
> > > > #PBS -l select=64:ncpus=1
> > > > #PBS -l walltime=200:00:00
> > > >
> > > > #
> > > > cd $PBS_O_WORKDIR
> > > >
> > > > export sander=/home/u/bbr/bin/amber11/bin/pmemd.MPI
> > > >
> > >
> > > In here, add the line
> > >
> > > CPUS=`cat $PBS_NODEFILE | wc -l`
> > >
> > >
> > > >
> > > > l=heat20
> > > > f=prod01
> > > > mpiexec -n 64 $sander -O -i $PWD/$f.in -o $PWD/$f.out -inf
> $PWD/$f.inf
> > \
> > > > -c $PWD/1D20_wat_tip3pf.$l -ref $PWD/1D20_wat_tip3pf.$l -r
> > > > $PWD/1D20_wat_tip3pf.$f \
> > > > -p $PWD/1D20_wat_tip3pf.top -x $PWD/1D20_wat_tip3pf$f.x -e
> > > > $PWD/1D20_wat_tip3pf$f.ene
> > > >
> > >
> > > change the beginning to "mpiexec -n $CPUS" instead of "mpiexec -n 64".
> > > pmemd.MPI should report how many processors are being used, which
> should
> > > help you make sure that you're at least allocating all the processors
> you
> > > want to be. You could also consider passing mpiexec the PBS_NODEFILE
> if
> > > you
> > > find out how to pass your mpiexec a hostfile or nodefile or something
> > (this
> > > makes sure that each thread is bound to the proper processor).
> > >
> > > HTH,
> > > Jason
> > >
> > > --
> > > Jason M. Swails
> > > Quantum Theory Project,
> > > University of Florida
> > > Ph.D. Candidate
> > > 352-392-4032
> > > _______________________________________________
> > > AMBER mailing list
> > > AMBER.ambermd.org
> > > http://lists.ambermd.org/mailman/listinfo/amber
> > >
> >
> >
> >
> > --
> > --
> > Bruno Barbosa Rodrigues
> > PhD Student - Physics Department
> > Universidade Federal de Minas Gerais - UFMG
> > Belo Horizonte - Brazil
> > _______________________________________________
> > AMBER mailing list
> > AMBER.ambermd.org
> > http://lists.ambermd.org/mailman/listinfo/amber
> >
>
>
>
> --
> Jason M. Swails
> Quantum Theory Project,
> University of Florida
> Ph.D. Candidate
> 352-392-4032
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
>
--
--
Bruno Barbosa Rodrigues
PhD Student - Physics Department
Universidade Federal de Minas Gerais - UFMG
Belo Horizonte - Brazil
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Mon Aug 29 2011 - 14:30:03 PDT
