In addition to what Dave said, I would suggest against using mpinab. If
you desire parallelism that much, go with OpenMP instead. The problem with
MPI in this case is that memory is not shared. Each thread has its own
data, in addition to a buffer that is commonly used for MPI operations,
like reductions and gathers. Therefore, for MPI I think you can expect
each thread to use 9.5 GB of RAM, including perhaps a second copy of the
Hessian for a MPI_Reduce. All told, if I assume you have 16 cores on your
node with 48 GB, I would expect your job to need ~150 GB of RAM.
If someone tried parallelizing the construction of the hessian and added a
reduce buffer to collect the contributions from each node, go ahead and
double this to get ~300 GB.
Of course the flaw in the above analysis is that I think NAB takes
advantage of the fact that the Hessian is symmetric, so it only needs to
store the upper triangular portion of the matrix in addition to the
diagonal elements, cutting the memory requirements (almost) in half. Of
course, the BLAS routines responsible for diagonalizing the Hessian needs
some scratch space allocated for the calculation, which would seem
reasonable to me that the memory requirements jump back to the 200 GB range.
OpenMP, on the other hand, uses shared memory parallelism, often
eliminating the need for separate copies of each array. The best option
here, IMO, is to follow Dave's advice and separate the minimization/normal
mode steps, in which you can reasonably expect to use mpinab for the
minimization step before executing the normal mode calculation in serial.
HTH,
Jason
On Mon, Dec 10, 2012 at 12:43 PM, Ryan Pavlovicz <pavlovicz.7.osu.edu>wrote:
> I am trying to do some nmode calculations with nab (mpinab) on a system of
> 11,795 atoms.
>
> >From what i understand, the nmode calculations would calculate a Hessian
> of
> 3Nx3N elements, and each element would take 8 bytes.
>
> Therefore, my system of 11,795 atoms, 35,385 coordinates, and 1.25 million
> matrix elements would need ~10 GB of memory. However, my testing on
> supercomputer nodes with 48 GB of physical memory have been confusing. Some
> of these jobs fail once they pass to the Newton Raphson part of the
> calculation and memory requirements exceed 48 GB. However some of my jobs
> are successful in completing a couple iterations of NR minimization before
> walltime runs out. When monitoring these successful jobs, i see that they
> are using ~211 GB of memory to complete these calculations.
>
> Is my estimation of required memory off by ~200 GB or is there a potential
> problem with the mpinab executable i am using?
>
> Thanks for your help,
>
> ryan
>
> input file, nmode_complex.nab:
>
> molecule m;
> float x[35385], fret;
>
> m = getpdb("../../../com1.pdb");
> readparm(m, "../../../complex.prmtop");
> getxyz("../../../snaps/BigD_2_1_com.crd.1",11795,x);
>
> mm_options("cut=999.0, ntpr=1, nsnb=99999, diel=C, gb=1, dielc=1.0");
> mme_init(m, NULL, "::Z", x, NULL);
> setxyz_from_mol(m, NULL, x);
>
> //conjugate gradient minimization
> conjgrad(x,3*m.natoms,fret,mme,0.001,0.001,20000);
>
> //newton-raphson minimization
> newton(x,3*m.natoms,fret,mme,mme2,0.0000001,0.0,1000);
>
> //get normal modes
> nmode(x,3*m.natoms,mme2,0,0,0.0,0.0,0);
>
> batch execution:
>
> $AMBERHOME/bin/mpinab -o nmode_complex nmode_complex.nab
> mpiexec ./nmode_complex > nmode_com1.out
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
>
--
Jason M. Swails
Quantum Theory Project,
University of Florida
Ph.D. Candidate
352-392-4032
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Mon Dec 10 2012 - 12:00:03 PST
