Hello,
On Sat, Oct 1, 2011 at 2:23 PM, Chicago Ji <chicago.ecnu.gmail.com> wrote:
> Dear Amber users,
>
> I want to add an extra energy term on part of the system (only few
> atoms) in Amber11.
>
> I need to add extra force to f(*) in runmd.f .
>
> But I'm don't know what was stored in f(*) exactly .
>
f(*) stores all of the forces. All forces are calculated, and this array is
populated, in force.f (subroutine force). This subroutine begins by zeroing
out the entire force array. Each processor then proceeds to calculate the
force (according to all of the potential terms that are included) on each
atom that *that processor is responsible for*.
> I tried to print out f(*) on each process.
>
> It seems that only part of f(*) was right on each process.
> ( from id "istart3" to "iend3" )
>
Correct (except for the very first dynamics step if you're not doing a
restart).
>
> I guess : f(*) calculted from each process wasn't summarized and brocast .
>
Correct. If you want to see what's happening, see the subroutines "fdist"
and "fsum" in $AMBERHOME/src/sander/parallel.f. (The force distribution
routine is called in force.f -- just search for the fdist call). The key to
this is that the force distribution is done via a MPI_Reduce_scatter call
(see around line 550 in parallel.f), rather than a call to MPI_Allreduce().
The MPI_Reduce_scatter sums up all of the forces on all threads to the
master thread, then scatters out the relevant parts for each slave processor
so that it can propagate the coordinates of the atoms it owns. An
MPI_Allreduce() call would have summed up all forces from all threads *to*
all threads, which is a more costly communication (and this is what's done
for the very first dynamics step if you're not doing a restart).
> So only part information of f(*) on each process was useful, am i right ?
>
Correct. This is OK, though, since (as you can see in runmd), each
processor propagates only the atomic positions of the atoms it owns. (From
runmd.f:
do i3 = istart3, iend3
f(i3) = x(i3)
if (i3 >= cphfirst_sol3 - 3) &
x(i3) = x(i3) + v(i3)*dtx
end do
where istart3 and iend3 are set based on atom ownership). Then of course
the positions have to be sent from everyone to everyone (xdist does this via
an MPI_Allgatherv call).
Note, though, that pmemd's parallelization scheme is _completely_ different
than sander's. For instance, it works via a spatial decomposition scheme
rather than sequence decomposition (and pmemd's scaling and performance is
obviously much better than sander's). Thus, any lessons learned about
sander parallelization must be relearned if you want to start modifying or
understanding pmemd.
HTH,
Jason
--
Jason M. Swails
Quantum Theory Project,
University of Florida
Ph.D. Candidate
352-392-4032
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Sat Oct 01 2011 - 12:30:02 PDT
