On Mon, 2015-07-06 at 09:18 -0400, John Dood wrote:
> Hi all,
>
> I am simulating large salt crystals and am exploring the effect that nfft
> has on the potential energy (considering the importance of electrostatics
> in salt). I initially ran into trouble when I ran a simulation with an nfft
> greater than 2048 getting the error message:
>
> PARAMETER RANGE CHECKING:
> parameter nfft1: (grid size) has value 4000
> This is outside the legal range
> Lower limit: 6 Upper limit: 2048
> The limits may be adjustable; search in the .h files
Holy cow, that's an *enormous* grid! Typical grid sizes are on the
order of 48-128 grid points per dimension -- typically around 1 grid
point per Angstrom, if memory serves.
Keep in mind this is a 3-dimensional grid, so the total number of grid
points scales exponentially with the dimensionality of your grid size.
For a 128x128x128 grid, you have 128**3=2M grid points. At full double
precision (64-bit, or 8-byte numbers), that's 2e6*8/1024.0**2=16 MB of
memory to store the entire grid. If you increase that to 2000, you
start to require 2000**3*8/1024.0**2 = 61,035 MB of memory (!!) to store
the entire grid. You also need working space and MPI send/recv buffers,
so the actual space sander tries to allocate for the grid is actually a
bit over 3 copies (so in this case, >180 GB of RAM).
> So I ran a simulation with half the nfft and got the following error
> message:
>
> nfft1-3 too large! check on MAXNFFT in ew_bspline.f
>
> and found that maxnfft has a value of 500. After increasing the value of
> maxnfft to 2048 as discussed in http://archive.ambermd.org/200811/0285.html
> and http://archive.ambermd.org/201006/0195.html I found that I could get
> nfft=800 to work but not nfft=2000 (the program ends with a segfault)
>
> I have a couple of questions. Why is there a different limit in
> ew_setup.F90 and ew_legal.h?
My guess is that the reason is more historical than anything else. Not
a *good* reason, but sander is an old program that has been developed
over >2 decades by many different scientists (not professional
programmers). It has all kinds of weird nooks and crannies like this.
> Why does a simulation of 800 work and not a
> simulation with 2000? If anyone has experience with this kind of issue I
> would love to hear your take on it.
A grid size of 800x800x800 requires ~488 MB of space to store. Contrast
to the 61 GB required for a grid of 2000 points. Add in the extra
copies needed for, e.g., parallel simulations, you're looking at several
hundreds of GB of RAM required. It's no surprise this results in a
segfault :).
Note that there are a few 'gotchas' when it comes to picking FFT
dimensions. For example, the FFT algorithm is only efficient for
relatively small prime factors, so grid sizes are usually adjusted so
that they *only* have the prime factors 2, 3, 5, and on occasion 7
(depending on the FFT implementation). So a grid size of 47 will have
MUCH worse performance than a grid size of 48 or 64, despite the fact
that the latter two are larger grids.
Also, Amber calculates and reports an error estimate associated with
using PME instead of pure Ewald (where the reciprocal energy is
evaluated on the grid instead of the atom centers). This ewald error
estimate is printed in the mdout file and should give you a general
sense of the accuracy of the method. You can also run a "pure" Ewald
simulation in sander (setting ew_type=1 in &ewald) and compare the PME
energies to the "exact" Ewald energies. I would think that this
approach would be equivalent to an "infinitely dense" PME grid, and is
probably a more tractable approach to quantifying the error due to
descritizing the charge density on a grid.
HTH,
Jason
--
Jason M. Swails
BioMaPS,
Rutgers University
Postdoctoral Researcher
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Mon Jul 06 2015 - 08:30:02 PDT
