- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]

From: Jason Swails <jason.swails.gmail.com>

Date: Mon, 06 Jul 2015 11:15:33 -0400

On Mon, 2015-07-06 at 09:18 -0400, John Dood wrote:

*> Hi all,
*

*>
*

*> I am simulating large salt crystals and am exploring the effect that nfft
*

*> has on the potential energy (considering the importance of electrostatics
*

*> in salt). I initially ran into trouble when I ran a simulation with an nfft
*

*> greater than 2048 getting the error message:
*

*>
*

*> PARAMETER RANGE CHECKING:
*

*> parameter nfft1: (grid size) has value 4000
*

*> This is outside the legal range
*

*> Lower limit: 6 Upper limit: 2048
*

*> The limits may be adjustable; search in the .h files
*

Holy cow, that's an *enormous* grid! Typical grid sizes are on the

order of 48-128 grid points per dimension -- typically around 1 grid

point per Angstrom, if memory serves.

Keep in mind this is a 3-dimensional grid, so the total number of grid

points scales exponentially with the dimensionality of your grid size.

For a 128x128x128 grid, you have 128**3=2M grid points. At full double

precision (64-bit, or 8-byte numbers), that's 2e6*8/1024.0**2=16 MB of

memory to store the entire grid. If you increase that to 2000, you

start to require 2000**3*8/1024.0**2 = 61,035 MB of memory (!!) to store

the entire grid. You also need working space and MPI send/recv buffers,

so the actual space sander tries to allocate for the grid is actually a

bit over 3 copies (so in this case, >180 GB of RAM).

*> So I ran a simulation with half the nfft and got the following error
*

*> message:
*

*>
*

*> nfft1-3 too large! check on MAXNFFT in ew_bspline.f
*

*>
*

*> and found that maxnfft has a value of 500. After increasing the value of
*

*> maxnfft to 2048 as discussed in http://archive.ambermd.org/200811/0285.html
*

*> and http://archive.ambermd.org/201006/0195.html I found that I could get
*

*> nfft=800 to work but not nfft=2000 (the program ends with a segfault)
*

*>
*

*> I have a couple of questions. Why is there a different limit in
*

*> ew_setup.F90 and ew_legal.h?
*

My guess is that the reason is more historical than anything else. Not

a *good* reason, but sander is an old program that has been developed

over >2 decades by many different scientists (not professional

programmers). It has all kinds of weird nooks and crannies like this.

*> Why does a simulation of 800 work and not a
*

*> simulation with 2000? If anyone has experience with this kind of issue I
*

*> would love to hear your take on it.
*

A grid size of 800x800x800 requires ~488 MB of space to store. Contrast

to the 61 GB required for a grid of 2000 points. Add in the extra

copies needed for, e.g., parallel simulations, you're looking at several

hundreds of GB of RAM required. It's no surprise this results in a

segfault :).

Note that there are a few 'gotchas' when it comes to picking FFT

dimensions. For example, the FFT algorithm is only efficient for

relatively small prime factors, so grid sizes are usually adjusted so

that they *only* have the prime factors 2, 3, 5, and on occasion 7

(depending on the FFT implementation). So a grid size of 47 will have

MUCH worse performance than a grid size of 48 or 64, despite the fact

that the latter two are larger grids.

Also, Amber calculates and reports an error estimate associated with

using PME instead of pure Ewald (where the reciprocal energy is

evaluated on the grid instead of the atom centers). This ewald error

estimate is printed in the mdout file and should give you a general

sense of the accuracy of the method. You can also run a "pure" Ewald

simulation in sander (setting ew_type=1 in &ewald) and compare the PME

energies to the "exact" Ewald energies. I would think that this

approach would be equivalent to an "infinitely dense" PME grid, and is

probably a more tractable approach to quantifying the error due to

descritizing the charge density on a grid.

HTH,

Jason

Date: Mon, 06 Jul 2015 11:15:33 -0400

On Mon, 2015-07-06 at 09:18 -0400, John Dood wrote:

Holy cow, that's an *enormous* grid! Typical grid sizes are on the

order of 48-128 grid points per dimension -- typically around 1 grid

point per Angstrom, if memory serves.

Keep in mind this is a 3-dimensional grid, so the total number of grid

points scales exponentially with the dimensionality of your grid size.

For a 128x128x128 grid, you have 128**3=2M grid points. At full double

precision (64-bit, or 8-byte numbers), that's 2e6*8/1024.0**2=16 MB of

memory to store the entire grid. If you increase that to 2000, you

start to require 2000**3*8/1024.0**2 = 61,035 MB of memory (!!) to store

the entire grid. You also need working space and MPI send/recv buffers,

so the actual space sander tries to allocate for the grid is actually a

bit over 3 copies (so in this case, >180 GB of RAM).

My guess is that the reason is more historical than anything else. Not

a *good* reason, but sander is an old program that has been developed

over >2 decades by many different scientists (not professional

programmers). It has all kinds of weird nooks and crannies like this.

A grid size of 800x800x800 requires ~488 MB of space to store. Contrast

to the 61 GB required for a grid of 2000 points. Add in the extra

copies needed for, e.g., parallel simulations, you're looking at several

hundreds of GB of RAM required. It's no surprise this results in a

segfault :).

Note that there are a few 'gotchas' when it comes to picking FFT

dimensions. For example, the FFT algorithm is only efficient for

relatively small prime factors, so grid sizes are usually adjusted so

that they *only* have the prime factors 2, 3, 5, and on occasion 7

(depending on the FFT implementation). So a grid size of 47 will have

MUCH worse performance than a grid size of 48 or 64, despite the fact

that the latter two are larger grids.

Also, Amber calculates and reports an error estimate associated with

using PME instead of pure Ewald (where the reciprocal energy is

evaluated on the grid instead of the atom centers). This ewald error

estimate is printed in the mdout file and should give you a general

sense of the accuracy of the method. You can also run a "pure" Ewald

simulation in sander (setting ew_type=1 in &ewald) and compare the PME

energies to the "exact" Ewald energies. I would think that this

approach would be equivalent to an "infinitely dense" PME grid, and is

probably a more tractable approach to quantifying the error due to

descritizing the charge density on a grid.

HTH,

Jason

-- Jason M. Swails BioMaPS, Rutgers University Postdoctoral Researcher _______________________________________________ AMBER mailing list AMBER.ambermd.org http://lists.ambermd.org/mailman/listinfo/amberReceived on Mon Jul 06 2015 - 08:30:02 PDT

Custom Search