Re: [AMBER] Problematic structure after minimization on GPU

From: Jan-Philip Gehrcke <jgehrcke.googlemail.com>
Date: Tue, 27 Aug 2013 11:13:11 +0200

Thanks, Ross and Dan, for the clarifications. Scott, thanks for
identifying these atom clashes (wondering why MOE did not detect them).
No need for me to try with smaller radii, I go with Ross's "The
structure is simply wrong - end of story".

The lesson for me is to run CPU or DPDP-GPU minimizations, especially
when performing many simulations, otherwise one just can't fire and forget.


All the best,

Jan-Philip

On 08/26/2013 05:15 PM, Ross Walker wrote:
> Hi Jan-Philip
>
>
>> My mail was too long, sorry. Important points:
>>
>> - the min1+min2+heat protocol runs fine with the CPU version, i.e. CPU
>> minimization solves the problem. The crash during heatup, however, also
>> appears in the CPU version when starting from the minimized structure as
>> created by the GPU code.
>
> Well, that explains it - it is simply a limitation of the precision model.
> If you use the SPDP or DPDP version of the GPU code it will likely work
> fine. Ultimately though I think the advice should simply be to carry out
> minimization using the CPU code.
>
>>
>> - the error message in the GPU version can be improved. The CPU version
>> informs about too large velocities. The GPU version just says 'launch
>> failure launching kernel kNLSkinTest'.
>
> This is extremely difficult to do without destroying performance.
> Essentially the advice here is the same as it has always been. If you see
> an unexplained crash on the GPU code try it with the CPU. Either way both
> simulations are wrong - just the error message is more informative on the
> CPU.
>
>>
>> - it is very interesting to understand what is wrong with the structure
>> in min2.rst as created by the GPU version. MOE does not identify
>> clashes, the structure looks fine. Any pointers what I can check to
>> identify the problematic part in this structure?
>
> The structure is simply wrong - end of story. There is nothing really to
> identify here. Essentially the forces are too large, they get truncated
> and the resulting gradient does not match the energy. Thus the
> minimization algorithm gets horribly confused since the change in energy
> that occurs when modifying the structure as indicated by the forces is not
> what it expected. Probably if you just stuck with steepest descent it
> 'might' work but the minimizer will still struggle. I think ultimately we
> should just make the truncation fatal and quit with an error message
> saying the structure is too strained for the GPU, switch to CPU
> minimization. I'll see if I can do this for the next update.
>
> In terms of your initial structure, if you really are interested I would
> dump the force array and look at the atoms with the largest forces. You
> could also take the CPU minimization where on step 1 it should report the
> atom with the highest force (GMax) take a look in the vicinity of that
> atom. It's probably coming from the VDW term. Likely two atoms are just
> inside the VDW radius of each other and the r^12 term here gives a massive
> force.
>
>>> If you are using the
>>> latest version of the code it truncates the forces at the largest
>>> representation that SPFP supports - in most cases this works and will
>>> get
>>> you out of trouble but if your initial structure is too strained it will
>>> also likely break the minimizer.
>>
>> When I read in the changelog that the truncation was implemented in GPU
>> minimization code, I switched back to GPU minimization. For the system
>> in question the mean part is that it 'works' in the sense that the
>> minimization does not quit with an error and the output looks fine.
>
> The truncation was a hack that I hoped would work - for most systems it
> seems to work fine but clearly there are exceptions that I wasn't
> anticipating and the minimizer is not as robust as I was hoping. Thus we
> probably need a bugfix to disable this.
>
> All the best
> Ross
>
>
>
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
>


_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Tue Aug 27 2013 - 02:30:03 PDT
Custom Search