Re: [AMBER] 4.5 fold increase in memory consumption for nmr restraints going from amber14 to amber16 from Charles Lin on 2017-05-09 (Amber Archive May 2017)

From: Charles Lin <clin92.ucsd.edu>
Date: Tue, 9 May 2017 16:02:45 +0000

Hi Pratul,

If you want that would be really helpful.

The next thing I was going to look at was checking performance of amber14 update.9 vs update.11. I was in the modifying my scripts around to making it work due to the code changes between amber14 vs amber16 (Primarily because I use cuda 8.0 and some of the assembly functions originally built for amber 14 became intrinsics when cuda 8.0 came around.). If the memory hit is there I was going to take a look at whether or not the allocation of the COMs code was correct or not first. This also gives you a diff to look at in terms of changes that might have caused the errors (The amber update files are essentially diff files so if you want to apply them individually you can download them and do a "patch -p0 < updatefile" command. http://ambermd.org/bugfixes14.html). If you want we can speak off the mailing list about this.

Charlie
________________________________________
From: Pratul K. Agarwal [agarwalpk.ornl.gov]
Sent: Tuesday, May 09, 2017 8:36 AM
To: AMBER Mailing List
Subject: Re: [AMBER] 4.5 fold increase in memory consumption for nmr restraints going from amber14 to amber16

Charlie,

Ilooked at the code, I may take a stab at it but just like you I am also
in the middle of many things. If you have any pointers on where to look
please send me a note. First place I was looking at was indexing of the
COM group atoms.

Paul,

Question: Did you compare the output with CPU(s) only run? Is it
similar? In my case whenwe used COM restraints(between two groups with
about 25 atoms), we were getting significant differencesbetween the GPU
and CPU runs.

Pratul

Pratul K. Agarwal, Ph.D.
(Editorial Board Member: PLoS ONE, Microbial Cell Factories)
Computational Biology Institute &
Computer Science and Engineering Division
Oak Ridge National Laboratory, Oak Ridge, TN 37831, USA
Phone: +1-865-574-7184
Web: http://www.agarwal-lab.org/

On 5/9/2017 11:24 AM, Charles Lin wrote:
> Its on my todo list. I took look to see if there was anything obvious that was causing the issue. Unfortunately, I'm fairly swamped, so there's no estimate on when it'll be done unless someone else wants to take a stab at it.
>
> Charlie
> ________________________________________
> From: Paul Westphaelinger [paul_westphaelinger.ewetel.net]
> Sent: Tuesday, May 09, 2017 1:00 AM
> To: AMBER Mailing List
> Subject: Re: [AMBER] 4.5 fold increase in memory consumption for nmr restraints going from amber14 to amber16
>
> Hello,
>
> just wanted to ask if there has been any progress.
>
> Best regards,
>
> Paul Westphälinger
>
> Am 05.04.2017 um 22:03 schrieb Paul Westphaelinger:
>> Hi,
>>
>> thanks.
>>
>> Best regards,
>> Paul Westphälinger
>>
>> Am 05.04.2017 um 21:46 schrieb Charles Lin:
>>> I was able to reproduce the error. I'll take a look into it.
>>>
>>> Thanks,
>>> Charlie
>>> ________________________________________
>>> From: Paul Westphaelinger [paul_westphaelinger.ewetel.net]
>>> Sent: Thursday, March 30, 2017 12:46 PM
>>> To: AMBER Mailing List
>>> Subject: Re: [AMBER] 4.5 fold increase in memory consumption for nmr restraints going from amber14 to amber16
>>>
>>> yes
>>>
>>>
>>> Am 30.03.2017 um 21:26 schrieb Charles Lin:
>>>> Hi,
>>>>
>>>> I've been meaning to take a look at this, but as I am currently in the middle of relocating, my resources are currently very limited for another week or so. Both amber14 and amber16 being run are fully patched correct?
>>>>
>>>> Thanks,
>>>> Charlie
>>>>
>>>> ________________________________________
>>>> From: Paul Westphälinger [paul_westphaelinger.ewetel.net]
>>>> Sent: Thursday, March 30, 2017 7:57 AM
>>>> To: AMBER Mailing List
>>>> Subject: [AMBER] 4.5 fold increase in memory consumption for nmr restraints going from amber14 to amber16
>>>>
>>>> Hello,
>>>>
>>>> I noticed a dramatic memory usage increase going from amber14 to amber16 when using nmr center
>>>> of mass restraints. I have documented this in this email:
>>>>
>>>>
>>>> I have run four simulations with nmropt=0 and nmropt=1 and both with
>>>> amber14.13 and amber16. Everything else exactly identical. The restrained
>>>> atom groups contain only three atoms each, and only one center of mass restrain is used
>>>> in the simulation.
>>>>
>>>> I posted the nvidia-smi outputs below in this email, and I uploaded the testcase file to this file
>>>> sharing service of my university:https://syncandshare.lrz.de/filestable/MkR2Qld2TDlWZXo4N3N5WXdndG1y
>>>> The ex_init.sh files execute the simulations, I would appreciate to be informed if these
>>>> test cases work for you or not. (If you are interested in receiving these files via mail
>>>> I'll be happy to send them to you or to the whole mailing list if that is desired)
>>>>
>>>>
>>>> To the nvidia-smi outputs:
>>>>
>>>> Please pay attention only to the 0-th GPU as this is the one I was
>>>> using. Notice the increase in memory use even without nmr restraints
>>>> from amber14 to 16 is small while with nmr restraint it is about a
>>>> factor of 4.5.
>>>>
>>>>
>>>> Best,
>>>>
>>>> Paul Westphälinger
>>>>
>>>>
>>>> I'll post the nvidia-smi outputs here:
>>>>
>>>> amber 14 with nmropt=1:
>>>>
>>>> Tue Mar 21 12:21:54 2017
>>>> +------------------------------------------------------+
>>>> | NVIDIA-SMI 352.93 Driver Version: 352.93 |
>>>> |-------------------------------+----------------------+----------------------+
>>>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile
>>>> Uncorr. ECC |
>>>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
>>>> Compute M. |
>>>> |===============================+======================+======================|
>>>> | 0 GeForce GTX TIT... Off | 0000:02:00.0 Off |
>>>> N/A |
>>>> | 30% 72C P2 177W / 250W | 292MiB / 12287MiB | 97% Default |
>>>> +-------------------------------+----------------------+----------------------+
>>>> | 1 GeForce GTX TIT... Off | 0000:03:00.0 Off |
>>>> N/A |
>>>> | 33% 73C P2 204W / 250W | 1262MiB / 12287MiB | 97% Default |
>>>> +-------------------------------+----------------------+----------------------+
>>>> | 2 GeForce GTX TIT... Off | 0000:81:00.0 Off |
>>>> N/A |
>>>> | 22% 34C P8 15W / 250W | 23MiB / 12287MiB | 0% Default |
>>>> +-------------------------------+----------------------+----------------------+
>>>> | 3 GeForce GTX TIT... Off | 0000:82:00.0 Off |
>>>> N/A |
>>>> | 22% 29C P8 15W / 250W | 23MiB / 12287MiB | 0% Default |
>>>> +-------------------------------+----------------------+----------------------+
>>>>
>>>> +-----------------------------------------------------------------------------+
>>>> | Processes: GPU Memory |
>>>> | GPU PID Type Process name Usage |
>>>> |=============================================================================|
>>>> | 0 28177 C pmemd.cuda
>>>> 267MiB |
>>>> | 1 25712 C /home/fzeller/amber16//bin/pmemd.cuda
>>>> 1237MiB |
>>>> +-----------------------------------------------------------------------------+
>>>>
>>>>
>>>> amber 14 with nmropt=0:
>>>>
>>>> Tue Mar 21 12:20:44 2017
>>>> +------------------------------------------------------+
>>>> | NVIDIA-SMI 352.93 Driver Version: 352.93 |
>>>> |-------------------------------+----------------------+----------------------+
>>>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile
>>>> Uncorr. ECC |
>>>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
>>>> Compute M. |
>>>> |===============================+======================+======================|
>>>> | 0 GeForce GTX TIT... Off | 0000:02:00.0 Off |
>>>> N/A |
>>>> | 22% 63C P2 188W / 250W | 292MiB / 12287MiB | 99% Default |
>>>> +-------------------------------+----------------------+----------------------+
>>>> | 1 GeForce GTX TIT... Off | 0000:03:00.0 Off |
>>>> N/A |
>>>> | 33% 73C P2 205W / 250W | 1262MiB / 12287MiB | 97% Default |
>>>> +-------------------------------+----------------------+----------------------+
>>>> | 2 GeForce GTX TIT... Off | 0000:81:00.0 Off |
>>>> N/A |
>>>> | 22% 34C P8 15W / 250W | 23MiB / 12287MiB | 0% Default |
>>>> +-------------------------------+----------------------+----------------------+
>>>> | 3 GeForce GTX TIT... Off | 0000:82:00.0 Off |
>>>> N/A |
>>>> | 22% 29C P8 15W / 250W | 23MiB / 12287MiB | 0% Default |
>>>> +-------------------------------+----------------------+----------------------+
>>>>
>>>> +-----------------------------------------------------------------------------+
>>>> | Processes: GPU Memory |
>>>> | GPU PID Type Process name Usage |
>>>> |=============================================================================|
>>>> | 0 28138 C pmemd.cuda
>>>> 267MiB |
>>>> | 1 25712 C /home/fzeller/amber16//bin/pmemd.cuda
>>>> 1237MiB |
>>>> +-----------------------------------------------------------------------------+
>>>>
>>>>
>>>> amber 16 with nmropt=1:
>>>>
>>>> Tue Mar 21 12:17:47 2017
>>>> +------------------------------------------------------+
>>>> | NVIDIA-SMI 352.93 Driver Version: 352.93 |
>>>> |-------------------------------+----------------------+----------------------+
>>>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile
>>>> Uncorr. ECC |
>>>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
>>>> Compute M. |
>>>> |===============================+======================+======================|
>>>> | 0 GeForce GTX TIT... Off | 0000:02:00.0 Off |
>>>> N/A |
>>>> | 23% 65C P2 194W / 250W | 1295MiB / 12287MiB | 96% Default |
>>>> +-------------------------------+----------------------+----------------------+
>>>> | 1 GeForce GTX TIT... Off | 0000:03:00.0 Off |
>>>> N/A |
>>>> | 32% 73C P2 205W / 250W | 1262MiB / 12287MiB | 97% Default |
>>>> +-------------------------------+----------------------+----------------------+
>>>> | 2 GeForce GTX TIT... Off | 0000:81:00.0 Off |
>>>> N/A |
>>>> | 22% 34C P8 15W / 250W | 23MiB / 12287MiB | 0% Default |
>>>> +-------------------------------+----------------------+----------------------+
>>>> | 3 GeForce GTX TIT... Off | 0000:82:00.0 Off |
>>>> N/A |
>>>> | 22% 29C P8 15W / 250W | 23MiB / 12287MiB | 0% Default |
>>>> +-------------------------------+----------------------+----------------------+
>>>>
>>>> +-----------------------------------------------------------------------------+
>>>> | Processes: GPU Memory |
>>>> | GPU PID Type Process name Usage |
>>>> |=============================================================================|
>>>> | 0 28007 C pmemd.cuda
>>>> 1270MiB |
>>>> | 1 25712 C /home/fzeller/amber16//bin/pmemd.cuda
>>>> 1237MiB |
>>>> +-----------------------------------------------------------------------------+
>>>>
>>>>
>>>> amber 16 with nmropt=0:
>>>>
>>>> Tue Mar 21 12:19:16 2017
>>>> +------------------------------------------------------+
>>>> | NVIDIA-SMI 352.93 Driver Version: 352.93 |
>>>> |-------------------------------+----------------------+----------------------+
>>>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile
>>>> Uncorr. ECC |
>>>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
>>>> Compute M. |
>>>> |===============================+======================+======================|
>>>> | 0 GeForce GTX TIT... Off | 0000:02:00.0 Off |
>>>> N/A |
>>>> | 33% 76C P2 202W / 250W | 327MiB / 12287MiB | 99% Default |
>>>> +-------------------------------+----------------------+----------------------+
>>>> | 1 GeForce GTX TIT... Off | 0000:03:00.0 Off |
>>>> N/A |
>>>> | 33% 73C P2 203W / 250W | 1262MiB / 12287MiB | 97% Default |
>>>> +-------------------------------+----------------------+----------------------+
>>>> | 2 GeForce GTX TIT... Off | 0000:81:00.0 Off |
>>>> N/A |
>>>> | 22% 34C P8 15W / 250W | 23MiB / 12287MiB | 0% Default |
>>>> +-------------------------------+----------------------+----------------------+
>>>> | 3 GeForce GTX TIT... Off | 0000:82:00.0 Off |
>>>> N/A |
>>>> | 22% 30C P8 15W / 250W | 23MiB / 12287MiB | 0% Default |
>>>> +-------------------------------+----------------------+----------------------+
>>>>
>>>> +-----------------------------------------------------------------------------+
>>>> | Processes: GPU Memory |
>>>> | GPU PID Type Process name Usage |
>>>> |=============================================================================|
>>>> | 0 28048 C pmemd.cuda
>>>> 302MiB |
>>>> | 1 25712 C /home/fzeller/amber16//bin/pmemd.cuda
>>>> 1237MiB |
>>>> +-----------------------------------------------------------------------------+
>>>>
>>>>
>>>> Greetings,
>>>>
>>>> Paul Westphälinger
>>>>
>>>> _______________________________________________
>>>> AMBER mailing list
>>>> AMBER.ambermd.org
>>>> http://lists.ambermd.org/mailman/listinfo/amber
>>>>
>>>> _______________________________________________
>>>> AMBER mailing list
>>>> AMBER.ambermd.org
>>>> http://lists.ambermd.org/mailman/listinfo/amber
>>>>
>>> _______________________________________________
>>> AMBER mailing list
>>> AMBER.ambermd.org
>>> http://lists.ambermd.org/mailman/listinfo/amber
>>>
>>> _______________________________________________
>>> AMBER mailing list
>>> AMBER.ambermd.org
>>> http://lists.ambermd.org/mailman/listinfo/amber
>>>
>> _______________________________________________
>> AMBER mailing list
>> AMBER.ambermd.org
>> http://lists.ambermd.org/mailman/listinfo/amber
>>
>
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
>
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
>

_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber

_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Tue May 09 2017 - 09:30:03 PDT