Custom Search
-- Ray Luo, Ph.D. Professor of Structural Biology/Biochemistry/Biophysics, Chemical and Materials Physics, Chemical and Biomolecular Engineering, Biomedical Engineering, and Materials Science and Engineering Department of Molecular Biology and Biochemistry University of California, Irvine, CA 92697-3900 On Tue, May 14, 2024 at 3:34 PM Chakrabarti, Mayukh (NIH/NCI) [C] < mayukh.chakrabarti.nih.gov> wrote: > Hi Yongxian, > > > > I wanted to briefly follow up on my previous message with the result of > some additional testing. I compiled Amber24 with Libtorch 2.1.0 and CUDA > 12.0 with the standard ‘release build’, without using the ‘-DCMAKE_BUILD_TYPE=Debug’ > flag. Interestingly, in this build, all of the PBSA tests run successfully, > even including the ‘Run.dmp.sasopt2’ PBSA test that fails using CUDA 11.8 > and CUDA 11.7, as you reported in your earlier message: > > > > cd pbsa_dmp && ./test > > working on ./Run.dmp.sasopt0 > > diffing mdout.dmp.min_0.save with mdout.dmp.min_0 > > PASSED > > ============================================================== > > working on ./Run.dmp.sasopt1 > > diffing mdout.dmp.min_1.save with mdout.dmp.min_1 > > PASSED > > ============================================================== > > working on ./Run.dmp.sasopt2 > > diffing mdout.dmp.min_2.save with mdout.dmp.min_2 > > PASSED > > ============================================================== > > > > This suggests that the SIGSEGV error may at least be related to the > version of CUDA used, if not Libtorch, since the same PBSA test that fails > using Libtorch 2.1.0 and CUDA 11.8 passes when using Libtorch 2.1.0 and > CUDA 12.0. Please let me know if you are able to replicate this behavior if > you get an opportunity to compile Amber24 with Libtorch 2.1.0 and CUDA 12.0. > > > > Best, > > > > *Mayukh Chakrabarti *(he/him) > > COMPUTATIONAL SCIENTIST > > > > *From: *Chakrabarti, Mayukh (NIH/NCI) [C] via AMBER <amber.ambermd.org> > *Date: *Monday, May 13, 2024 at 5:39 PM > *To: *Yongxian Wu <yongxian.wu.uci.edu> > *Cc: *Ray Luo <rluo.uci.edu>, Chakrabarti, Mayukh (NIH/NCI) [C] via AMBER > <amber.ambermd.org> > *Subject: *Re: [AMBER] [EXTERNAL] Re: Amber22/AmberTools23: Enabling of > libtorch & cudnn libraries breaks pbsa binaries > > Hi Yongxian, > > Thank you very much for looking into this further. Based on your advice, I > re-compiled Amber24 with debugging symbols, using my existing Torch 2.1.0 > with CUDA 11.8 setup. Miraculously, upon compiling with the > ‘-DCMAKE_BUILD_TYPE=Debug’ flag, I am able to replicate your testing result > exactly. I did not make any changes to my ‘run_cmake’ compilation script > other than the addition of this build flag. The PBSA tests that previously > failed are now passing, and I encounter the same memory error that you > encounter with Run.dmp.sasopt2: > > working on ./Run.dmp.sasopt2 > > Program received signal SIGSEGV: Segmentation fault - invalid memory > reference. > Backtrace for this error: > > Could not print backtrace: unrecognized DWARF version in .debug_info at 6 > Could not print backtrace: unrecognized DWARF version in .debug_info at 6 > Could not print backtrace: unrecognized DWARF version in .debug_info at 6 > Could not print backtrace: unrecognized DWARF version in .debug_info at 6 > #0 0x7f242a5e8171 in ??? > #1 0x7f242a5e7313 in ??? > #2 0x7f242a06fc0f in ??? > #3 0x452554 in pb_lpbene > at <install-dir> > /amber24_src/AmberTools/src/pbsa/pb_fdfrc.F90:1296 > #4 0x452554 in pb_fdfrc_ > at <install-dir> > /amber24_src/AmberTools/src/pbsa/pb_fdfrc.F90:504 > #5 0x434aaa in __poisson_boltzmann_MOD_pb_force > at <install-dir> > /amber24_src/AmberTools/src/pbsa/pb_force.F90:438 > #6 0x420499 in force_ > at <install-dir> > /amber24_src/AmberTools/src/pbsa/force.F90:168 > #7 0x41fe50 in runmin_ > at <install-dir> > /amber24_src/AmberTools/src/pbsa/runmin.F90:155 > #8 0x40cd28 in pbsa_ > at <install-dir> > /amber24_src/AmberTools/src/pbsa/pbsa.F90:195 > #9 0x40eb4f in pbsamain > at <install-dir> > /amber24_src/AmberTools/src/pbsa/pbsa.F90:26 > #10 0x40eb95 in main > at <install-dir> > /amber24_src/AmberTools/src/pbsa/pbsa.F90:30 > Segmentation fault (core dumped) > ./Run.dmp.sasopt2: Program error > > This makes me wonder, is there some difference in the compilation of the > ‘Debug’ build vs. the default ‘release’ build that could be causing the > PBSA tests to fail in the ‘release’ build, and not in the ‘Debug’ build? > > Out of curiosity, I also went back and re-compiled my version of Amber24 > with Libtorch 1.12.1 and CUDA 10.2 using the ‘-DCMAKE_BUILD_TYPE=Debug’ > flag, but doing this did not fix the SIGSEGV issues in the PBSA tests, nor > did it allow me to obtain a readable backtrace (despite the Debug build). > > Best, > > Mayukh Chakrabarti (he/him) > COMPUTATIONAL SCIENTIST > > From: Yongxian Wu <yongxian.wu.uci.edu> > Date: Saturday, May 11, 2024 at 10:47 PM > To: Chakrabarti, Mayukh (NIH/NCI) [C] <mayukh.chakrabarti.nih.gov> > Cc: Chakrabarti, Mayukh (NIH/NCI) [C] via AMBER <amber.ambermd.org>, Ray > Luo <rluo.uci.edu> > Subject: Re: [AMBER] [EXTERNAL] Re: Amber22/AmberTools23: Enabling of > libtorch & cudnn libraries breaks pbsa binaries > > Hi Mayukh, > > I tested Amber 24 with Libtorch 2.0.1 and CUDA 11.7, and I could not > reproduce the error you encountered. In my tests, both pbsa_bcopt and > pbsa_saopt passed without any segmentation errors. Here is a screenshot of > my testing results. > [cid:ii_lw2xizn40] > > However, I did find a memory issue with pbsa, although it seems unrelated > to Libtorch. When I run multiple pbsa testing processes simultaneously, the > pbsa_dmp test case fails due to an invalid memory reference. > [cid:ii_lw2xjrak1] > > I attempted to compile Amber with debugging symbols to obtain readable > backtrace information. The error message appears as follows: > [cid:ii_lw2xkk822] > > Based on my initial investigation, I believe this problem is caused by a > Fortran memory issue and occurs only when multiple tests are run > simultaneously; otherwise, this test case would pass smoothly without any > errors. > > To solve your problem, I suggest the following steps: > > 1. Please align your environment settings with mine to avoid any > unexpected errors. You can try using Libtorch 2.0.1 + CUDA 11.7, as I have > already tested it. If you have CUDA 11.8 installed, it should be quite > straightforward to switch to CUDA 11.7. > 2. Since the backtrace information attached in your email is > unreadable, you might consider recompiling Amber 24 with debugging symbols > to get a readable error message. To do this, simply add > -DCMAKE_BUILD_TYPE=Debug to your cmake command. > > > Let me know if you need further assistance. > > > > Best, > > Yongxian Wu > > On Fri, May 10, 2024 at 7:07 AM Chakrabarti, Mayukh (NIH/NCI) [C] < > mayukh.chakrabarti.nih.gov<mailto:mayukh.chakrabarti.nih.gov>> wrote: > Hi Yongxian, > > Thank you for testing this. As mentioned in my earlier message, I > re-tested by compiling a version of Amber24 with Pytorch 1.12.1 and CUDA > 10.2. I used ‘libtorch-shared-with-deps-1.12.1+cu102.zip’, which > corresponds to the specific CUDA version. As you mentioned, the compilation > and build process proceeds without any issues. However, while the ‘make > test.serial’ runs and reports the final testing statistics when it > completes, it still reports errors when testing the pbsa binaries, e.g.: > > cd pbsa_bcopt && ./Run.dmp.min > > Program received signal SIGSEGV: Segmentation fault - invalid memory > reference. > > Backtrace for this error: > #0 0x7ff0f7ca0171 in ??? > #1 0x7ff0f7c9f313 in ??? > #2 0x7ff0f735cc0f in ??? > #3 0x7ff15479a219 in ??? > #4 0x7ff0f735f856 in ??? > #5 0x7ff15477b722 in ??? > Segmentation fault (core dumped) > ./Run.dmp.min: Program error > make[2]: [Makefile:199: test.pbsa] Error 1 (ignored) > > cd pbsa_saopt && ./Run.dmp.min > > Program received signal SIGSEGV: Segmentation fault - invalid memory > reference. > > Backtrace for this error: > #0 0x7ff58f59f171 in ??? > #1 0x7ff58f59e313 in ??? > #2 0x7ff58ec5bc0f in ??? > #3 0x7ff5ec099219 in ??? > #4 0x7ff58ec5e856 in ??? > #5 0x7ff5ec07a722 in ??? > Segmentation fault (core dumped) > ./Run.dmp.min: Program error > > I ran ‘ldd’ on both the pbsa and pbsa.cuda binaries, and there don’t > appear to be any missing linked libraries. You mentioned that “Libtorch > should still function even if you encounter errors with pbsa_test”. Does > this mean it is safe to ignore these SIGSEGV errors, assuming that libtorch > is otherwise properly installed and corresponds to the CUDA version being > used? Is it some issue with the test itself? > > Best, > > Mayukh Chakrabarti (he/him) > COMPUTATIONAL SCIENTIST > > From: Yongxian Wu <yongxian.wu.uci.edu<mailto:yongxian.wu.uci.edu>> > Date: Friday, May 10, 2024 at 1:56 AM > To: Chakrabarti, Mayukh (NIH/NCI) [C] <mayukh.chakrabarti.nih.gov<mailto: > mayukh.chakrabarti.nih.gov>>, AMBER Mailing List <amber.ambermd.org > <mailto:amber.ambermd.org>> > Cc: Ray Luo <rluo.uci.edu<mailto:rluo.uci.edu>> > Subject: Re: [AMBER] [EXTERNAL] Re: Amber22/AmberTools23: Enabling of > libtorch & cudnn libraries breaks pbsa binaries > Hi Mayukh, > > I have tested compiling Amber24 with CUDA 11.7.0 and PyTorch 2.0.1. The > compilation and build process proceeded without any issues, and make > test.serial passed without any stopping errors. Please ensure you use the > version of libtorch that corresponds to the specific CUDA version. However, > note that the pbsa_test did not involve libtorch or MLSES. Libtorch should > still function even if you encounter errors with pbsa_test. Here are the > specific variables I used: > > > -DLIBTORCH=ON \ > > -DCUDA_HOME=$CUDA_PATH/cuda_11.7.0 \ > > -DTORCH_HOME=$LIBTORCH_PATH/libtorch \ > > -DCUDA_TOOLKIT_ROOT_DIR=/$CUDA_PATH/cuda_11.7.0 \ > > -DCUDNN=TRUE \ > > -DCUDNN_INCLUDE_PATH=$CUDNN_PATH/include \ > > -DCUDNN_LIBRARY_PATH=$CUDNN_PATH/lib/libcudnn.so \ > > > > > > Best regards, > > Yongxian Wu > > On Wed, May 8, 2024 at 2:26 PM Chakrabarti, Mayukh (NIH/NCI) [C] via > AMBER <amber.ambermd.org<mailto:amber.ambermd.org>> wrote: > Hi Ray and Yongxian, > > Thank you for your response. I don’t have CUDA 11.3 available on my > system, but I do have CUDA 11.8, which I tried to use instead. Upon your > advice, I tried to re-compile Amber 24 & AmberTools 24 using an updated > version of libtorch, version 2.1.0 with CUDA 11.8. I also used > “cudnn-linux-x86_64-8.7.0.84_cuda11-archive.tar.xz”. Unfortunately, > following compilation, I still encounter the same segmentation fault errors > (SIGSEGV) in the pbsa binaries. I will next try with PyTorch 1.12.1 and > CUDA 10.2, and report back. > > Best, > > Mayukh Chakrabarti (he/him) > COMPUTATIONAL SCIENTIST > > From: Ray Luo <rluo.uci.edu<mailto:rluo.uci.edu>> > Date: Wednesday, May 8, 2024 at 1:04 PM > To: Chakrabarti, Mayukh (NIH/NCI) [C] <mayukh.chakrabarti.nih.gov<mailto: > mayukh.chakrabarti.nih.gov>>, AMBER Mailing List <amber.ambermd.org > <mailto:amber.ambermd.org>> > Subject: [EXTERNAL] Re: [AMBER] Amber22/AmberTools23: Enabling of libtorch > & cudnn libraries breaks pbsa binaries > Hi Mayukh, > > The issue might be due to the version of PyTorch being used. > > I recommend using the same version of libtorch that we currently have, > which is version 1.12.1 with CUDA 11.3. PyTorch 1.9 with CUDA 10.2 is > outdated. However, PyTorch 1.12.1 also supports CUDA 10.2, although we > recommend using CUDA 11 since CUDA 10 is outdated. > > Let me know if there are any other questions. > > Best, > Yongxian and Ray > -- > Ray Luo, Ph.D. > Professor of Structural Biology/Biochemistry/Biophysics, > Chemical and Materials Physics, Chemical and Biomolecular Engineering, > Biomedical Engineering, and Materials Science and Engineering > Department of Molecular Biology and Biochemistry > University of California, Irvine, CA 92697-3900 > > > On Tue, May 7, 2024 at 8:57 AM Chakrabarti, Mayukh (NIH/NCI) [C] via > AMBER <amber.ambermd.org<mailto:amber.ambermd.org><mailto: > amber.ambermd.org<mailto:amber.ambermd.org>>> wrote: > Hello, > > I wanted to update my report below to mention that the issue with pbsa > binaries breaking upon enabling the LibTorch & cudnn libraries still > persists in Amber24 with AmberTools 24. I compiled with CUDA 10.2, gcc 8.5, > OpenMPI 4.1.5, python 3.10, and cmake 3.25.2 on a Red Hat Enterprise Linux > release 8.8 (Ootpa) system. I am not aware of any workaround or resolution > for this issue. > > Best, > > Mayukh Chakrabarti (he/him) > COMPUTATIONAL SCIENTIST > > From: Chakrabarti, Mayukh (NIH/NCI) [C] <mayukh.chakrabarti.nih.gov > <mailto:mayukh.chakrabarti.nih.gov><mailto:mayukh.chakrabarti.nih.gov > <mailto:mayukh.chakrabarti.nih.gov>>> > Date: Tuesday, October 31, 2023 at 11:28 AM > To: amber.ambermd.org< > mailto:amber.ambermd.org><mailto:amber.ambermd.org<mailto:amber.ambermd.org > <amber.ambermd.org%3e%3cmailto:amber.ambermd.org%3cmailto:amber.ambermd.org>>> > <amber.ambermd.org<mailto:amber.ambermd.org><mailto:amber.ambermd.org > <mailto:amber.ambermd.org>>> > Subject: Amber22/AmberTools23: Enabling of libtorch & cudnn libraries > breaks pbsa binaries > Hello, > > I am encountering a problem in which enabling LibTorch libraries with > Amber22 and AmberTools23 causes segmentation fault errors (SIGSEGV) in the > pbsa binaries upon running the serial tests (make test.serial). I have > successfully compiled a version of Amber22/AmberTools23 in which this > library is not enabled, and none of the pbsa tests in AmberTools break > (i.e., no segmentation fault errors). > > Further details: > > I am running my compilation with CUDA 10.2, gcc 8.5, OpenMPI 4.1.5, python > 3.6, and cmake 3.25.1 on a Red Hat Enterprise Linux release 8.8 (Ootpa) > system. As per the manual, I have tried both “Built-in” mode and > “User-installed” mode, both resulting in the same errors. For the > “User-installed” mode, I manually downloaded and extracted > “libtorch-shared-with-deps-1.9.1+cu102.zip” from the PyTorch website to > correspond to CUDA 10.2, and > “cudnn-linux-x86_64-8.7.0.84_cuda10-archive.tar.xz” directly from the > NVIDIA website, and specified the following variables to CMAKE: > > > -DLIBTORCH=ON \ > > -DTORCH_HOME=/path_to_libtorch \ > > -DCUDNN=TRUE \ > > -DCAFFE2_USE_CUDNN=1 \ > > -DCUDNN_INCLUDE_PATH=/path_to_cudnn_include \ > > -DCUDNN_LIBRARY_PATH=/path_to_libcudnn.so \ > > > Building and compiling proceed without issue. However, when running the > serial tests, I get errors akin to the following (example shown after > sourcing amber.sh and running AmberTools pbsa_ligand test): > > Program received signal SIGSEGV: Segmentation fault - invalid memory > reference. > > Backtrace for this error: > #0 0x7f480cb65171 in ??? > #1 0x7f480cb64313 in ??? > #2 0x7f480c221c0f in ??? > #3 0x7f4869667219 in ??? > #4 0x7f480c224856 in ??? > #5 0x7f4869648722 in ??? > ./Run.t4bnz.min: line 34: 2436022 Segmentation fault (core dumped) > $DO_PARALLEL $TESTpbsa -O -i min.in< > https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__http%3A%2F%2Fmin.in%2F__%3B!!CzAuKJ42GuquVTTmVmPViYEvSg!MVPl6p4KbMJpbOqO568Ns0Kz5i7sbGNJ-naGNHZf9Ycly3ycDbDToZ4J3VrVyvCP1-1IQPwcf2tM6Z4nwjoll1JKRHaF82Q%24&data=05%7C02%7Cmayukh.chakrabarti%40nih.gov%7C31c8ab4842ad46791eb808dc73952e2e%7C14b77578977342d58507251ca2dc2b06%7C0%7C0%7C638512331794003736%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=VbrBfg1xWZ%2BAhn%2BsT5WX16%2BYYD3278mBiheBepnrBWE%3D&reserved=0 > <https://urldefense.com/v3/__http://min.in/__;!!CzAuKJ42GuquVTTmVmPViYEvSg!MVPl6p4KbMJpbOqO568Ns0Kz5i7sbGNJ-naGNHZf9Ycly3ycDbDToZ4J3VrVyvCP1-1IQPwcf2tM6Z4nwjoll1JKRHaF82Q$> > >< > https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__http%3A%2F%2Fmin.in%2F__%3B!!CzAuKJ42GuquVTTmVmPViYEvSg!NC0Jwxkjnbm_mXLPxsbm3fva29jev44srLmSv7hv0XZuaw716Egbv1itPhJek6uNYpgJ3GbSbeK6jyZvAA%24&data=05%7C02%7Cmayukh.chakrabarti%40nih.gov%7C31c8ab4842ad46791eb808dc73952e2e%7C14b77578977342d58507251ca2dc2b06%7C0%7C0%7C638512331794014226%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=%2FUBhJta6XreNd4aiAPCX6s%2F8yvIK8X7oAMeqNv9II7I%3D&reserved=0 > > -o $output < /dev/null > ./Run.t4bnz.min: Program error > > When running the exact same test with the version of Amber22 not > containing the LibTorch libraries: > > diffing mdout.lig.min.save with mdout.lig.min > PASSED > ============================================================== > > I have run ‘ldd’ on the pbsa binary to try to identify any libraries that > may be missing, but there don’t appear to be any missing libraries. Could > anybody please provide any insight into how to fix this issue? > > Best, > > Mayukh Chakrabarti > COMPUTATIONAL SCIENTIST > > > > > > > > _______________________________________________ > AMBER mailing list > AMBER.ambermd.org< > mailto:AMBER.ambermd.org><mailto:AMBER.ambermd.org<mailto:AMBER.ambermd.org > <AMBER.ambermd.org%3e%3cmailto:AMBER.ambermd.org%3cmailto:AMBER.ambermd.org> > >> > > https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__http%3A%2F%2Flists.ambermd.org%2Fmailman%2Flistinfo%2Famber__%3B!!CzAuKJ42GuquVTTmVmPViYEvSg!PLN2_ZzXkAP-XJnmppjnXaq9ruj3FNFdRR163QR8ehrjX6YYBOUfKc8Eo0t0w7N8I3CtOwXatyYNzg%24&data=05%7C02%7Cmayukh.chakrabarti%40nih.gov%7C31c8ab4842ad46791eb808dc73952e2e%7C14b77578977342d58507251ca2dc2b06%7C0%7C0%7C638512331794021432%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=z5CkieyNhSwJUIqvRSv2keiwyOhCy%2BIVC3GObhYVZOE%3D&reserved=0 > <https://urldefense.com/v3/__http://lists.ambermd.org/mailman/listinfo/amber__;!!CzAuKJ42GuquVTTmVmPViYEvSg!PLN2_ZzXkAP-XJnmppjnXaq9ruj3FNFdRR163QR8ehrjX6YYBOUfKc8Eo0t0w7N8I3CtOwXatyYNzg$> > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you recognize the sender and are > confident the content is safe. > > _______________________________________________ > AMBER mailing list > AMBER.ambermd.org<mailto:AMBER.ambermd.org <AMBER.ambermd.org>> > > https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__http%3A%2F%2Flists.ambermd.org%2Fmailman%2Flistinfo%2Famber__%3B!!CzAuKJ42GuquVTTmVmPViYEvSg!NC0Jwxkjnbm_mXLPxsbm3fva29jev44srLmSv7hv0XZuaw716Egbv1itPhJek6uNYpgJ3GbSbeI1bOw3SQ%24&data=05%7C02%7Cmayukh.chakrabarti%40nih.gov%7C31c8ab4842ad46791eb808dc73952e2e%7C14b77578977342d58507251ca2dc2b06%7C0%7C0%7C638512331794025630%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=D82tN0NRCLeteENhW9O2VL7e1cftjYCK%2BvxSVZkqdIs%3D&reserved=0 > <https://urldefense.com/v3/__http://lists.ambermd.org/mailman/listinfo/amber__;!!CzAuKJ42GuquVTTmVmPViYEvSg!NC0Jwxkjnbm_mXLPxsbm3fva29jev44srLmSv7hv0XZuaw716Egbv1itPhJek6uNYpgJ3GbSbeI1bOw3SQ$> > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you recognize the sender and are > confident the content is safe. > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you recognize the sender and are > confident the content is safe. > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you recognize the sender and are > confident the content is safe. > _______________________________________________ AMBER mailing list AMBER.ambermd.org http://lists.ambermd.org/mailman/listinfo/amberReceived on Thu May 16 2024 - 15:00:02 PDT