Hi everybody,
I've been running repeatedly pmemd.MPI on a 16 core Linux machine without any problems. Till I came across segmentation fault error. I'm attaching the following for diagnostics.
Your suggestions will be greatly appreciated.
Best regards
George
Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
[imp200:04485] *** Process received signal ***
[imp200:04485] Signal: Segmentation fault (11)
[imp200:04485] Signal code: Address not mapped (1)
[imp200:04485] Failing at address: 0x221f5d40
[imp201:06851] *** Process received signal ***
[imp201:06851] Signal: Segmentation fault (11)
[imp201:06851] Signal code: Address not mapped (1)
[imp201:06851] Failing at address: 0x19327a90
[imp200:04485] [ 0] /lib64/libpthread.so.0 [0x3ebfe0eb10]
[imp200:04485] [ 1] /csoft/amber11-parallel/exe/pmemd.MPI(nb_pairlist_mod_mp_map
_pairlist_imgs_+0x1f7) [0x488497]
[imp200:04485] [ 2] /csoft/amber11-parallel/exe/pmemd.MPI(pme_direct_mod_mp_pme_
list_+0x1c9) [0x484c19]
[imp200:04485] [ 3] /csoft/amber11-parallel/exe/pmemd.MPI(pme_force_mod_mp_pme_f
orce_+0x26b) [0x47f0ab]
[imp200:04485] [ 4] /csoft/amber11-parallel/exe/pmemd.MPI(runmd_mod_mp_runmd_+0x
4c1f) [0x46acaf]
[imp200:04485] [ 5] /csoft/amber11-parallel/exe/pmemd.MPI(MAIN__+0x8ca) [0x41ae3
a]
[imp200:04485] [ 6] /csoft/amber11-parallel/exe/pmemd.MPI(main+0x3c) [0x41a55c]
[imp200:04485] [ 7] /lib64/libc.so.6(__libc_start_main+0xf4) [0x3ebf21d994]
[imp200:04485] [ 8] /csoft/amber11-parallel/exe/pmemd.MPI [0x41a469]
[imp201:06851] [ 0] /lib64/libpthread.so.0 [0x381b40eb10]
[imp201:06851] [ 1] /csoft/amber11-parallel/exe/pmemd.MPI(nb_pairlist_mod_mp_map
_pairlist_imgs_+0x1f7) [0x488497]
[imp201:06851] [ 2] /csoft/amber11-parallel/exe/pmemd.MPI(pme_direct_mod_mp_pme_
list_+0x1c9) [0x484c19]
[imp201:06851] [ 3] /csoft/amber11-parallel/exe/pmemd.MPI(pme_force_mod_mp_pme_f
orce_+0x26b) [0x47f0ab]
[imp200:04485] *** End of error message ***
[imp201:06851] [ 4] /csoft/amber11-parallel/exe/pmemd.MPI(runmd_mod_mp_runmd_+0x
4c1f) [0x46acaf]
[imp201:06851] [ 5] /csoft/amber11-parallel/exe/pmemd.MPI(MAIN__+0x8ca) [0x41ae3
a]
[imp201:06851] [ 6] /csoft/amber11-parallel/exe/pmemd.MPI(main+0x3c) [0x41a55c]
[imp201:06851] [ 7] /lib64/libc.so.6(__libc_start_main+0xf4) [0x381a81d994]
[imp201:06851] [ 8] /csoft/amber11-parallel/exe/pmemd.MPI [0x41a469]
[imp201:06851] *** End of error message ***
[imp105:05048] *** Process received signal ***
[imp105:05048] Signal: Segmentation fault (11)
[imp105:05048] Signal code: Address not mapped (1)
[imp105:05048] Failing at address: 0x1cb09710
[imp105:05048] [ 0] /lib64/libpthread.so.0 [0x34dfc0eb10]
[imp105:05048] [ 1] /csoft/amber11-parallel/exe/pmemd.MPI(nb_pairlist_mod_mp_map
_pairlist_imgs_+0x1f7) [0x488497]
[imp105:05048] [ 2] /csoft/amber11-parallel/exe/pmemd.MPI(pme_direct_mod_mp_pme_
list_+0x1c9) [0x484c19]
[imp105:05048] [ 3] /csoft/amber11-parallel/exe/pmemd.MPI(pme_force_mod_mp_pme_f
orce_+0x26b) [0x47f0ab]
[imp105:05048] [ 4] /csoft/amber11-parallel/exe/pmemd.MPI(runmd_mod_mp_runmd_+0x
4c1f) [0x46acaf]
[imp105:05048] [ 5] /csoft/amber11-parallel/exe/pmemd.MPI(MAIN__+0x8ca) [0x41ae3
a]
[imp105:05048] [ 6] /csoft/amber11-parallel/exe/pmemd.MPI(main+0x3c) [0x41a55c]
[imp105:05048] [ 7] /lib64/libc.so.6(__libc_start_main+0xf4) [0x34df01d994]
[imp105:05048] [ 8] /csoft/amber11-parallel/exe/pmemd.MPI [0x41a469]
[imp105:05048] *** End of error message ***
[imp202.open.ac.uk][0,1,15][btl_tcp_frag.c:202:mca_btl_tcp_frag_recv] mca_btl_tc
p_frag_recv: readv failed with errno=104
[imp113.open.ac.uk][0,1,14][btl_tcp_frag.c:202:mca_btl_tcp_frag_recv] mca_btl_tc
p_frag_recv: readv failed with errno=104
[imp109.open.ac.uk][0,1,12][btl_tcp_frag.c:202:mca_btl_tcp_frag_recv] mca_btl_tc
p_frag_recv: readv failed with errno=104
[imp101.open.ac.uk][0,1,10][btl_tcp_frag.c:202:mca_btl_tcp_frag_recv] mca_btl_tc
p_frag_recv: readv failed with errno=104
[imp110.open.ac.uk][0,1,7][btl_tcp_frag.c:202:mca_btl_tcp_frag_recv] mca_btl_tcp
_frag_recv: readv failed with errno=104
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libc.so.6 00000035A3C306D0 Unknown Unknown Unknown
libopen-pal.so.0 00002AAC94E9E2DD Unknown Unknown Unknown
libopen-pal.so.0 00002AAC94E9ECB6 Unknown Unknown Unknown
libopen-pal.so.0 00002AAC94E9B5FC Unknown Unknown Unknown
libopen-pal.so.0 00002AAC94E9B484 Unknown Unknown Unknown
libopen-pal.so.0 00002AAC94E95AC7 Unknown Unknown Unknown
libmpi.so.0 00002AAC9497CC29 Unknown Unknown Unknown
libmpi.so.0 00002AAC949B0D21 Unknown Unknown Unknown
libmpi_f77.so.0 00002AAC947301B9 Unknown Unknown Unknown
pmemd.MPI 0000000000493E28 Unknown Unknown Unknown
pmemd.MPI 0000000000491FD3 Unknown Unknown Unknown
pmemd.MPI 000000000048EC3D Unknown Unknown Unknown
pmemd.MPI 00000000004814BA Unknown Unknown Unknown
pmemd.MPI 000000000046ACAF Unknown Unknown Unknown
pmemd.MPI 000000000041AE3A Unknown Unknown Unknown
pmemd.MPI 000000000041A55C Unknown Unknown Unknown
libc.so.6 00000035A3C1D994 Unknown Unknown Unknown
pmemd.MPI 000000000041A469 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libc.so.6 00000031D8E306D0 Unknown Unknown Unknown
libopen-pal.so.0 00002AF2189672DD Unknown Unknown Unknown
libopen-pal.so.0 00002AF218967C41 Unknown Unknown Unknown
libopen-pal.so.0 00002AF2189644E1 Unknown Unknown Unknown
libopen-pal.so.0 00002AF218964484 Unknown Unknown Unknown
libopen-pal.so.0 00002AF21895EAC7 Unknown Unknown Unknown
libmpi.so.0 00002AF218445C29 Unknown Unknown Unknown
libmpi.so.0 00002AF218479D21 Unknown Unknown Unknown
libmpi_f77.so.0 00002AF2181F91B9 Unknown Unknown Unknown
pmemd.MPI 0000000000493E28 Unknown Unknown Unknown
pmemd.MPI 0000000000491FD3 Unknown Unknown Unknown
pmemd.MPI 000000000048EC3D Unknown Unknown Unknown
pmemd.MPI 00000000004814BA Unknown Unknown Unknown
pmemd.MPI 000000000046ACAF Unknown Unknown Unknown
pmemd.MPI 000000000041AE3A Unknown Unknown Unknown
pmemd.MPI 000000000041A55C Unknown Unknown Unknown
libc.so.6 00000031D8E1D994 Unknown Unknown Unknown
pmemd.MPI 000000000041A469 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libpthread.so.0 000000316EE0EC12 Unknown Unknown Unknown
libopen-pal.so.0 00002B0393A013F4 Unknown Unknown Unknown
libopen-pal.so.0 00002B0393A01CB6 Unknown Unknown Unknown
libopen-pal.so.0 00002B03939FE5FC Unknown Unknown Unknown
libopen-pal.so.0 00002B03939FE484 Unknown Unknown Unknown
libopen-pal.so.0 00002B03939F8AC7 Unknown Unknown Unknown
libmpi.so.0 00002B03934DFC29 Unknown Unknown Unknown
libmpi.so.0 00002B0393513D21 Unknown Unknown Unknown
libmpi_f77.so.0 00002B03932931B9 Unknown Unknown Unknown
pmemd.MPI 00000000004AEA6F Unknown Unknown Unknown
pmemd.MPI 000000000047FEF5 Unknown Unknown Unknown
pmemd.MPI 000000000046ACAF Unknown Unknown Unknown
pmemd.MPI 000000000041AE3A Unknown Unknown Unknown
pmemd.MPI 000000000041A55C Unknown Unknown Unknown
libc.so.6 000000316E21D994 Unknown Unknown Unknown
pmemd.MPI 000000000041A469 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libc.so.6 000000361FA797C0 Unknown Unknown Unknown
libc.so.6 000000361FA46B69 Unknown Unknown Unknown
libc.so.6 000000361FA47F36 Unknown Unknown Unknown
libc.so.6 000000361FA42A1C Unknown Unknown Unknown
libmpi.so.0 00002ABBD325FE76 Unknown Unknown Unknown
mca_btl_tcp.so 00002ABBD940CD8B Unknown Unknown Unknown
mca_btl_tcp.so 00002ABBD940B9D2 Unknown Unknown Unknown
libopen-pal.so.0 00002ABBD374685D Unknown Unknown Unknown
libopen-pal.so.0 00002ABBD3746669 Unknown Unknown Unknown
libopen-pal.so.0 00002ABBD3746484 Unknown Unknown Unknown
libopen-pal.so.0 00002ABBD3740AC7 Unknown Unknown Unknown
libmpi.so.0 00002ABBD3227C29 Unknown Unknown Unknown
libmpi.so.0 00002ABBD325BD21 Unknown Unknown Unknown
libmpi_f77.so.0 00002ABBD2FDB1B9 Unknown Unknown Unknown
pmemd.MPI 0000000000493E28 Unknown Unknown Unknown
pmemd.MPI 0000000000491FD3 Unknown Unknown Unknown
pmemd.MPI 000000000048EC3D Unknown Unknown Unknown
pmemd.MPI 00000000004814BA Unknown Unknown Unknown
pmemd.MPI 000000000046ACAF Unknown Unknown Unknown
pmemd.MPI 000000000041AE3A Unknown Unknown Unknown
pmemd.MPI 000000000041A55C Unknown Unknown Unknown
libc.so.6 000000361FA1D994 Unknown Unknown Unknown
pmemd.MPI 000000000041A469 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libpthread.so.0 000000359B80EC12 Unknown Unknown Unknown
libopen-pal.so.0 00002B9F4D0983F4 Unknown Unknown Unknown
libopen-pal.so.0 00002B9F4D098C41 Unknown Unknown Unknown
libopen-pal.so.0 00002B9F4D0954E1 Unknown Unknown Unknown
libopen-pal.so.0 00002B9F4D095484 Unknown Unknown Unknown
libopen-pal.so.0 00002B9F4D08FAC7 Unknown Unknown Unknown
libmpi.so.0 00002B9F4CB76C29 Unknown Unknown Unknown
libmpi.so.0 00002B9F4CBAAD21 Unknown Unknown Unknown
libmpi_f77.so.0 00002B9F4C92A1B9 Unknown Unknown Unknown
pmemd.MPI 00000000004AEA6F Unknown Unknown Unknown
pmemd.MPI 000000000047FEF5 Unknown Unknown Unknown
pmemd.MPI 000000000046ACAF Unknown Unknown Unknown
pmemd.MPI 000000000041AE3A Unknown Unknown Unknown
pmemd.MPI 000000000041A55C Unknown Unknown Unknown
libc.so.6 000000359AC1D994 Unknown Unknown Unknown
pmemd.MPI 000000000041A469 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libpthread.so.0 000000359B80EC14 Unknown Unknown Unknown
libopen-pal.so.0 00002AD79259E3F4 Unknown Unknown Unknown
libopen-pal.so.0 00002AD79259EC41 Unknown Unknown Unknown
libopen-pal.so.0 00002AD79259B4E1 Unknown Unknown Unknown
libopen-pal.so.0 00002AD79259B484 Unknown Unknown Unknown
libopen-pal.so.0 00002AD792595AC7 Unknown Unknown Unknown
libmpi.so.0 00002AD79207CC29 Unknown Unknown Unknown
libmpi.so.0 00002AD7920B0D21 Unknown Unknown Unknown
libmpi_f77.so.0 00002AD791E301B9 Unknown Unknown Unknown
pmemd.MPI 0000000000493E28 Unknown Unknown Unknown
pmemd.MPI 0000000000491FD3 Unknown Unknown Unknown
pmemd.MPI 000000000048EC3D Unknown Unknown Unknown
pmemd.MPI 00000000004814BA Unknown Unknown Unknown
pmemd.MPI 000000000046ACAF Unknown Unknown Unknown
pmemd.MPI 000000000041AE3A Unknown Unknown Unknown
pmemd.MPI 000000000041A55C Unknown Unknown Unknown
libc.so.6 000000359AC1D994 Unknown Unknown Unknown
pmemd.MPI 000000000041A469 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libc.so.6 00000035BE4306D0 Unknown Unknown Unknown
libopen-pal.so.0 00002B1E73B4E442 Unknown Unknown Unknown
libopen-pal.so.0 00002B1E73B4EC61 Unknown Unknown Unknown
libopen-pal.so.0 00002B1E73B4B5FC Unknown Unknown Unknown
libopen-pal.so.0 00002B1E73B4B484 Unknown Unknown Unknown
libopen-pal.so.0 00002B1E73B45AC7 Unknown Unknown Unknown
libmpi.so.0 00002B1E7362CC29 Unknown Unknown Unknown
libmpi.so.0 00002B1E73660D21 Unknown Unknown Unknown
libmpi_f77.so.0 00002B1E733E01B9 Unknown Unknown Unknown
pmemd.MPI 00000000004AEA6F Unknown Unknown Unknown
pmemd.MPI 000000000047FEF5 Unknown Unknown Unknown
pmemd.MPI 000000000046ACAF Unknown Unknown Unknown
pmemd.MPI 000000000041AE3A Unknown Unknown Unknown
pmemd.MPI 000000000041A55C Unknown Unknown Unknown
libc.so.6 00000035BE41D994 Unknown Unknown Unknown
pmemd.MPI 000000000041A469 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libc.so.6 00000035AA2306D0 Unknown Unknown Unknown
libopen-pal.so.0 00002B9267881442 Unknown Unknown Unknown
libopen-pal.so.0 00002B9267881C61 Unknown Unknown Unknown
libopen-pal.so.0 00002B926787E5FC Unknown Unknown Unknown
libopen-pal.so.0 00002B926787E484 Unknown Unknown Unknown
libopen-pal.so.0 00002B9267878AC7 Unknown Unknown Unknown
libmpi.so.0 00002B926735FC29 Unknown Unknown Unknown
libmpi.so.0 00002B9267393D21 Unknown Unknown Unknown
libmpi_f77.so.0 00002B92671131B9 Unknown Unknown Unknown
pmemd.MPI 0000000000493E28 Unknown Unknown Unknown
pmemd.MPI 0000000000491FD3 Unknown Unknown Unknown
pmemd.MPI 000000000048EC3D Unknown Unknown Unknown
pmemd.MPI 00000000004814BA Unknown Unknown Unknown
pmemd.MPI 000000000046ACAF Unknown Unknown Unknown
pmemd.MPI 000000000041AE3A Unknown Unknown Unknown
pmemd.MPI 000000000041A55C Unknown Unknown Unknown
libc.so.6 00000035AA21D994 Unknown Unknown Unknown
pmemd.MPI 000000000041A469 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libpthread.so.0 00000034DFC0EC12 Unknown Unknown Unknown
libopen-pal.so.0 00002B87AA3383F4 Unknown Unknown Unknown
libopen-pal.so.0 00002B87AA338CB6 Unknown Unknown Unknown
libopen-pal.so.0 00002B87AA3355FC Unknown Unknown Unknown
libopen-pal.so.0 00002B87AA335484 Unknown Unknown Unknown
libopen-pal.so.0 00002B87AA32FAC7 Unknown Unknown Unknown
libmpi.so.0 00002B87A9E16C29 Unknown Unknown Unknown
libmpi.so.0 00002B87A9E4AD21 Unknown Unknown Unknown
libmpi_f77.so.0 00002B87A9BCA1B9 Unknown Unknown Unknown
pmemd.MPI 00000000004AEA6F Unknown Unknown Unknown
pmemd.MPI 000000000047FEF5 Unknown Unknown Unknown
pmemd.MPI 000000000046ACAF Unknown Unknown Unknown
pmemd.MPI 000000000041AE3A Unknown Unknown Unknown
pmemd.MPI 000000000041A55C Unknown Unknown Unknown
libc.so.6 00000034DF01D994 Unknown Unknown Unknown
pmemd.MPI 000000000041A469 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libpthread.so.0 000000313880EA32 Unknown Unknown Unknown
libopen-pal.so.0 00002ABE76C0C3F4 Unknown Unknown Unknown
libopen-pal.so.0 00002ABE76C0CC41 Unknown Unknown Unknown
libopen-pal.so.0 00002ABE76C094E1 Unknown Unknown Unknown
libopen-pal.so.0 00002ABE76C09484 Unknown Unknown Unknown
libopen-pal.so.0 00002ABE76C03AC7 Unknown Unknown Unknown
libmpi.so.0 00002ABE766EAC29 Unknown Unknown Unknown
libmpi.so.0 00002ABE7671ED21 Unknown Unknown Unknown
libmpi_f77.so.0 00002ABE7649E1B9 Unknown Unknown Unknown
pmemd.MPI 00000000004AEA6F Unknown Unknown Unknown
pmemd.MPI 000000000047FEF5 Unknown Unknown Unknown
pmemd.MPI 000000000046ACAF Unknown Unknown Unknown
pmemd.MPI 000000000041AE3A Unknown Unknown Unknown
pmemd.MPI 000000000041A55C Unknown Unknown Unknown
libc.so.6 0000003137C1D994 Unknown Unknown Unknown
pmemd.MPI 000000000041A469 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libc.so.6 000000314A4306D0 Unknown Unknown Unknown
libopen-pal.so.0 00002AE698C592DD Unknown Unknown Unknown
libopen-pal.so.0 00002AE698C59CB6 Unknown Unknown Unknown
libopen-pal.so.0 00002AE698C565FC Unknown Unknown Unknown
libopen-pal.so.0 00002AE698C56484 Unknown Unknown Unknown
libopen-pal.so.0 00002AE698C50AC7 Unknown Unknown Unknown
libmpi.so.0 00002AE698737C29 Unknown Unknown Unknown
libmpi.so.0 00002AE69876BD21 Unknown Unknown Unknown
libmpi_f77.so.0 00002AE6984EB1B9 Unknown Unknown Unknown
pmemd.MPI 0000000000493E28 Unknown Unknown Unknown
pmemd.MPI 0000000000491FD3 Unknown Unknown Unknown
pmemd.MPI 000000000048EC3D Unknown Unknown Unknown
pmemd.MPI 00000000004814BA Unknown Unknown Unknown
pmemd.MPI 000000000046ACAF Unknown Unknown Unknown
pmemd.MPI 000000000041AE3A Unknown Unknown Unknown
pmemd.MPI 000000000041A55C Unknown Unknown Unknown
libc.so.6 000000314A41D994 Unknown Unknown Unknown
pmemd.MPI 000000000041A469 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libc.so.6 000000381A8306D0 Unknown Unknown Unknown
libopen-pal.so.0 00002B8D746632DD Unknown Unknown Unknown
libopen-pal.so.0 00002B8D74663CB6 Unknown Unknown Unknown
libopen-pal.so.0 00002B8D746605FC Unknown Unknown Unknown
libopen-pal.so.0 00002B8D74660484 Unknown Unknown Unknown
libopen-pal.so.0 00002B8D7465AAC7 Unknown Unknown Unknown
libmpi.so.0 00002B8D74141C29 Unknown Unknown Unknown
libmpi.so.0 00002B8D74175D21 Unknown Unknown Unknown
libmpi_f77.so.0 00002B8D73EF51B9 Unknown Unknown Unknown
pmemd.MPI 00000000004AEA6F Unknown Unknown Unknown
pmemd.MPI 000000000047FEF5 Unknown Unknown Unknown
pmemd.MPI 000000000046ACAF Unknown Unknown Unknown
pmemd.MPI 000000000041AE3A Unknown Unknown Unknown
pmemd.MPI 000000000041A55C Unknown Unknown Unknown
libc.so.6 000000381A81D994 Unknown Unknown Unknown
pmemd.MPI 000000000041A469 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libc.so.6 0000003EBF2306D0 Unknown Unknown Unknown
libopen-pal.so.0 00002AD67CE5D2DD Unknown Unknown Unknown
libopen-pal.so.0 00002AD67CE5DCB6 Unknown Unknown Unknown
libopen-pal.so.0 00002AD67CE5A5FC Unknown Unknown Unknown
libopen-pal.so.0 00002AD67CE5A484 Unknown Unknown Unknown
libopen-pal.so.0 00002AD67CE54AC7 Unknown Unknown Unknown
libmpi.so.0 00002AD67C93BC29 Unknown Unknown Unknown
libmpi.so.0 00002AD67C96FD21 Unknown Unknown Unknown
libmpi_f77.so.0 00002AD67C6EF1B9 Unknown Unknown Unknown
pmemd.MPI 00000000004AEA6F Unknown Unknown Unknown
pmemd.MPI 000000000047FEF5 Unknown Unknown Unknown
pmemd.MPI 000000000046ACAF Unknown Unknown Unknown
pmemd.MPI 000000000041AE3A Unknown Unknown Unknown
pmemd.MPI 000000000041A55C Unknown Unknown Unknown
libc.so.6 0000003EBF21D994 Unknown Unknown Unknown
pmemd.MPI 000000000041A469 Unknown Unknown Unknown
mpirun noticed that job rank 1 with PID 4485 on node imp200.open.ac.uk exited on
signal 11 (Segmentation fault).
4 additional processes aborted (not shown)
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Mon Aug 30 2010 - 09:00:05 PDT
