/opt/gridware/ge/default/spool/int12/job_scripts/202717: line 6: : command not found -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 1. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source libc.so.6 00000034676E62A3 Unknown Unknown Unknown libmpi.so.1 00002B67F1052121 Unknown Unknown Unknown libmpi.so.1 00002B67F1052FA4 Unknown Unknown Unknown libmpi.so.1 00002B67F1052E42 Unknown Unknown Unknown libmpi.so.1 00002B67F108320C Unknown Unknown Unknown libmpi.so.1 00002B67F0FB3998 Unknown Unknown Unknown mca_coll_tuned.so 00002B67F5E7989A Unknown Unknown Unknown mca_coll_tuned.so 00002B67F5E792C2 Unknown Unknown Unknown mca_coll_tuned.so 00002B67F5E70607 Unknown Unknown Unknown mca_coll_sync.so 00002B67F5C6761F Unknown Unknown Unknown libmpi.so.1 00002B67F0FC0501 Unknown Unknown Unknown libmpi_f77.so.1 00002B67F0D37E6A Unknown Unknown Unknown sander.MPI 00000000005338B7 Unknown Unknown Unknown sander.MPI 000000000046215C Unknown Unknown Unknown libc.so.6 000000346761ECDD Unknown Unknown Unknown sander.MPI 0000000000462059 Unknown Unknown Unknown forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source libc.so.6 00000034676E62A3 Unknown Unknown Unknown libmpi.so.1 00002AB77B22A121 Unknown Unknown Unknown libmpi.so.1 00002AB77B22AFA4 Unknown Unknown Unknown libmpi.so.1 00002AB77B22AE42 Unknown Unknown Unknown libmpi.so.1 00002AB77B25B20C Unknown Unknown Unknown libmpi.so.1 00002AB77B18B998 Unknown Unknown Unknown mca_coll_tuned.so 00002AB78005189A Unknown Unknown Unknown mca_coll_tuned.so 00002AB7800512C2 Unknown Unknown Unknown mca_coll_tuned.so 00002AB780048607 Unknown Unknown Unknown mca_coll_sync.so 00002AB77FE3F61F Unknown Unknown Unknown libmpi.so.1 00002AB77B198501 Unknown Unknown Unknown libmpi_f77.so.1 00002AB77AF0FE6A Unknown Unknown Unknown sander.MPI 00000000005338B7 Unknown Unknown Unknown sander.MPI 000000000046215C Unknown Unknown Unknown libc.so.6 000000346761ECDD Unknown Unknown Unknown sander.MPI 0000000000462059 Unknown Unknown Unknown forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source libc.so.6 00000034676E62A3 Unknown Unknown Unknown libmpi.so.1 00002B7FD94D8121 Unknown Unknown Unknown libmpi.so.1 00002B7FD94D8FA4 Unknown Unknown Unknown libmpi.so.1 00002B7FD94D8E42 Unknown Unknown Unknown libmpi.so.1 00002B7FD950920C Unknown Unknown Unknown libmpi.so.1 00002B7FD9439998 Unknown Unknown Unknown mca_coll_tuned.so 00002B7FDE2FF89A Unknown Unknown Unknown mca_coll_tuned.so 00002B7FDE2FF2C2 Unknown Unknown Unknown mca_coll_tuned.so 00002B7FDE2F6607 Unknown Unknown Unknown mca_coll_sync.so 00002B7FDE0ED61F Unknown Unknown Unknown libmpi.so.1 00002B7FD9446501 Unknown Unknown Unknown libmpi_f77.so.1 00002B7FD91BDE6A Unknown Unknown Unknown sander.MPI 00000000005338B7 Unknown Unknown Unknown sander.MPI 000000000046215C Unknown Unknown Unknown libc.so.6 000000346761ECDD Unknown Unknown Unknown sander.MPI 0000000000462059 Unknown Unknown Unknown forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source libc.so.6 00000034676E62A3 Unknown Unknown Unknown libmpi.so.1 00002B6C215A9121 Unknown Unknown Unknown libmpi.so.1 00002B6C215A9FA4 Unknown Unknown Unknown libmpi.so.1 00002B6C215A9E42 Unknown Unknown Unknown libmpi.so.1 00002B6C215DA20C Unknown Unknown Unknown libmpi.so.1 00002B6C2150A998 Unknown Unknown Unknown mca_coll_tuned.so 00002B6C263D089A Unknown Unknown Unknown mca_coll_tuned.so 00002B6C263D02C2 Unknown Unknown Unknown mca_coll_tuned.so 00002B6C263C7607 Unknown Unknown Unknown mca_coll_sync.so 00002B6C261BE61F Unknown Unknown Unknown libmpi.so.1 00002B6C21517501 Unknown Unknown Unknown libmpi_f77.so.1 00002B6C2128EE6A Unknown Unknown Unknown sander.MPI 00000000005338B7 Unknown Unknown Unknown sander.MPI 000000000046215C Unknown Unknown Unknown libc.so.6 000000346761ECDD Unknown Unknown Unknown sander.MPI 0000000000462059 Unknown Unknown Unknown forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source libc.so.6 00000034676E62A3 Unknown Unknown Unknown libmpi.so.1 00002B157461C121 Unknown Unknown Unknown libmpi.so.1 00002B157461CFA4 Unknown Unknown Unknown libmpi.so.1 00002B157461CE42 Unknown Unknown Unknown libmpi.so.1 00002B157464D20C Unknown Unknown Unknown libmpi.so.1 00002B157457D998 Unknown Unknown Unknown mca_coll_tuned.so 00002B157944389A Unknown Unknown Unknown mca_coll_tuned.so 00002B15794432C2 Unknown Unknown Unknown mca_coll_tuned.so 00002B157943A607 Unknown Unknown Unknown mca_coll_sync.so 00002B157923161F Unknown Unknown Unknown libmpi.so.1 00002B157458A501 Unknown Unknown Unknown libmpi_f77.so.1 00002B1574301E6A Unknown Unknown Unknown sander.MPI 00000000005338B7 Unknown Unknown Unknown sander.MPI 000000000046215C Unknown Unknown Unknown libc.so.6 000000346761ECDD Unknown Unknown Unknown sander.MPI 0000000000462059 Unknown Unknown Unknown forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source libc.so.6 00000034676E62A3 Unknown Unknown Unknown libmpi.so.1 00002B94A8E3A121 Unknown Unknown Unknown libmpi.so.1 00002B94A8E3AFA4 Unknown Unknown Unknown libmpi.so.1 00002B94A8E3AE42 Unknown Unknown Unknown libmpi.so.1 00002B94A8E6B20C Unknown Unknown Unknown libmpi.so.1 00002B94A8D9B998 Unknown Unknown Unknown mca_coll_tuned.so 00002B94ADC6169A Unknown Unknown Unknown mca_coll_tuned.so 00002B94ADC612C2 Unknown Unknown Unknown mca_coll_tuned.so 00002B94ADC58607 Unknown Unknown Unknown mca_coll_sync.so 00002B94ADA4F61F Unknown Unknown Unknown libmpi.so.1 00002B94A8DA8501 Unknown Unknown Unknown libmpi_f77.so.1 00002B94A8B1FE6A Unknown Unknown Unknown sander.MPI 00000000005338B7 Unknown Unknown Unknown sander.MPI 000000000046215C Unknown Unknown Unknown libc.so.6 000000346761ECDD Unknown Unknown Unknown sander.MPI 0000000000462059 Unknown Unknown Unknown forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source libc.so.6 00000034676E62A3 Unknown Unknown Unknown libmpi.so.1 00002B09B8A10121 Unknown Unknown Unknown libmpi.so.1 00002B09B8A10FA4 Unknown Unknown Unknown libmpi.so.1 00002B09B8A10E42 Unknown Unknown Unknown libmpi.so.1 00002B09B8A4120C Unknown Unknown Unknown libmpi.so.1 00002B09B8971998 Unknown Unknown Unknown mca_coll_tuned.so 00002B09BD83769A Unknown Unknown Unknown mca_coll_tuned.so 00002B09BD8372C2 Unknown Unknown Unknown mca_coll_tuned.so 00002B09BD82E607 Unknown Unknown Unknown mca_coll_sync.so 00002B09BD62561F Unknown Unknown Unknown libmpi.so.1 00002B09B897E501 Unknown Unknown Unknown libmpi_f77.so.1 00002B09B86F5E6A Unknown Unknown Unknown sander.MPI 00000000005338B7 Unknown Unknown Unknown sander.MPI 000000000046215C Unknown Unknown Unknown libc.so.6 000000346761ECDD Unknown Unknown Unknown sander.MPI 0000000000462059 Unknown Unknown Unknown forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source libc.so.6 00000034676E62A3 Unknown Unknown Unknown libmpi.so.1 00002AFE7EE6B121 Unknown Unknown Unknown libmpi.so.1 00002AFE7EE6BFA4 Unknown Unknown Unknown libmpi.so.1 00002AFE7EE6BE42 Unknown Unknown Unknown libmpi.so.1 00002AFE7EE9C20C Unknown Unknown Unknown libmpi.so.1 00002AFE7EDCC998 Unknown Unknown Unknown mca_coll_tuned.so 00002AFE83C9269A Unknown Unknown Unknown mca_coll_tuned.so 00002AFE83C922C2 Unknown Unknown Unknown mca_coll_tuned.so 00002AFE83C89607 Unknown Unknown Unknown mca_coll_sync.so 00002AFE83A8061F Unknown Unknown Unknown libmpi.so.1 00002AFE7EDD9501 Unknown Unknown Unknown libmpi_f77.so.1 00002AFE7EB50E6A Unknown Unknown Unknown sander.MPI 00000000005338B7 Unknown Unknown Unknown sander.MPI 000000000046215C Unknown Unknown Unknown libc.so.6 000000346761ECDD Unknown Unknown Unknown sander.MPI 0000000000462059 Unknown Unknown Unknown forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source libc.so.6 00000034676E62A3 Unknown Unknown Unknown libmpi.so.1 00002AAAD070E121 Unknown Unknown Unknown libmpi.so.1 00002AAAD070EFA4 Unknown Unknown Unknown libmpi.so.1 00002AAAD070EE42 Unknown Unknown Unknown libmpi.so.1 00002AAAD073F20C Unknown Unknown Unknown libmpi.so.1 00002AAAD066F998 Unknown Unknown Unknown mca_coll_tuned.so 00002AAAD553589A Unknown Unknown Unknown mca_coll_tuned.so 00002AAAD55352C2 Unknown Unknown Unknown mca_coll_tuned.so 00002AAAD552C607 Unknown Unknown Unknown mca_coll_sync.so 00002AAAD532361F Unknown Unknown Unknown libmpi.so.1 00002AAAD067C501 Unknown Unknown Unknown libmpi_f77.so.1 00002AAAD03F3E6A Unknown Unknown Unknown sander.MPI 00000000005338B7 Unknown Unknown Unknown sander.MPI 000000000046215C Unknown Unknown Unknown libc.so.6 000000346761ECDD Unknown Unknown Unknown sander.MPI 0000000000462059 Unknown Unknown Unknown forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source libc.so.6 00000034676E62A3 Unknown Unknown Unknown libmpi.so.1 00002B3D5E1FD121 Unknown Unknown Unknown libmpi.so.1 00002B3D5E1FDFA4 Unknown Unknown Unknown libmpi.so.1 00002B3D5E1FDE42 Unknown Unknown Unknown libmpi.so.1 00002B3D5E22E20C Unknown Unknown Unknown libmpi.so.1 00002B3D5E15E998 Unknown Unknown Unknown mca_coll_tuned.so 00002B3D6302489A Unknown Unknown Unknown mca_coll_tuned.so 00002B3D630242C2 Unknown Unknown Unknown mca_coll_tuned.so 00002B3D6301B607 Unknown Unknown Unknown mca_coll_sync.so 00002B3D62E1261F Unknown Unknown Unknown libmpi.so.1 00002B3D5E16B501 Unknown Unknown Unknown libmpi_f77.so.1 00002B3D5DEE2E6A Unknown Unknown Unknown sander.MPI 00000000005338B7 Unknown Unknown Unknown sander.MPI 000000000046215C Unknown Unknown Unknown libc.so.6 000000346761ECDD Unknown Unknown Unknown sander.MPI 0000000000462059 Unknown Unknown Unknown forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source libc.so.6 00000034676E62A3 Unknown Unknown Unknown libmpi.so.1 00002AD4B194A121 Unknown Unknown Unknown libmpi.so.1 00002AD4B194AFA4 Unknown Unknown Unknown libmpi.so.1 00002AD4B194AE42 Unknown Unknown Unknown libmpi.so.1 00002AD4B197B20C Unknown Unknown Unknown libmpi.so.1 00002AD4B18AB998 Unknown Unknown Unknown mca_coll_tuned.so 00002AD4B677189A Unknown Unknown Unknown mca_coll_tuned.so 00002AD4B67712C2 Unknown Unknown Unknown mca_coll_tuned.so 00002AD4B6768607 Unknown Unknown Unknown mca_coll_sync.so 00002AD4B655F61F Unknown Unknown Unknown libmpi.so.1 00002AD4B18B8501 Unknown Unknown Unknown libmpi_f77.so.1 00002AD4B162FE6A Unknown Unknown Unknown sander.MPI 00000000005338B7 Unknown Unknown Unknown sander.MPI 000000000046215C Unknown Unknown Unknown libc.so.6 000000346761ECDD Unknown Unknown Unknown sander.MPI 0000000000462059 Unknown Unknown Unknown -------------------------------------------------------------------------- mpiexec has exited due to process rank 0 with PID 26629 on node int12 exiting improperly. There are two reasons this could occur: 1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination. 2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination" This may have caused other processes in the application to be terminated by signals sent by mpiexec (as reported here). -------------------------------------------------------------------------- /opt/gridware/ge/default/spool/int12/job_scripts/202717: line 8: -i: command not found /opt/gridware/ge/default/spool/int12/job_scripts/202717: line 9: -o: command not found /opt/gridware/ge/default/spool/int12/job_scripts/202717: line 10: -p: command not found /opt/gridware/ge/default/spool/int12/job_scripts/202717: line 11: -c: command not found /opt/gridware/ge/default/spool/int12/job_scripts/202717: line 12: -r: command not found /opt/gridware/ge/default/spool/int12/job_scripts/202717: line 13: -inf: command not found /opt/gridware/ge/default/spool/int12/job_scripts/202717: line 14: : command not found /opt/gridware/ge/default/spool/int12/job_scripts/202717: line 15: : command not found