MPIRUN: MPI progress Quiescence Detected. MPIRUN: 48 out of 64 ranks showed no MPI send or receive progress in 900 seconds. MPIRUN: 0 out of 64 ranks could not return a ping request. MPIRUN: Per-rank details are the following: MPIRUN: Rank 0 (node020 ) caused MPI progress Quiescence. MPIRUN: Rank 1 (node020 ) caused MPI progress Quiescence. MPIRUN: Rank 2 (node020 ) caused MPI progress Quiescence. MPIRUN: Rank 3 (node020 ) caused MPI progress Quiescence. MPIRUN: Rank 4 (node018 ) caused MPI progress Quiescence. MPIRUN: Rank 5 (node018 ) caused MPI progress Quiescence. MPIRUN: Rank 6 (node018 ) caused MPI progress Quiescence. MPIRUN: Rank 7 (node018 ) caused MPI progress Quiescence. MPIRUN: Rank 8 (node017 ) caused MPI progress Quiescence. MPIRUN: Rank 9 (node017 ) caused MPI progress Quiescence. MPIRUN: Rank 10 (node017 ) caused MPI progress Quiescence. MPIRUN: Rank 11 (node017 ) caused MPI progress Quiescence. MPIRUN: Rank 12 (node016 ) caused MPI progress Quiescence. MPIRUN: Rank 13 (node016 ) caused MPI progress Quiescence. MPIRUN: Rank 14 (node016 ) caused MPI progress Quiescence. MPIRUN: Rank 15 (node016 ) caused MPI progress Quiescence. MPIRUN: Rank 16 (node015 ) caused MPI progress Quiescence. MPIRUN: Rank 17 (node015 ) caused MPI progress Quiescence. MPIRUN: Rank 18 (node015 ) caused MPI progress Quiescence. MPIRUN: Rank 19 (node015 ) caused MPI progress Quiescence. MPIRUN: Rank 20 (node014 ) caused MPI progress Quiescence. MPIRUN: Rank 21 (node014 ) caused MPI progress Quiescence. MPIRUN: Rank 22 (node014 ) caused MPI progress Quiescence. MPIRUN: Rank 23 (node014 ) caused MPI progress Quiescence. MPIRUN: Rank 24 (node013 ) caused MPI progress Quiescence. MPIRUN: Rank 25 (node013 ) caused MPI progress Quiescence. MPIRUN: Rank 26 (node013 ) caused MPI progress Quiescence. MPIRUN: Rank 27 (node013 ) caused MPI progress Quiescence. MPIRUN: Rank 28 (node012 ) caused MPI progress Quiescence. MPIRUN: Rank 29 (node012 ) caused MPI progress Quiescence. MPIRUN: Rank 30 (node012 ) caused MPI progress Quiescence. MPIRUN: Rank 31 (node012 ) caused MPI progress Quiescence. MPIRUN: Rank 32 (node011 ) caused MPI progress Quiescence. MPIRUN: Rank 33 (node011 ) caused MPI progress Quiescence. MPIRUN: Rank 34 (node011 ) caused MPI progress Quiescence. MPIRUN: Rank 35 (node011 ) caused MPI progress Quiescence. MPIRUN: Rank 36 (node010 ) caused MPI progress Quiescence. MPIRUN: Rank 37 (node010 ) caused MPI progress Quiescence. MPIRUN: Rank 38 (node010 ) caused MPI progress Quiescence. MPIRUN: Rank 39 (node010 ) caused MPI progress Quiescence. MPIRUN: Rank 40 (node009 ) caused MPI progress Quiescence. MPIRUN: Rank 41 (node009 ) caused MPI progress Quiescence. MPIRUN: Rank 42 (node009 ) caused MPI progress Quiescence. MPIRUN: Rank 43 (node009 ) caused MPI progress Quiescence. MPIRUN: Rank 44 (node008 ) caused MPI progress Quiescence. MPIRUN: Rank 45 (node008 ) caused MPI progress Quiescence. MPIRUN: Rank 46 (node008 ) caused MPI progress Quiescence. MPIRUN: Rank 47 (node008 ) caused MPI progress Quiescence. node017:9.Quiescence detected. Message Queues: Group = 47 WorldRank = 47 NodeID = 0 node008:47.SHM Free control buffers: 2048 node008:47.Quiescence detected. Message Queues: node009:43.Quiescence detected. Message Queues: node011:34.Quiescence detected. Message Queues: node012:28.Quiescence detected. Message Queues: node016:14.Quiescence detected. Message Queues: Group = 53 WorldRank = 53 NodeID = 0 node006:53.SHM Free control buffers: 2048 node006:53.Quiescence detected. Message Queues: node014:23.Quiescence detected. Message Queues: Group = 4 WorldRank = 4 NodeID = 0 node018:4.SHM Free control buffers: 2048 node018:4.Quiescence detected. Message Queues: node007:49.Quiescence detected. Message Queues: Group = 61 WorldRank = 61 NodeID = 0 node004:61.SHM Free control buffers: 2048 node004:61.Quiescence detected. Message Queues: Group = 11 WorldRank = 11 NodeID = 0 node017:11.SHM Free control buffers: 2048 node017:11.Quiescence detected. Message Queues: node020:2.Quiescence detected. Message Queues: Group = 27 WorldRank = 27 NodeID = 0 node013:27.SHM Free control buffers: 2048 node013:27.Quiescence detected. Message Queues: Group = 18 WorldRank = 18 NodeID = 0 node015:18.SHM Free control buffers: 2048 node015:18.Quiescence detected. Message Queues: Group = 17 WorldRank = 17 NodeID = 0 node015:17.SHM Free control buffers: 2048 node015:17.Quiescence detected. Message Queues: node016:15.Quiescence detected. Message Queues: node009:42.Quiescence detected. Message Queues: node020:3.Quiescence detected. Message Queues: node016:12.Quiescence detected. Message Queues: Group = 24 WorldRank = 24 NodeID = 0 node013:24.SHM Free control buffers: 2048 node013:24.Quiescence detected. Message Queues: Group = 30 WorldRank = 30 NodeID = 0 node012:30.SHM Free control buffers: 2048 node012:30.Quiescence detected. Message Queues: Group = 32 WorldRank = 32 NodeID = 0 node011:32.SHM Free control buffers: 2048 node011:32.Quiescence detected. Message Queues: node018:5.Quiescence detected. Message Queues: Group = 29 WorldRank = 29 NodeID = 0 node012:29.SHM Free control buffers: 2048 node012:29.Quiescence detected. Message Queues: node011:35.Quiescence detected. Message Queues: node014:22.Quiescence detected. Message Queues: node008:45.Quiescence detected. Message Queues: Group = 52 WorldRank = 52 NodeID = 0 node006:52.SHM Free control buffers: 2048 node006:52.Quiescence detected. Message Queues: Group = 33 WorldRank = 33 NodeID = 0 node011:33.SHM Free control buffers: 2048 node011:33.Quiescence detected. Message Queues: node010:36.Quiescence detected. Message Queues: Group = 38 WorldRank = 38 NodeID = 0 node010:38.SHM Free control buffers: 2048 node010:38.Quiescence detected. Message Queues: Group = 7 WorldRank = 7 NodeID = 0 node018:7.SHM Free control buffers: 2048 node018:7.Quiescence detected. Message Queues: Group = 41 WorldRank = 41 NodeID = 0 node009:41.SHM Free control buffers: 2048 node009:41.Quiescence detected. Message Queues: node006:55.Quiescence detected. Message Queues: Group = 31 WorldRank = 31 NodeID = 0 node012:31.SHM Free control buffers: 2048 node012:31.Quiescence detected. Message Queues: Group = 54 WorldRank = 54 NodeID = 0 node006:54.SHM Free control buffers: 2048 node006:54.Quiescence detected. Message Queues: Group = 46 WorldRank = 46 NodeID = 0 node008:46.SHM Free control buffers: 2048 node008:46.Quiescence detected. Message Queues: node017:10.Quiescence detected. Message Queues: Group = 48 WorldRank = 48 NodeID = 0 node007:48.SHM Free control buffers: 2048 node007:48.Quiescence detected. Message Queues: Group = 37 WorldRank = 37 NodeID = 0 node010:37.SHM Free control buffers: 2048 node010:37.Quiescence detected. Message Queues: node010:39.Quiescence detected. Message Queues: Group = 40 WorldRank = 40 NodeID = 0 node009:40.SHM Free control buffers: 2048 node009:40.Quiescence detected. Message Queues: Group = 6 WorldRank = 6 NodeID = 0 node018:6.SHM Free control buffers: 2048 node018:6.Quiescence detected. Message Queues: node004:62.Quiescence detected. Message Queues: Group = 59 WorldRank = 59 NodeID = 0 node005:59.SHM Free control buffers: 2048 node005:59.Quiescence detected. Message Queues: Group = 51 WorldRank = 51 NodeID = 0 node007:51.SHM Free control buffers: 2048 node007:51.Quiescence detected. Message Queues: Group = 44 WorldRank = 44 NodeID = 0 node008:44.SHM Free control buffers: 2048 node008:44.Quiescence detected. Message Queues: node004:63.Quiescence detected. Message Queues: Group = 57 WorldRank = 57 NodeID = 0 node005:57.SHM Free control buffers: 2048 node005:57.Quiescence detected. Message Queues: Group = 56 WorldRank = 56 NodeID = 0 node005:56.SHM Free control buffers: 2048 node005:56.Quiescence detected. Message Queues: Group = 58 WorldRank = 58 NodeID = 0 node005:58.SHM Free control buffers: 2048 node005:58.Quiescence detected. Message Queues: Group = 60 WorldRank = 60 NodeID = 0 node004:60.SHM Free control buffers: 2048 node004:60.Quiescence detected. Message Queues: Group = 8 WorldRank = 8 NodeID = 0 node017:8.SHM Free control buffers: 2048 node017:8.Quiescence detected. Message Queues: Group = 21 WorldRank = 21 NodeID = 0 node014:21.SHM Free control buffers: 2048 node014:21.Quiescence detected. Message Queues: node014:20.Quiescence detected. Message Queues: Group = 26 WorldRank = 26 NodeID = 0 node013:26.SHM Free control buffers: 2048 node013:26.Quiescence detected. Message Queues: Group = 25 WorldRank = 25 NodeID = 0 node013:25.SHM Free control buffers: 2048 node013:25.Quiescence detected. Message Queues: Group = 16 WorldRank = 16 NodeID = 0 node015:16.SHM Free control buffers: 2048 node015:16.Quiescence detected. Message Queues: Group = 19 WorldRank = 19 NodeID = 0 node015:19.SHM Free control buffers: 2048 node015:19.Quiescence detected. Message Queues: Group = 13 WorldRank = 13 NodeID = 0 node016:13.SHM Free control buffers: 2048 node016:13.Quiescence detected. Message Queues: Running multisander version of sander amber9 Total processors = 64 Number of groups = 64 Looping over processors: WorldRank is the global PE rank NodeID is the local PE rank in current group Group = 0 WorldRank = 0 NodeID = 0 node020:0.SHM Free control buffers: 2048 node020:0.Quiescence detected. Message Queues: Group = 50 WorldRank = 50 NodeID = 0 node007:50.SHM Free control buffers: 2048 node007:50.Quiescence detected. Message Queues: Group = 1 WorldRank = 1 NodeID = 0 node020:1.SHM Free control buffers: 2048 node020:1.Quiescence detected. Message Queues: Group = 2 WorldRank = 2 NodeID = 0 node020:2.SHM Free control buffers: 2048 Group = 15 WorldRank = 15 NodeID = 0 node016:15.SHM Free control buffers: 2048 Group = 12 WorldRank = 12 NodeID = 0 node016:12.SHM Free control buffers: 2048 Group = 3 WorldRank = 3 NodeID = 0 node020:3.SHM Free control buffers: 2048 Group = 49 WorldRank = 49 NodeID = 0 node007:49.SHM Free control buffers: 2048 Group = 28 WorldRank = 28 NodeID = 0 node012:28.SHM Free control buffers: 2048 Group = 34 WorldRank = 34 NodeID = 0 node011:34.SHM Free control buffers: 2048 Group = 5 WorldRank = 5 NodeID = 0 node018:5.SHM Free control buffers: 2048 Group = 9 WorldRank = 9 NodeID = 0 node017:9.SHM Free control buffers: 2048 Group = 14 WorldRank = 14 NodeID = 0 node016:14.SHM Free control buffers: 2048 Group = 22 WorldRank = 22 NodeID = 0 node014:22.SHM Free control buffers: 2048 Group = 43 WorldRank = 43 NodeID = 0 node009:43.SHM Free control buffers: 2048 Group = 42 WorldRank = 42 NodeID = 0 node009:42.SHM Free control buffers: 2048 Group = 23 WorldRank = 23 NodeID = 0 node014:23.SHM Free control buffers: 2048 Group = 45 WorldRank = 45 NodeID = 0 node008:45.SHM Free control buffers: 2048 Group = 35 WorldRank = 35 NodeID = 0 node011:35.SHM Free control buffers: 2048 Group = 36 WorldRank = 36 NodeID = 0 node010:36.SHM Free control buffers: 2048 Group = 10 WorldRank = 10 NodeID = 0 node017:10.SHM Free control buffers: 2048 Group = 55 WorldRank = 55 NodeID = 0 node006:55.SHM Free control buffers: 2048 Group = 20 WorldRank = 20 NodeID = 0 node014:20.SHM Free control buffers: 2048 Group = 39 WorldRank = 39 NodeID = 0 node010:39.SHM Free control buffers: 2048 Group = 63 WorldRank = 63 NodeID = 0 node004:63.SHM Free control buffers: 2048 Group = 62 WorldRank = 62 NodeID = 0 node004:62.SHM Free control buffers: 2048