Dear Amber users,

I'm tring to Amber BMT (on Amber's website) with 2 node 4 GPGPU (2 GPGPU in a node).
When I test "Factor IX NPT", "Factor IX NVE", "JAC NPT" and "JAC NVE", often (about 70 %) the test doesn't finish conpletely.
Processes are working but it seems that GPUs don't work anymore and there are no error logs.

However, there is no problem with other test (TRPCage, nucleosome, myoglobin, Cellulose NPT and NVE")

Could you give me an advice ?

Following is system configuration and logs.
To be safe, I attach log files on this E-mail.


### Configuration ###

- CPU: E5-2680
- GPU: K20 or K20X

- OS: RHEL6.1
- Amber 12 (patched bugfix 1 to 18)
- AmberTool 13 (patched bugfix 1 to 15)
- Intel Compiler: 2013.1.117
- MVAPICH2: 1.9
- GPU Device dirver: 304.64 and 319.32 (latest version)
- CUDA: 5.0.35
- Interconnect: Mellanox FDR HCA card (Mellanox OFED 1.5.3-3.0.0)

** The same issue occur with the case of Intel compiler + Intel MPI.


### config.h ###

[root@RX350s7ambertest amber12]# cat config.h
#  Amber configuration file, created with: ./configure -mpi -cuda intel

###############################################################################

# (1)  Location of the installation

BASEDIR=/root/amber/amber12
BINDIR=/root/amber/amber12/bin
LIBDIR=/root/amber/amber12/lib
INCDIR=/root/amber/amber12/include
DATDIR=/root/amber/amber12/dat
LOGDIR=/root/amber/amber12/logs

###############################################################################


#  (2) If you want to search additional libraries by default, add them
#      to the FLIBS variable here.  (External libraries can also be linked into
#      NAB programs simply by including them on the command line; libraries
#      included in FLIBS are always searched.)

FLIBS=  -lsff_mpi -lpbsa   -larpack  $(BASEDIR)/lib/libnetcdf.a -Wl,--start-group /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_intel_lp64.a /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_sequential.a /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -L/opt/intel/composer_xe_2013.1.117/lib/intel64 -lifport -lifcore -lsvml
FLIBS_PTRAJ= -larpack  -Wl,--start-group /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_intel_lp64.a /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_sequential.a /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -L/opt/intel/composer_xe_2013.1.117/lib/intel64 -lifport -lifcore -lsvml
FLIBSF= -larpack  -Wl,--start-group /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_intel_lp64.a /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_sequential.a /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread  -lsvml
FLIBS_FFTW3=
###############################################################################

#  (3)  Modify any of the following if you need to change, e.g. to use gcc
#        rather than cc, etc.

SHELL=/bin/sh
INSTALLTYPE=cuda_parallel
BUILDAMBER=amber

#  Set the C compiler, etc.

#  The configure script should be fine, but if you need to hand-edit,
#  here is some info:

#   Example:  CC-->gcc; LEX-->flex; YACC-->yacc (built in byacc)
#     Note: If your lexer is "really" flex, you need to set
#     LEX=flex below.  For example, on some distributions,
#     /usr/bin/lex is really just a pointer to /usr/bin/flex,
#     so LEX=flex is necessary.  In general, gcc seems to need flex.

#   The compiler flags CFLAGS and CXXFLAGS should always be used.
#   By contrast, *OPTFLAGS and *NOOPTFLAGS will only be used with
#   certain files, and usually at compile-time but not link-time.
#   Where *OPTFLAGS and *NOOPTFLAGS are requested (in Makefiles,
#   makedepend and depend), they should come before CFLAGS or
#   CXXFLAGS; this allows the user to override *OPTFLAGS and
#   *NOOPTFLAGS using the BUILDFLAGS variable.

#   AMBERBUILDFLAGS provides a hook into all stages of the build process.
#   It can be used to build debug versions, invoke special features, etc.
#   Example:  make AMBERBUILDFLAGS='-O0 -g' sander
#
CC=mpicc
CFLAGS= -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -DBINTRAJ -DHASGZ -DHASBZ2 -DMPI  $(CUSTOMBUILDFLAGS) -I/opt/intel/composer_xe_2013.1.117/mkl/include $(AMBERBUILDFLAGS)
CNOOPTFLAGS=
COPTFLAGS=-ip -O3 -xHost
AMBERCFLAGS= $(AMBERBUILDFLAGS)

CXX=icpc
CPLUSPLUS=icpc
CXXFLAGS= -DMPI  $(CUSTOMBUILDFLAGS) $(AMBERBUILDFLAGS)
CXXNOOPTFLAGS=
CXXOPTFLAGS=-O3
AMBERCXXFLAGS= $(AMBERBUILDFLAGS)

NABFLAGS= $(AMBERBUILDFLAGS)
PBSAFLAG= $(AMBERBUILDFLAGS)

LDFLAGS=-shared-intel  $(CUSTOMBUILDFLAGS) $(AMBERBUILDFLAGS)
AMBERLDFLAGS=$(AMBERBUILDFLAGS)

LEX=   flex
YACC=  $(BINDIR)/yacc
AR=    ar rv
M4=    m4
RANLIB=ranlib

#  Set the C-preprocessor.  Code for a small preprocessor is in
#    ucpp-1.3;  it gets installed as $(BINDIR)/ucpp;

CPP=ucpp -l

#  These variables control whether we will use compiled versions of BLAS
#  and LAPACK (which are generally slower), or whether those libraries are
#  already available (presumably in an optimized form).

LAPACK=skip
BLAS=skip
F2C=skip

#  These variables determine whether builtin versions of certain components
#  can be used, or whether we need to compile our own versions.

UCPP=install
C9XCOMPLEX=skip

#  For Windows/cygwin, set SFX to ".exe"; for Unix/Linux leave it empty:
#  Set OBJSFX to ".obj" instead of ".o" on Windows:

SFX=
OSFX=.o
MV=mv
RM=rm
CP=cp

#  Information about Fortran compilation:

FC=mpif90
FFLAGS=  $(LOCALFLAGS) $(CUSTOMBUILDFLAGS) -I$(INCDIR) $(NETCDFINC) -I/opt/intel/composer_xe_2013.1.117/mkl/include $(AMBERBUILDFLAGS)
FNOOPTFLAGS= -O0
FOPTFLAGS= -ip -O3 -xHost
AMBERFFLAGS=$(AMBERBUILDFLAGS)
FREEFORMAT_FLAG= -FR
LM=-lm
FPP=cpp -traditional -P
FPPFLAGS= -DMKL -DBINTRAJ -DMPI  $(CUSTOMBUILDFLAGS) $(AMBERBUILDFLAGS)
AMBERFPPFLAGS=$(AMBERBUILDFLAGS)
FCREAL8=

XHOME= /usr
XLIBS= -L/usr/lib64 -L/usr/lib
MAKE_XLEAP=install_xleap

NETCDF=$(BASEDIR)/include/netcdf.mod
NETCDFLIB=$(BASEDIR)/lib/libnetcdf.a
NETCDFINC=-I$(BASEDIR)/include
FFTWLIB=

ZLIB=-lz
BZLIB=-lbz2

HASFC=yes
MTKPP=
XBLAS=
FFTW3=
MDGX=no

COMPILER=intel
MKL=/opt/intel/composer_xe_2013.1.117/mkl
MKL_PROCESSOR=intel64

#CUDA Specific build flags
NVCC=/usr/local/cuda-5.0/bin/nvcc -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -use_fast_math -O3
PMEMD_CU_INCLUDES=-I$(CUDA_HOME)/include -IB40C -IB40C/KernelCommon -I/usr/local/cuda-5.0/include -I/usr/local/cuda-5.0/include -I/usr/local/include
PMEMD_CU_LIBS=./cuda/cuda.a -L$(CUDA_HOME)/lib64 -L$(CUDA_HOME)/lib -lcurand -lcufft -lcudart -L/usr/lib64 -lstdc++
PMEMD_CU_DEFINES=-DCUDA -DMPI -DMPICH_IGNORE_CXX_SEEK -Duse_SPFP

#PMEMD Specific build flags
PMEMD_F90=mpif90 -DMPI   -DMKL -DBINTRAJ -DDIRFRC_EFS -DDIRFRC_COMTRANS -DDIRFRC_NOVEC -DFFTLOADBAL_2PROC -DPUBFFT
PMEMD_FOPTFLAGS=-ip -O3 -no-prec-div -xHost $(AMBERBUILDFLAGS)
PMEMD_CC=mpicc
PMEMD_COPTFLAGS=-ip -O3 -no-prec-div -xHost -DMPICH_IGNORE_CXX_SEEK -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -DBINTRAJ -DMPI  $(AMBERBUILDFLAGS)
PMEMD_FLIBSF=-Wl,--start-group /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_intel_lp64.a /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_sequential.a /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread
PMEMD_LD= mpif90  $(AMBERBUILDFLAGS)
LDOUT= -o

#for NAB:
MPI=mpi

#1D-RISM
RISM=no

#3D-RISM NAB
RISMSFF=
SFF_RISM_INTERFACE=
TESTRISMSFF=

#3D-RISM SANDER
RISMSANDER=
SANDER_RISM_INTERFACE=
FLIBS_RISMSANDER=
TESTRISMSANDER=

#PUPIL
PUPILLIBS=-lrt -lm -lc -L${PUPIL_PATH}/lib -lPUPIL -lPUPILBlind

#Python interpreter we are using
PYTHON=/usr/bin/python2.6
[root@RX350s7ambertest amber12]#


### mdin (sample: FactorIX NPT, the same as mdin file on Amber's website) ###

[root@RX350s7ambertest FactorIX_production_NPT]# less mdin
 Typical Production MD NVT
 &cntrl
  ntx=5, irest=1,
  ntc=2, ntf=2,
  nstlim=10000,
  ntpr=1000, ntwx=1000,
  ntwr=10000,
  dt=0.002, cut=8.,
  ntt=1, tautp=10.0,
  temp0=300.0,
  ntb=2, ntp=1, taup=10.0,
  ioutfm=1,
 /


### Execution command ###
mpirun -np 4 --hostfile hostfile -env MV2_CPU_MAPPING 8:9 -env MVA2_IBA_HCA mlx4_0 pmemd.cuda.MPI -O -i mdin -p prmtop -c inpcrd -o mdout_intmva2_gpu2node-4pro_0816-1


### Console log (After GPU didn't work, I typed [ctrl + C]. ### 

[root@RX350s7ambertest FactorIX_production_NPT]# mpirun -np 4 --hostfile hostfile -env MV2_CPU_MAPPING 8-9 -env MVA2_IBA_HCA mlx4_0 pmemd.cuda.MPI -O -i mdin -p prmtop -c inpcrd -o mdout_intmva2_gpu2node-4pro_0816-1
^C[mpiexec@RX350s7ambertest] Sending Ctrl-C to processes as requested
[mpiexec@RX350s7ambertest] Press Ctrl-C again to force abort
forrtl: error (69): process interrupted (SIGINT)
Image              PC                Routine            Line        Source
pmemd.cuda.MPI     0000000000799059  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000079B52C  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000076AC22  Unknown               Unknown  Unknown
pmemd.cuda.MPI     00000000007689BB  Unknown               Unknown  Unknown
pmemd.cuda.MPI     00000000007314ED  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000073133A  Unknown               Unknown  Unknown
pmemd.cuda.MPI     0000000000885048  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000088696B  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000086A571  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000072F41A  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000058DA28  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000058DAE8  Unknown               Unknown  Unknown
pmemd.cuda.MPI     00000000004955FA  Unknown               Unknown  Unknown
pmemd.cuda.MPI     00000000004BCEFF  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000050B18D  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000040A20C  Unknown               Unknown  Unknown
libc.so.6          0000003BEA61EC9D  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000040A109  Unknown               Unknown  Unknown
forrtl: error (69): process interrupted (SIGINT)
Image              PC                Routine            Line        Source
pmemd.cuda.MPI     000000000079AFE7  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000076AC22  Unknown               Unknown  Unknown
pmemd.cuda.MPI     0000000000769CC3  Unknown               Unknown  Unknown
pmemd.cuda.MPI     0000000000723B6A  Unknown               Unknown  Unknown
pmemd.cuda.MPI     0000000000812CD7  Unknown               Unknown  Unknown
pmemd.cuda.MPI     0000000000808DC6  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000080B720  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000070EC0D  Unknown               Unknown  Unknown
pmemd.cuda.MPI     0000000000700B52  Unknown               Unknown  Unknown
pmemd.cuda.MPI     0000000000495FE0  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000049564A  Unknown               Unknown  Unknown
pmemd.cuda.MPI     00000000004BCEFF  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000050B18D  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000040A20C  Unknown               Unknown  Unknown
libc.so.6          0000003BEA61EC9D  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000040A109  Unknown               Unknown  Unknown
forrtl: error (69): process interrupted (SIGINT)
Image              PC                Routine            Line        Source
pmemd.cuda.MPI     000000000079939E  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000076AC00  Unknown               Unknown  Unknown
pmemd.cuda.MPI     00000000007689BB  Unknown               Unknown  Unknown
pmemd.cuda.MPI     00000000007314ED  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000073133A  Unknown               Unknown  Unknown
pmemd.cuda.MPI     0000000000885048  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000088696B  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000086A571  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000072F41A  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000058DA28  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000058DAE8  Unknown               Unknown  Unknown
pmemd.cuda.MPI     00000000004955FA  Unknown               Unknown  Unknown
pmemd.cuda.MPI     00000000004BCEFF  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000050B18D  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000040A20C  Unknown               Unknown  Unknown
libc.so.6          0000003D7B21EC9D  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000040A109  Unknown               Unknown  Unknown
forrtl: error (69): process interrupted (SIGINT)
Image              PC                Routine            Line        Source
pmemd.cuda.MPI     00000000007707B5  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000076893A  Unknown               Unknown  Unknown
pmemd.cuda.MPI     00000000007314ED  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000073133A  Unknown               Unknown  Unknown
pmemd.cuda.MPI     0000000000885048  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000088696B  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000086A571  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000072F41A  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000058DA28  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000058DAE8  Unknown               Unknown  Unknown
pmemd.cuda.MPI     00000000004955FA  Unknown               Unknown  Unknown
pmemd.cuda.MPI     00000000004BCEFF  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000050B18D  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000040A20C  Unknown               Unknown  Unknown
libc.so.6          0000003D7B21EC9D  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000040A109  Unknown               Unknown  Unknown
^C[mpiexec@RX350s7ambertest] Sending Ctrl-C to processes as requested
[mpiexec@RX350s7ambertest] Press Ctrl-C again to force abort
forrtl: error (69): process interrupted (SIGINT)
Image              PC                Routine            Line        Source
pmemd.cuda.MPI     000000000076A473  Unknown               Unknown  Unknown
pmemd.cuda.MPI     0000000000723B6A  Unknown               Unknown  Unknown
pmemd.cuda.MPI     0000000000812CD7  Unknown               Unknown  Unknown
pmemd.cuda.MPI     0000000000808DC6  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000080B720  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000070EC0D  Unknown               Unknown  Unknown
pmemd.cuda.MPI     0000000000700B52  Unknown               Unknown  Unknown
pmemd.cuda.MPI     0000000000495FE0  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000049564A  Unknown               Unknown  Unknown
pmemd.cuda.MPI     00000000004BCEFF  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000050B18D  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000040A20C  Unknown               Unknown  Unknown
libc.so.6          0000003BEA61EC9D  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000040A109  Unknown               Unknown  Unknown
forrtl: error (69): process interrupted (SIGINT)
Image              PC                Routine            Line        Source
pmemd.cuda.MPI     000000000076DBC2  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000076890B  Unknown               Unknown  Unknown
pmemd.cuda.MPI     00000000007314ED  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000073133A  Unknown               Unknown  Unknown
pmemd.cuda.MPI     0000000000885048  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000088696B  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000086A571  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000072F41A  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000058DA28  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000058DAE8  Unknown               Unknown  Unknown
pmemd.cuda.MPI     00000000004955FA  Unknown               Unknown  Unknown
pmemd.cuda.MPI     00000000004BCEFF  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000050B18D  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000040A20C  Unknown               Unknown  Unknown
libc.so.6          0000003D7B21EC9D  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000040A109  Unknown               Unknown  Unknown
forrtl: error (69): process interrupted (SIGINT)
Image              PC                Routine            Line        Source
pmemd.cuda.MPI     00000000007707B5  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000076893A  Unknown               Unknown  Unknown
pmemd.cuda.MPI     00000000007314ED  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000073133A  Unknown               Unknown  Unknown
pmemd.cuda.MPI     0000000000885048  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000088696B  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000086A571  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000072F41A  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000058DA28  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000058DAE8  Unknown               Unknown  Unknown
pmemd.cuda.MPI     00000000004955FA  Unknown               Unknown  Unknown
pmemd.cuda.MPI     00000000004BCEFF  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000050B18D  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000040A20C  Unknown               Unknown  Unknown
libc.so.6          0000003D7B21EC9D  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000040A109  Unknown               Unknown  Unknown
forrtl: error (69): process interrupted (SIGINT)
Image              PC                Routine            Line        Source
libmlx4-m-rdmav2.  00007F4DD7C5F4C0  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000079B0B6  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000076AC22  Unknown               Unknown  Unknown
pmemd.cuda.MPI     00000000007689BB  Unknown               Unknown  Unknown
pmemd.cuda.MPI     00000000007314ED  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000073133A  Unknown               Unknown  Unknown
pmemd.cuda.MPI     0000000000885048  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000088696B  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000086A571  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000072F41A  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000058DA28  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000058DAE8  Unknown               Unknown  Unknown
pmemd.cuda.MPI     00000000004955FA  Unknown               Unknown  Unknown
pmemd.cuda.MPI     00000000004BCEFF  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000050B18D  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000040A20C  Unknown               Unknown  Unknown
libc.so.6          0000003BEA61EC9D  Unknown               Unknown  Unknown
pmemd.cuda.MPI     000000000040A109  Unknown               Unknown  Unknown
^CCtrl-C caught... cleaning up processes
[proxy:0:1@TX300S7Am]


### mdout (sample: Factor IX NPT) ### 

[root@RX350s7ambertest FactorIX_production_NPT]# less mdout_intmva2_gpu4node-4pro_0818-1

          -------------------------------------------------------
          Amber 12 SANDER                              2012
          -------------------------------------------------------

| PMEMD implementation of SANDER, Release 12

| Run on 08/16/2013 at 11:01:27

  [-O]verwriting output

File Assignments:
|   MDIN: mdin
|  MDOUT: mdout_intmva2_gpu4node-4pro_0818-1
| INPCRD: inpcrd
|   PARM: prmtop
| RESTRT: restrt
|   REFC: refc
|  MDVEL: mdvel
|   MDEN: mden
|  MDCRD: mdcrd
| MDINFO: mdinfo
|LOGFILE: logfile


 Here is the input file:

 Typical Production MD NVT
 &cntrl
  ntx=5, irest=1,
  ntc=2, ntf=2,
  nstlim=10000,
  ntpr=1000, ntwx=1000,
  ntwr=10000,
  dt=0.002, cut=8.,
  ntt=1, tautp=10.0,
  temp0=300.0,
  ntb=2, ntp=1, taup=10.0,
  ioutfm=1,
 /




|--------------------- INFORMATION ----------------------
| GPU (CUDA) Version of PMEMD in use: NVIDIA GPU IN USE.
|                     Version 12.3
|
|                      04/24/2013
|
| Implementation by:
|                    Ross C. Walker     (SDSC)
|                    Scott Le Grand     (nVIDIA)
|                    Duncan Poole       (nVIDIA)
|
| CAUTION: The CUDA code is currently experimental.
|          You use it at your own risk. Be sure to
|          check ALL results carefully.
|
| Precision model in use:
|      [SPFP] - Mixed Single/Double/Fixed Point Precision.
|               (Default)
|
|--------------------------------------------------------

|----------------- CITATION INFORMATION -----------------
|
|    When publishing work that utilized the CUDA version
|    of AMBER, please cite the following in addition to
|    the regular AMBER citations:
|
|  - Romelia Salomon-Ferrer; Andreas W. Goetz; Duncan
|    Poole; Scott Le Grand; Ross C. Walker "Routine
|    microsecond molecular dynamics simulations with
|    AMBER - Part II: Particle Mesh Ewald", J. Chem.
|    Theory Comput., 2012, (In review).
|
|  - Andreas W. Goetz; Mark J. Williamson; Dong Xu;
|    Duncan Poole; Scott Le Grand; Ross C. Walker
|    "Routine microsecond molecular dynamics simulations
|    with AMBER - Part I: Generalized Born", J. Chem.
|    Theory Comput., 2012, 8 (5), pp1542-1555.
|
|  - Scott Le Grand; Andreas W. Goetz; Ross C. Walker
|    "SPFP: Speed without compromise - a mixed precision
|    model for GPU accelerated molecular dynamics
|    simulations.", Comp. Phys. Comm., 2013, 184
|    pp374-380, DOI: 10.1016/j.cpc.2012.09.022
|
|--------------------------------------------------------

|------------------- GPU DEVICE INFO --------------------
|
|                         Task ID:      0
|   CUDA Capable Devices Detected:      2
|           CUDA Device ID in use:      0
|                CUDA Device Name: Tesla K20Xm
|     CUDA Device Global Mem Size:   6143 MB
| CUDA Device Num Multiprocessors:     14
|           CUDA Device Core Freq:   0.73 GHz
|
|
|                         Task ID:      1
|   CUDA Capable Devices Detected:      2
|           CUDA Device ID in use:      1
|                CUDA Device Name: Tesla K20Xm
|     CUDA Device Global Mem Size:   6143 MB
| CUDA Device Num Multiprocessors:     14
|           CUDA Device Core Freq:   0.73 GHz
|
|
|                         Task ID:      2
|   CUDA Capable Devices Detected:      2
|           CUDA Device ID in use:      0
|                CUDA Device Name: Tesla K20Xm
|     CUDA Device Global Mem Size:   6143 MB
| CUDA Device Num Multiprocessors:     14
|           CUDA Device Core Freq:   0.73 GHz
|
|
|                         Task ID:      3
|   CUDA Capable Devices Detected:      2
|           CUDA Device ID in use:      1
|                CUDA Device Name: Tesla K20Xm
|     CUDA Device Global Mem Size:   6143 MB
| CUDA Device Num Multiprocessors:     14
|           CUDA Device Core Freq:   0.73 GHz
|
|--------------------------------------------------------

| INFO:    Axis order optimization will be used.


| Conditional Compilation Defines Used:
| DIRFRC_COMTRANS
| DIRFRC_EFS
| DIRFRC_NOVEC
| MPI
| PUBFFT
| FFTLOADBAL_2PROC

          -------------------------------------------------------
          Amber 12 SANDER                              2012
          -------------------------------------------------------

| PMEMD implementation of SANDER, Release 12

| Run on 08/16/2013 at 11:45:26

  [-O]verwriting output

File Assignments:
|   MDIN: mdin
|  MDOUT: mdout_intmva2_gpu4node-4pro_0818-1
| INPCRD: inpcrd
|   PARM: prmtop
| RESTRT: restrt
|   REFC: refc
|  MDVEL: mdvel
|   MDEN: mden
|  MDCRD: mdcrd
| MDINFO: mdinfo
|LOGFILE: logfile


 Here is the input file:

 Typical Production MD NVT
 &cntrl
  ntx=5, irest=1,
  ntc=2, ntf=2,
  nstlim=10000,
  ntpr=1000, ntwx=1000,
  ntwr=10000,
  dt=0.002, cut=8.,
  ntt=1, tautp=10.0,
  temp0=300.0,
  ntb=2, ntp=1, taup=10.0,
  ioutfm=1,
 /




|--------------------- INFORMATION ----------------------
| GPU (CUDA) Version of PMEMD in use: NVIDIA GPU IN USE.
|                     Version 12.3
|
|                      04/24/2013
[root@RX350s7ambertest FactorIX_production_NPT]#
[root@RX350s7ambertest FactorIX_production_NPT]# cat mdout_intmva2_gpu4node-4pro_0818-1

          -------------------------------------------------------
          Amber 12 SANDER                              2012
          -------------------------------------------------------

| PMEMD implementation of SANDER, Release 12

| Run on 08/16/2013 at 11:45:26

  [-O]verwriting output

File Assignments:
|   MDIN: mdin
|  MDOUT: mdout_intmva2_gpu4node-4pro_0818-1
| INPCRD: inpcrd
|   PARM: prmtop
| RESTRT: restrt
|   REFC: refc
|  MDVEL: mdvel
|   MDEN: mden
|  MDCRD: mdcrd
| MDINFO: mdinfo
|LOGFILE: logfile


 Here is the input file:

 Typical Production MD NVT
 &cntrl
  ntx=5, irest=1,
  ntc=2, ntf=2,
  nstlim=10000,
  ntpr=1000, ntwx=1000,
  ntwr=10000,
  dt=0.002, cut=8.,
  ntt=1, tautp=10.0,
  temp0=300.0,
  ntb=2, ntp=1, taup=10.0,
  ioutfm=1,
 /




|--------------------- INFORMATION ----------------------
| GPU (CUDA) Version of PMEMD in use: NVIDIA GPU IN USE.
|                     Version 12.3
|
|                      04/24/2013
|
| Implementation by:
|                    Ross C. Walker     (SDSC)
|                    Scott Le Grand     (nVIDIA)
|                    Duncan Poole       (nVIDIA)
|
| CAUTION: The CUDA code is currently experimental.
|          You use it at your own risk. Be sure to
|          check ALL results carefully.
|
| Precision model in use:
|      [SPFP] - Mixed Single/Double/Fixed Point Precision.
|               (Default)
|
|--------------------------------------------------------

|----------------- CITATION INFORMATION -----------------
|
|    When publishing work that utilized the CUDA version
|    of AMBER, please cite the following in addition to
|    the regular AMBER citations:
|
|  - Romelia Salomon-Ferrer; Andreas W. Goetz; Duncan
|    Poole; Scott Le Grand; Ross C. Walker "Routine
|    microsecond molecular dynamics simulations with
|    AMBER - Part II: Particle Mesh Ewald", J. Chem.
|    Theory Comput., 2012, (In review).
|
|  - Andreas W. Goetz; Mark J. Williamson; Dong Xu;
|    Duncan Poole; Scott Le Grand; Ross C. Walker
|    "Routine microsecond molecular dynamics simulations
|    with AMBER - Part I: Generalized Born", J. Chem.
|    Theory Comput., 2012, 8 (5), pp1542-1555.
|
|  - Scott Le Grand; Andreas W. Goetz; Ross C. Walker
|    "SPFP: Speed without compromise - a mixed precision
|    model for GPU accelerated molecular dynamics
|    simulations.", Comp. Phys. Comm., 2013, 184
|    pp374-380, DOI: 10.1016/j.cpc.2012.09.022
|
|--------------------------------------------------------

|------------------- GPU DEVICE INFO --------------------
|
|                         Task ID:      0
|   CUDA Capable Devices Detected:      2
|           CUDA Device ID in use:      0
|                CUDA Device Name: Tesla K20Xm
|     CUDA Device Global Mem Size:   6143 MB
| CUDA Device Num Multiprocessors:     14
|           CUDA Device Core Freq:   0.73 GHz
|
|
|                         Task ID:      1
|   CUDA Capable Devices Detected:      2
|           CUDA Device ID in use:      1
|                CUDA Device Name: Tesla K20Xm
|     CUDA Device Global Mem Size:   6143 MB
| CUDA Device Num Multiprocessors:     14
|           CUDA Device Core Freq:   0.73 GHz
|
|
|                         Task ID:      2
|   CUDA Capable Devices Detected:      2
|           CUDA Device ID in use:      0
|                CUDA Device Name: Tesla K20Xm
|     CUDA Device Global Mem Size:   6143 MB
| CUDA Device Num Multiprocessors:     14
|           CUDA Device Core Freq:   0.73 GHz
|
|
|                         Task ID:      3
|   CUDA Capable Devices Detected:      2
|           CUDA Device ID in use:      1
|                CUDA Device Name: Tesla K20Xm
|     CUDA Device Global Mem Size:   6143 MB
| CUDA Device Num Multiprocessors:     14
|           CUDA Device Core Freq:   0.73 GHz
|
|--------------------------------------------------------

| INFO:    Axis order optimization will be used.


| Conditional Compilation Defines Used:
| DIRFRC_COMTRANS
| DIRFRC_EFS
| DIRFRC_NOVEC
| MPI
| PUBFFT
| FFTLOADBAL_2PROC
| BINTRAJ
| MKL
| CUDA

| Largest sphere to fit in unit cell has radius =    39.339

|  INFO: Old style PARM file read


| Note: 1-4 EEL scale factors were NOT found in the topology file.
|       Using default value of 1.2.

| Note: 1-4 VDW scale factors were NOT found in the topology file.
|       Using default value of 2.0.
| Duplicated  437 dihedrals

| Duplicated 1846 dihedrals

--------------------------------------------------------------------------------
   1.  RESOURCE   USE:
--------------------------------------------------------------------------------

 getting new box info from bottom of inpcrd

 NATOM  =   90906 NTYPES =      19 NBONH =   87891 MBONA  =    3077
 NTHETH =    6433 MTHETA =    4178 NPHIH =   11305 MPHIA  =    5519
 NHPARM =       0 NPARM  =       0 NNB   =  145596 NRES   =   28750
 NBONA  =    3077 NTHETA =    4178 NPHIA =    5519 NUMBND =      54
 NUMANG =     126 NPTRA  =      75 NATYP =      31 NPHB   =       1
 IFBOX  =       1 NMXRS  =      24 IFCAP =       0 NEXTRA =       0
 NCOPY  =       0

| Coordinate Index Table dimensions:    28   16   15
| Direct force subcell size =     5.0745    5.2086    5.2452

     BOX TYPE: RECTILINEAR

--------------------------------------------------------------------------------
   2.  CONTROL  DATA  FOR  THE  RUN
--------------------------------------------------------------------------------

factor IX (ACTIVATED PROTEIN)

General flags:
     imin    =       0, nmropt  =       0

Nature and format of input:
     ntx     =       5, irest   =       1, ntrx    =       1

Nature and format of output:
     ntxo    =       1, ntpr    =    1000, ntrx    =       1, ntwr    =   10000
     iwrap   =       0, ntwx    =    1000, ntwv    =       0, ntwe    =       0
     ioutfm  =       1, ntwprt  =       0, idecomp =       0, rbornstat=      0

Potential function:
     ntf     =       2, ntb     =       2, igb     =       0, nsnb    =      25
     ipol    =       0, gbsa    =       0, iesp    =       0
     dielc   =   1.00000, cut     =   8.00000, intdiel =   1.00000

Frozen or restrained atoms:
     ibelly  =       0, ntr     =       0

Molecular dynamics:
     nstlim  =     10000, nscm    =      1000, nrespa  =         1
     t       =   0.00000, dt      =   0.00200, vlimit  =  -1.00000

Berendsen (weak-coupling) temperature regulation:
     temp0   = 300.00000, tempi   =   0.00000, tautp   =  10.00000

Pressure regulation:
     ntp     =       1
     pres0   =   1.00000, comp    =  44.60000, taup    =  10.00000

SHAKE:
     ntc     =       2, jfastw  =       0
     tol     =   0.00001

| Intermolecular bonds treatment:
|     no_intermolecular_bonds =       1

| Energy averages sample interval:
|     ene_avg_sampling =    1000

Ewald parameters:
     verbose =       0, ew_type =       0, nbflag  =       1, use_pme =       1
     vdwmeth =       1, eedmeth =       1, netfrc  =       1
     Box X =  142.086   Box Y =   83.337   Box Z =   78.678
     Alpha =   90.000   Beta  =   90.000   Gamma =   90.000
     NFFT1 =  144       NFFT2 =   84       NFFT3 =   80
     Cutoff=    8.000   Tol   =0.100E-04
     Ewald Coefficient =  0.34864
     Interpolation order =    4

| PMEMD ewald parallel performance parameters:
|     block_fft =    0
|     fft_blk_y_divisor =    2
|     excl_recip =    0
|     excl_master =    0
|     atm_redist_freq =  320

--------------------------------------------------------------------------------
   3.  ATOMIC COORDINATES AND VELOCITIES
--------------------------------------------------------------------------------

factor IX (ACTIVATED PROTEIN)
 begin time read from input coords =  2542.675 ps


 Number of triangulated 3-point waters found:    28358

     Sum of charges from parm topology file =   0.00031225
     Forcing neutrality...

| Dynamic Memory, Types Used:
| Reals             3876951
| Integers          3639776

| Nonbonded Pairs Initial Allocation:     5181642

| GPU memory information (estimate):
| KB of GPU memory in use:     96029
| KB of CPU memory in use:     96029

| Running AMBER/MPI version on    4 nodes


--------------------------------------------------------------------------------
   4.  RESULTS
--------------------------------------------------------------------------------

 ---------------------------------------------------
 APPROXIMATING switch and d/dx switch using CUBIC SPLINE INTERPOLATION
 using   5000.0 points per unit in tabled values
 TESTING RELATIVE ERROR over r ranging from 0.0 to cutoff
| CHECK switch(x): max rel err =   0.2738E-14   at   2.422500
| CHECK d/dx switch(x): max rel err =   0.8332E-11   at   2.782960
 ---------------------------------------------------
|---------------------------------------------------
| APPROXIMATING direct energy using CUBIC SPLINE INTERPOLATION
|  with   50.0 points per unit in tabled values
| Relative Error Limit not exceeded for r .gt.   2.47
| APPROXIMATING direct force using CUBIC SPLINE INTERPOLATION
|  with   50.0 points per unit in tabled values
| Relative Error Limit not exceeded for r .gt.   2.89
|---------------------------------------------------
check COM velocity, temp:        0.000018     0.00(Removed)

 NSTEP =     1000   TIME(PS) =    2544.675  TEMP(K) =   296.90  PRESS =  -694.5
 Etot   =   -233748.6862  EKtot   =     54523.3398  EPtot      =   -288272.0261
 BOND   =      1137.6172  ANGLE   =      2934.5131  DIHED      =      2246.0865
 1-4 NB =      1288.3081  1-4 EEL =     15112.1345  VDWAALS    =     36457.3797
 EELEC  =   -347448.0652  EHBOND  =         0.0000  RESTRAINT  =         0.0000
 EKCMT  =     25361.2361  VIRIAL  =     39519.1411  VOLUME     =    944163.2170
                                                    Density    =         0.9747
 ------------------------------------------------------------------------------

check COM velocity, temp:        0.000027     0.00(Removed)

 NSTEP =     2000   TIME(PS) =    2546.675  TEMP(K) =   295.89  PRESS =  -968.7
 Etot   =   -233339.5533  EKtot   =     54337.9648  EPtot      =   -287677.5181
 BOND   =      1127.3253  ANGLE   =      2924.9643  DIHED      =      2215.0624
 1-4 NB =      1298.3079  1-4 EEL =     15114.4109  VDWAALS    =     35626.9347
 EELEC  =   -345984.5237  EHBOND  =         0.0000  RESTRAINT  =         0.0000
 EKCMT  =     25131.5725  VIRIAL  =     45093.5005  VOLUME     =    954386.4124
                                                    Density    =         0.9643
 ------------------------------------------------------------------------------



### "nvidia-smi -l 1" log (node 0):  Job doesn't finished but ###

Fri Aug 16 11:01:27 2013                                                          *** test start
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   31C    P8    23W / 235W |       14MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   32C    P8    19W / 235W |       14MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|  No running compute processes found                                         |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:28 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   31C    P0    22W / 235W |       84MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   32C    P0    21W / 235W |       84MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                        68MB  |
|    1     30845  pmemd.cuda.MPI                                        68MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:29 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   32C    P0    62W / 235W |      197MB /  6143MB |     54%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   33C    P0    61W / 235W |      240MB /  6143MB |     62%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                       180MB  |
|    1     30845  pmemd.cuda.MPI                                       224MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:30 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   33C    P0    88W / 235W |      197MB /  6143MB |     54%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   34C    P0    87W / 235W |      240MB /  6143MB |     60%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                       180MB  |
|    1     30845  pmemd.cuda.MPI                                       224MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:31 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   34C    P0    96W / 235W |      197MB /  6143MB |     52%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   34C    P0    95W / 235W |      240MB /  6143MB |     61%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                       180MB  |
|    1     30845  pmemd.cuda.MPI                                       224MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:32 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   34C    P0    98W / 235W |      197MB /  6143MB |     53%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   34C    P0    97W / 235W |      240MB /  6143MB |     61%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                       180MB  |
|    1     30845  pmemd.cuda.MPI                                       224MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:33 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   34C    P0    99W / 235W |      197MB /  6143MB |     53%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   35C    P0    98W / 235W |      240MB /  6143MB |     61%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                       180MB  |
|    1     30845  pmemd.cuda.MPI                                       224MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:34 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   34C    P0   100W / 235W |      197MB /  6143MB |     54%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   35C    P0    98W / 235W |      240MB /  6143MB |     60%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                       180MB  |
|    1     30845  pmemd.cuda.MPI                                       224MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:35 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   35C    P0    99W / 235W |      197MB /  6143MB |     53%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   35C    P0    98W / 235W |      240MB /  6143MB |     61%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                       180MB  |
|    1     30845  pmemd.cuda.MPI                                       224MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:36 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   34C    P0    99W / 235W |      197MB /  6143MB |     53%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   35C    P0    98W / 235W |      240MB /  6143MB |     61%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                       180MB  |
|    1     30845  pmemd.cuda.MPI                                       224MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:37 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   35C    P0   100W / 235W |      197MB /  6143MB |     54%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   35C    P0    98W / 235W |      240MB /  6143MB |     62%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                       180MB  |
|    1     30845  pmemd.cuda.MPI                                       224MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:38 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   35C    P0   100W / 235W |      197MB /  6143MB |     53%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   35C    P0    98W / 235W |      240MB /  6143MB |     60%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                       180MB  |
|    1     30845  pmemd.cuda.MPI                                       224MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:39 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   35C    P0    99W / 235W |      197MB /  6143MB |     53%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   35C    P0    98W / 235W |      240MB /  6143MB |     60%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                       180MB  |
|    1     30845  pmemd.cuda.MPI                                       224MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:40 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   35C    P0    99W / 235W |      197MB /  6143MB |     53%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   35C    P0    98W / 235W |      240MB /  6143MB |     60%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                       180MB  |
|    1     30845  pmemd.cuda.MPI                                       224MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:41 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   35C    P0    99W / 235W |      197MB /  6143MB |     53%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   36C    P0    98W / 235W |      240MB /  6143MB |     60%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                       180MB  |
|    1     30845  pmemd.cuda.MPI                                       224MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:42 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   35C    P0    99W / 235W |      197MB /  6143MB |     53%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   36C    P0    98W / 235W |      240MB /  6143MB |     60%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                       180MB  |
|    1     30845  pmemd.cuda.MPI                                       224MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:43 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   35C    P0   100W / 235W |      197MB /  6143MB |     53%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   36C    P0    98W / 235W |      240MB /  6143MB |     60%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                       180MB  |
|    1     30845  pmemd.cuda.MPI                                       224MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:44 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   35C    P0    99W / 235W |      197MB /  6143MB |     54%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   36C    P0    98W / 235W |      240MB /  6143MB |     60%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                       180MB  |
|    1     30845  pmemd.cuda.MPI                                       224MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:45 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   35C    P0    99W / 235W |      197MB /  6143MB |     53%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   36C    P0    98W / 235W |      240MB /  6143MB |     60%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                       180MB  |
|    1     30845  pmemd.cuda.MPI                                       224MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:46 2013                                                          *** GPU stop and don't work any more
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   34C    P0    69W / 235W |      197MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   35C    P0    68W / 235W |      240MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                       180MB  |
|    1     30845  pmemd.cuda.MPI                                       224MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:47 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   34C    P0    59W / 235W |      197MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   34C    P0    59W / 235W |      240MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                       180MB  |
|    1     30845  pmemd.cuda.MPI                                       224MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:48 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   34C    P0    57W / 235W |      197MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   34C    P0    56W / 235W |      240MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                       180MB  |
|    1     30845  pmemd.cuda.MPI                                       224MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:49 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   34C    P0    56W / 235W |      197MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   34C    P0    55W / 235W |      240MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                       180MB  |
|    1     30845  pmemd.cuda.MPI                                       224MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:50 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   34C    P0    55W / 235W |      197MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   34C    P0    55W / 235W |      240MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                       180MB  |
|    1     30845  pmemd.cuda.MPI                                       224MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:51 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   33C    P0    55W / 235W |      197MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   34C    P0    55W / 235W |      240MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     30844  pmemd.cuda.MPI                                       180MB  |
|    1     30845  pmemd.cuda.MPI                                       224MB  |
+-----------------------------------------------------------------------------+
^C[root@RX350s7ambertest ~]#




### "nvidia-smi -l 1" log (node 1) ###

Fri Aug 16 11:01:26 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   29C    P8    28W / 235W |       14MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   31C    P8    23W / 235W |       14MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|  No running compute processes found                                         |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:27 2013                                                                 *** test start
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   29C    P0    21W / 235W |       84MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   31C    P0    21W / 235W |       84MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     32138  pmemd.cuda.MPI                                        68MB  |
|    1     32139  pmemd.cuda.MPI                                        68MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:28 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   30C    P0    55W / 235W |      239MB /  6143MB |     54%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   32C    P0    57W / 235W |      239MB /  6143MB |     32%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     32138  pmemd.cuda.MPI                                       223MB  |
|    1     32139  pmemd.cuda.MPI                                       223MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:29 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   31C    P0    85W / 235W |      239MB /  6143MB |     59%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   32C    P0    87W / 235W |      239MB /  6143MB |     59%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     32138  pmemd.cuda.MPI                                       223MB  |
|    1     32139  pmemd.cuda.MPI                                       223MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:30 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   31C    P0    94W / 235W |      239MB /  6143MB |     60%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   33C    P0    96W / 235W |      239MB /  6143MB |     60%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     32138  pmemd.cuda.MPI                                       223MB  |
|    1     32139  pmemd.cuda.MPI                                       223MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:31 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   32C    P0    97W / 235W |      239MB /  6143MB |     59%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   33C    P0    98W / 235W |      239MB /  6143MB |     59%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     32138  pmemd.cuda.MPI                                       223MB  |
|    1     32139  pmemd.cuda.MPI                                       223MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:32 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   32C    P0    98W / 235W |      239MB /  6143MB |     61%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   34C    P0    99W / 235W |      239MB /  6143MB |     59%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     32138  pmemd.cuda.MPI                                       223MB  |
|    1     32139  pmemd.cuda.MPI                                       223MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:33 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   32C    P0    98W / 235W |      239MB /  6143MB |     59%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   34C    P0   100W / 235W |      239MB /  6143MB |     60%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     32138  pmemd.cuda.MPI                                       223MB  |
|    1     32139  pmemd.cuda.MPI                                       223MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:34 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   32C    P0    98W / 235W |      239MB /  6143MB |     59%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   34C    P0    99W / 235W |      239MB /  6143MB |     58%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     32138  pmemd.cuda.MPI                                       223MB  |
|    1     32139  pmemd.cuda.MPI                                       223MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:35 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   32C    P0    98W / 235W |      239MB /  6143MB |     59%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   34C    P0   100W / 235W |      239MB /  6143MB |     60%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     32138  pmemd.cuda.MPI                                       223MB  |
|    1     32139  pmemd.cuda.MPI                                       223MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:36 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   33C    P0    98W / 235W |      239MB /  6143MB |     60%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   34C    P0   100W / 235W |      239MB /  6143MB |     59%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     32138  pmemd.cuda.MPI                                       223MB  |
|    1     32139  pmemd.cuda.MPI                                       223MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:37 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   32C    P0    98W / 235W |      239MB /  6143MB |     58%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   34C    P0   100W / 235W |      239MB /  6143MB |     58%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     32138  pmemd.cuda.MPI                                       223MB  |
|    1     32139  pmemd.cuda.MPI                                       223MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:38 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   33C    P0    98W / 235W |      239MB /  6143MB |     58%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   34C    P0   100W / 235W |      239MB /  6143MB |     59%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     32138  pmemd.cuda.MPI                                       223MB  |
|    1     32139  pmemd.cuda.MPI                                       223MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:39 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   33C    P0    97W / 235W |      239MB /  6143MB |     55%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   34C    P0    99W / 235W |      239MB /  6143MB |     57%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     32138  pmemd.cuda.MPI                                       223MB  |
|    1     32139  pmemd.cuda.MPI                                       223MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:40 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   33C    P0    97W / 235W |      239MB /  6143MB |     59%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   34C    P0   100W / 235W |      239MB /  6143MB |     59%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     32138  pmemd.cuda.MPI                                       223MB  |
|    1     32139  pmemd.cuda.MPI                                       223MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:41 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   33C    P0    97W / 235W |      239MB /  6143MB |     58%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   35C    P0   100W / 235W |      239MB /  6143MB |     59%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     32138  pmemd.cuda.MPI                                       223MB  |
|    1     32139  pmemd.cuda.MPI                                       223MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:42 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   33C    P0    97W / 235W |      239MB /  6143MB |     58%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   35C    P0   100W / 235W |      239MB /  6143MB |     59%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     32138  pmemd.cuda.MPI                                       223MB  |
|    1     32139  pmemd.cuda.MPI                                       223MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:43 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   33C    P0    98W / 235W |      239MB /  6143MB |     58%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   35C    P0   100W / 235W |      239MB /  6143MB |     59%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     32138  pmemd.cuda.MPI                                       223MB  |
|    1     32139  pmemd.cuda.MPI                                       223MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:44 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   33C    P0    98W / 235W |      239MB /  6143MB |     60%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   35C    P0   100W / 235W |      239MB /  6143MB |     59%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     32138  pmemd.cuda.MPI                                       223MB  |
|    1     32139  pmemd.cuda.MPI                                       223MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:45 2013                                                         *** GPU stop and don't work any more
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   32C    P0    71W / 235W |      239MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   34C    P0    72W / 235W |      239MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     32138  pmemd.cuda.MPI                                       223MB  |
|    1     32139  pmemd.cuda.MPI                                       223MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:46 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   32C    P0    60W / 235W |      239MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   33C    P0    61W / 235W |      239MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     32138  pmemd.cuda.MPI                                       223MB  |
|    1     32139  pmemd.cuda.MPI                                       223MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:47 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   32C    P0    56W / 235W |      239MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   33C    P0    58W / 235W |      239MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     32138  pmemd.cuda.MPI                                       223MB  |
|    1     32139  pmemd.cuda.MPI                                       223MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:48 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   32C    P0    55W / 235W |      239MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   33C    P0    57W / 235W |      239MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     32138  pmemd.cuda.MPI                                       223MB  |
|    1     32139  pmemd.cuda.MPI                                       223MB  |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:49 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20Xm         Off  | 0000:84:00.0     Off |                  Off |
| N/A   31C    P0    55W / 235W |      239MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K20Xm         Off  | 0000:85:00.0     Off |                  Off |
| N/A   33C    P0    56W / 235W |      239MB /  6143MB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0     32138  pmemd.cuda.MPI                                       223MB  |
|    1     32139  pmemd.cuda.MPI                                       223MB  |
+-----------------------------------------------------------------------------+
^C[root@TX300S7Am ~]#


Thank you for your support, 
Best wishes,

Y. Nakashima