[AMBER] Often, Amber BMT doesn't finish properly with Malti-GPGPU. (pmemd.cuda.MPI problem)

From: Nakashima, Yoshihisa <nakashima_y.jp.fujitsu.com>
Date: Fri, 16 Aug 2013 08:16:24 +0000

Dear Amber users,

I'm trying to Amber BMT (on Amber's website) with 2 node 4 GPGPU (2 GPGPU in a node).
When I test "Factor IX NPT", "Factor IX NVE", "JAC NPT" and "JAC NVE", often (not always! About 60 %) the test doesn't finish properly.
Processes are working but it seems that GPUs don't work anymore and there are no error logs.

However, there is no problem with other test (TRPCage, nucleosome, myoglobin, Cellulose NPT and NVE")

Could you give me an advice ?

Following is system configuration and logs.
To be safe, I attach log files on this E-mail.

----------------------------------------------------------------
### Configuration ###

- CPU: E5-2680
- GPU: Tesla K20 or K20X

- OS: RHEL6.1
- Amber 12 (patched bugfix 1 to 18)
- AmberTool 13 (patched bugfix 1 to 15)
- Intel Compiler: 2013.1.117
- MVAPICH2: 1.9
- GPU Device driver: 304.64 and 319.32 (latest version)
- CUDA: 5.0.35
- Interconnect: Mellanox FDR HCA card x1 per node (with Mellanox OFED 1.5.3-3.0.0)

** The same issue occur with the case of Intel compiler + Intel MPI.


### config.h ###

[root.RX350s7ambertest amber12]# cat config.h
# Amber configuration file, created with: ./configure -mpi -cuda intel

###############################################################################

# (1) Location of the installation

BASEDIR=/root/amber/amber12
BINDIR=/root/amber/amber12/bin
LIBDIR=/root/amber/amber12/lib
INCDIR=/root/amber/amber12/include
DATDIR=/root/amber/amber12/dat
LOGDIR=/root/amber/amber12/logs

###############################################################################


# (2) If you want to search additional libraries by default, add them
# to the FLIBS variable here. (External libraries can also be linked into
# NAB programs simply by including them on the command line; libraries
# included in FLIBS are always searched.)

FLIBS= -lsff_mpi -lpbsa -larpack $(BASEDIR)/lib/libnetcdf.a -Wl,--start-group /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_intel_lp64.a /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_sequential.a /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -L/opt/intel/composer_xe_2013.1.117/lib/intel64 -lifport -lifcore -lsvml
FLIBS_PTRAJ= -larpack -Wl,--start-group /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_intel_lp64.a /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_sequential.a /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -L/opt/intel/composer_xe_2013.1.117/lib/intel64 -lifport -lifcore -lsvml
FLIBSF= -larpack -Wl,--start-group /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_intel_lp64.a /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_sequential.a /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lsvml
FLIBS_FFTW3=
###############################################################################

# (3) Modify any of the following if you need to change, e.g. to use gcc
# rather than cc, etc.

SHELL=/bin/sh
INSTALLTYPE=cuda_parallel
BUILDAMBER=amber

# Set the C compiler, etc.

# The configure script should be fine, but if you need to hand-edit,
# here is some info:

# Example: CC-->gcc; LEX-->flex; YACC-->yacc (built in byacc)
# Note: If your lexer is "really" flex, you need to set
# LEX=flex below. For example, on some distributions,
# /usr/bin/lex is really just a pointer to /usr/bin/flex,
# so LEX=flex is necessary. In general, gcc seems to need flex.

# The compiler flags CFLAGS and CXXFLAGS should always be used.
# By contrast, *OPTFLAGS and *NOOPTFLAGS will only be used with
# certain files, and usually at compile-time but not link-time.
# Where *OPTFLAGS and *NOOPTFLAGS are requested (in Makefiles,
# makedepend and depend), they should come before CFLAGS or
# CXXFLAGS; this allows the user to override *OPTFLAGS and
# *NOOPTFLAGS using the BUILDFLAGS variable.

# AMBERBUILDFLAGS provides a hook into all stages of the build process.
# It can be used to build debug versions, invoke special features, etc.
# Example: make AMBERBUILDFLAGS='-O0 -g' sander
#
CC=mpicc
CFLAGS= -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -DBINTRAJ -DHASGZ -DHASBZ2 -DMPI $(CUSTOMBUILDFLAGS) -I/opt/intel/composer_xe_2013.1.117/mkl/include $(AMBERBUILDFLAGS)
CNOOPTFLAGS=
COPTFLAGS=-ip -O3 -xHost
AMBERCFLAGS= $(AMBERBUILDFLAGS)

CXX=icpc
CPLUSPLUS=icpc
CXXFLAGS= -DMPI $(CUSTOMBUILDFLAGS) $(AMBERBUILDFLAGS)
CXXNOOPTFLAGS=
CXXOPTFLAGS=-O3
AMBERCXXFLAGS= $(AMBERBUILDFLAGS)

NABFLAGS= $(AMBERBUILDFLAGS)
PBSAFLAG= $(AMBERBUILDFLAGS)

LDFLAGS=-shared-intel $(CUSTOMBUILDFLAGS) $(AMBERBUILDFLAGS)
AMBERLDFLAGS=$(AMBERBUILDFLAGS)

LEX= flex
YACC= $(BINDIR)/yacc
AR= ar rv
M4= m4
RANLIB=ranlib

# Set the C-preprocessor. Code for a small preprocessor is in
# ucpp-1.3; it gets installed as $(BINDIR)/ucpp;

CPP=ucpp -l

# These variables control whether we will use compiled versions of BLAS
# and LAPACK (which are generally slower), or whether those libraries are
# already available (presumably in an optimized form).

LAPACK=skip
BLAS=skip
F2C=skip

# These variables determine whether builtin versions of certain components
# can be used, or whether we need to compile our own versions.

UCPP=install
C9XCOMPLEX=skip

# For Windows/cygwin, set SFX to ".exe"; for Unix/Linux leave it empty:
# Set OBJSFX to ".obj" instead of ".o" on Windows:

SFX=
OSFX=.o
MV=mv
RM=rm
CP=cp

# Information about Fortran compilation:

FC=mpif90
FFLAGS= $(LOCALFLAGS) $(CUSTOMBUILDFLAGS) -I$(INCDIR) $(NETCDFINC) -I/opt/intel/composer_xe_2013.1.117/mkl/include $(AMBERBUILDFLAGS)
FNOOPTFLAGS= -O0
FOPTFLAGS= -ip -O3 -xHost
AMBERFFLAGS=$(AMBERBUILDFLAGS)
FREEFORMAT_FLAG= -FR
LM=-lm
FPP=cpp -traditional -P
FPPFLAGS= -DMKL -DBINTRAJ -DMPI $(CUSTOMBUILDFLAGS) $(AMBERBUILDFLAGS)
AMBERFPPFLAGS=$(AMBERBUILDFLAGS)
FCREAL8=

XHOME= /usr
XLIBS= -L/usr/lib64 -L/usr/lib
MAKE_XLEAP=install_xleap

NETCDF=$(BASEDIR)/include/netcdf.mod
NETCDFLIB=$(BASEDIR)/lib/libnetcdf.a
NETCDFINC=-I$(BASEDIR)/include
FFTWLIB=

ZLIB=-lz
BZLIB=-lbz2

HASFC=yes
MTKPP=
XBLAS=
FFTW3=
MDGX=no

COMPILER=intel
MKL=/opt/intel/composer_xe_2013.1.117/mkl
MKL_PROCESSOR=intel64

#CUDA Specific build flags
NVCC=/usr/local/cuda-5.0/bin/nvcc -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -use_fast_math -O3
PMEMD_CU_INCLUDES=-I$(CUDA_HOME)/include -IB40C -IB40C/KernelCommon -I/usr/local/cuda-5.0/include -I/usr/local/cuda-5.0/include -I/usr/local/include
PMEMD_CU_LIBS=./cuda/cuda.a -L$(CUDA_HOME)/lib64 -L$(CUDA_HOME)/lib -lcurand -lcufft -lcudart -L/usr/lib64 -lstdc++
PMEMD_CU_DEFINES=-DCUDA -DMPI -DMPICH_IGNORE_CXX_SEEK -Duse_SPFP

#PMEMD Specific build flags
PMEMD_F90=mpif90 -DMPI -DMKL -DBINTRAJ -DDIRFRC_EFS -DDIRFRC_COMTRANS -DDIRFRC_NOVEC -DFFTLOADBAL_2PROC -DPUBFFT
PMEMD_FOPTFLAGS=-ip -O3 -no-prec-div -xHost $(AMBERBUILDFLAGS)
PMEMD_CC=mpicc
PMEMD_COPTFLAGS=-ip -O3 -no-prec-div -xHost -DMPICH_IGNORE_CXX_SEEK -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -DBINTRAJ -DMPI $(AMBERBUILDFLAGS)
PMEMD_FLIBSF=-Wl,--start-group /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_intel_lp64.a /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_sequential.a /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread
PMEMD_LD= mpif90 $(AMBERBUILDFLAGS)
LDOUT= -o

#for NAB:
MPI=mpi

#1D-RISM
RISM=no

#3D-RISM NAB
RISMSFF=
SFF_RISM_INTERFACE=
TESTRISMSFF=

#3D-RISM SANDER
RISMSANDER=
SANDER_RISM_INTERFACE=
FLIBS_RISMSANDER=
TESTRISMSANDER=

#PUPIL
PUPILLIBS=-lrt -lm -lc -L${PUPIL_PATH}/lib -lPUPIL -lPUPILBlind

#Python interpreter we are using
PYTHON=/usr/bin/python2.6
[root.RX350s7ambertest amber12]#


### mdin (sample: FactorIX NPT, the same as mdin file on Amber's website) ###

[root.RX350s7ambertest FactorIX_production_NPT]# less mdin
 Typical Production MD NVT
 &cntrl
  ntx=5, irest=1,
  ntc=2, ntf=2,
  nstlim=10000,
  ntpr=1000, ntwx=1000,
  ntwr=10000,
  dt=0.002, cut=8.,
  ntt=1, tautp=10.0,
  temp0=300.0,
  ntb=2, ntp=1, taup=10.0,
  ioutfm=1,
 /


### Execution command ###
mpirun -np 4 --hostfile hostfile -env MV2_CPU_MAPPING 8:9 -env MVA2_IBA_HCA mlx4_0 pmemd.cuda.MPI -O -i mdin -p prmtop -c inpcrd -o mdout_intmva2_gpu2node-4pro_0816-1


### Console log (After GPU didn't work, I typed [ctrl + C]. ###

[root.RX350s7ambertest FactorIX_production_NPT]# mpirun -np 4 --hostfile hostfile -env MV2_CPU_MAPPING 8-9 -env MVA2_IBA_HCA mlx4_0 pmemd.cuda.MPI -O -i mdin -p prmtop -c inpcrd -o mdout_intmva2_gpu2node-4pro_0816-1
^C[mpiexec.RX350s7ambertest] Sending Ctrl-C to processes as requested
[mpiexec.RX350s7ambertest] Press Ctrl-C again to force abort
forrtl: error (69): process interrupted (SIGINT)
Image PC Routine Line Source
pmemd.cuda.MPI 0000000000799059 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000079B52C Unknown Unknown Unknown
pmemd.cuda.MPI 000000000076AC22 Unknown Unknown Unknown
pmemd.cuda.MPI 00000000007689BB Unknown Unknown Unknown
pmemd.cuda.MPI 00000000007314ED Unknown Unknown Unknown
pmemd.cuda.MPI 000000000073133A Unknown Unknown Unknown
pmemd.cuda.MPI 0000000000885048 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000088696B Unknown Unknown Unknown
pmemd.cuda.MPI 000000000086A571 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000072F41A Unknown Unknown Unknown
pmemd.cuda.MPI 000000000058DA28 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000058DAE8 Unknown Unknown Unknown
pmemd.cuda.MPI 00000000004955FA Unknown Unknown Unknown
pmemd.cuda.MPI 00000000004BCEFF Unknown Unknown Unknown
pmemd.cuda.MPI 000000000050B18D Unknown Unknown Unknown
pmemd.cuda.MPI 000000000040A20C Unknown Unknown Unknown
libc.so.6 0000003BEA61EC9D Unknown Unknown Unknown
pmemd.cuda.MPI 000000000040A109 Unknown Unknown Unknown
forrtl: error (69): process interrupted (SIGINT)
Image PC Routine Line Source
pmemd.cuda.MPI 000000000079AFE7 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000076AC22 Unknown Unknown Unknown
pmemd.cuda.MPI 0000000000769CC3 Unknown Unknown Unknown
pmemd.cuda.MPI 0000000000723B6A Unknown Unknown Unknown
pmemd.cuda.MPI 0000000000812CD7 Unknown Unknown Unknown
pmemd.cuda.MPI 0000000000808DC6 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000080B720 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000070EC0D Unknown Unknown Unknown
pmemd.cuda.MPI 0000000000700B52 Unknown Unknown Unknown
pmemd.cuda.MPI 0000000000495FE0 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000049564A Unknown Unknown Unknown
pmemd.cuda.MPI 00000000004BCEFF Unknown Unknown Unknown
pmemd.cuda.MPI 000000000050B18D Unknown Unknown Unknown
pmemd.cuda.MPI 000000000040A20C Unknown Unknown Unknown
libc.so.6 0000003BEA61EC9D Unknown Unknown Unknown
pmemd.cuda.MPI 000000000040A109 Unknown Unknown Unknown
forrtl: error (69): process interrupted (SIGINT)
Image PC Routine Line Source
pmemd.cuda.MPI 000000000079939E Unknown Unknown Unknown
pmemd.cuda.MPI 000000000076AC00 Unknown Unknown Unknown
pmemd.cuda.MPI 00000000007689BB Unknown Unknown Unknown
pmemd.cuda.MPI 00000000007314ED Unknown Unknown Unknown
pmemd.cuda.MPI 000000000073133A Unknown Unknown Unknown
pmemd.cuda.MPI 0000000000885048 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000088696B Unknown Unknown Unknown
pmemd.cuda.MPI 000000000086A571 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000072F41A Unknown Unknown Unknown
pmemd.cuda.MPI 000000000058DA28 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000058DAE8 Unknown Unknown Unknown
pmemd.cuda.MPI 00000000004955FA Unknown Unknown Unknown
pmemd.cuda.MPI 00000000004BCEFF Unknown Unknown Unknown
pmemd.cuda.MPI 000000000050B18D Unknown Unknown Unknown
pmemd.cuda.MPI 000000000040A20C Unknown Unknown Unknown
libc.so.6 0000003D7B21EC9D Unknown Unknown Unknown
pmemd.cuda.MPI 000000000040A109 Unknown Unknown Unknown
forrtl: error (69): process interrupted (SIGINT)
Image PC Routine Line Source
pmemd.cuda.MPI 00000000007707B5 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000076893A Unknown Unknown Unknown
pmemd.cuda.MPI 00000000007314ED Unknown Unknown Unknown
pmemd.cuda.MPI 000000000073133A Unknown Unknown Unknown
pmemd.cuda.MPI 0000000000885048 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000088696B Unknown Unknown Unknown
pmemd.cuda.MPI 000000000086A571 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000072F41A Unknown Unknown Unknown
pmemd.cuda.MPI 000000000058DA28 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000058DAE8 Unknown Unknown Unknown
pmemd.cuda.MPI 00000000004955FA Unknown Unknown Unknown
pmemd.cuda.MPI 00000000004BCEFF Unknown Unknown Unknown
pmemd.cuda.MPI 000000000050B18D Unknown Unknown Unknown
pmemd.cuda.MPI 000000000040A20C Unknown Unknown Unknown
libc.so.6 0000003D7B21EC9D Unknown Unknown Unknown
pmemd.cuda.MPI 000000000040A109 Unknown Unknown Unknown
^C[mpiexec.RX350s7ambertest] Sending Ctrl-C to processes as requested
[mpiexec.RX350s7ambertest] Press Ctrl-C again to force abort
forrtl: error (69): process interrupted (SIGINT)
Image PC Routine Line Source
pmemd.cuda.MPI 000000000076A473 Unknown Unknown Unknown
pmemd.cuda.MPI 0000000000723B6A Unknown Unknown Unknown
pmemd.cuda.MPI 0000000000812CD7 Unknown Unknown Unknown
pmemd.cuda.MPI 0000000000808DC6 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000080B720 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000070EC0D Unknown Unknown Unknown
pmemd.cuda.MPI 0000000000700B52 Unknown Unknown Unknown
pmemd.cuda.MPI 0000000000495FE0 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000049564A Unknown Unknown Unknown
pmemd.cuda.MPI 00000000004BCEFF Unknown Unknown Unknown
pmemd.cuda.MPI 000000000050B18D Unknown Unknown Unknown
pmemd.cuda.MPI 000000000040A20C Unknown Unknown Unknown
libc.so.6 0000003BEA61EC9D Unknown Unknown Unknown
pmemd.cuda.MPI 000000000040A109 Unknown Unknown Unknown
forrtl: error (69): process interrupted (SIGINT)
Image PC Routine Line Source
pmemd.cuda.MPI 000000000076DBC2 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000076890B Unknown Unknown Unknown
pmemd.cuda.MPI 00000000007314ED Unknown Unknown Unknown
pmemd.cuda.MPI 000000000073133A Unknown Unknown Unknown
pmemd.cuda.MPI 0000000000885048 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000088696B Unknown Unknown Unknown
pmemd.cuda.MPI 000000000086A571 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000072F41A Unknown Unknown Unknown
pmemd.cuda.MPI 000000000058DA28 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000058DAE8 Unknown Unknown Unknown
pmemd.cuda.MPI 00000000004955FA Unknown Unknown Unknown
pmemd.cuda.MPI 00000000004BCEFF Unknown Unknown Unknown
pmemd.cuda.MPI 000000000050B18D Unknown Unknown Unknown
pmemd.cuda.MPI 000000000040A20C Unknown Unknown Unknown
libc.so.6 0000003D7B21EC9D Unknown Unknown Unknown
pmemd.cuda.MPI 000000000040A109 Unknown Unknown Unknown
forrtl: error (69): process interrupted (SIGINT)
Image PC Routine Line Source
pmemd.cuda.MPI 00000000007707B5 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000076893A Unknown Unknown Unknown
pmemd.cuda.MPI 00000000007314ED Unknown Unknown Unknown
pmemd.cuda.MPI 000000000073133A Unknown Unknown Unknown
pmemd.cuda.MPI 0000000000885048 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000088696B Unknown Unknown Unknown
pmemd.cuda.MPI 000000000086A571 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000072F41A Unknown Unknown Unknown
pmemd.cuda.MPI 000000000058DA28 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000058DAE8 Unknown Unknown Unknown
pmemd.cuda.MPI 00000000004955FA Unknown Unknown Unknown
pmemd.cuda.MPI 00000000004BCEFF Unknown Unknown Unknown
pmemd.cuda.MPI 000000000050B18D Unknown Unknown Unknown
pmemd.cuda.MPI 000000000040A20C Unknown Unknown Unknown
libc.so.6 0000003D7B21EC9D Unknown Unknown Unknown
pmemd.cuda.MPI 000000000040A109 Unknown Unknown Unknown
forrtl: error (69): process interrupted (SIGINT)
Image PC Routine Line Source
libmlx4-m-rdmav2. 00007F4DD7C5F4C0 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000079B0B6 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000076AC22 Unknown Unknown Unknown
pmemd.cuda.MPI 00000000007689BB Unknown Unknown Unknown
pmemd.cuda.MPI 00000000007314ED Unknown Unknown Unknown
pmemd.cuda.MPI 000000000073133A Unknown Unknown Unknown
pmemd.cuda.MPI 0000000000885048 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000088696B Unknown Unknown Unknown
pmemd.cuda.MPI 000000000086A571 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000072F41A Unknown Unknown Unknown
pmemd.cuda.MPI 000000000058DA28 Unknown Unknown Unknown
pmemd.cuda.MPI 000000000058DAE8 Unknown Unknown Unknown
pmemd.cuda.MPI 00000000004955FA Unknown Unknown Unknown
pmemd.cuda.MPI 00000000004BCEFF Unknown Unknown Unknown
pmemd.cuda.MPI 000000000050B18D Unknown Unknown Unknown
pmemd.cuda.MPI 000000000040A20C Unknown Unknown Unknown
libc.so.6 0000003BEA61EC9D Unknown Unknown Unknown
pmemd.cuda.MPI 000000000040A109 Unknown Unknown Unknown
^CCtrl-C caught... cleaning up processes
[proxy:0:1.TX300S7Am]


### mdout (sample: Factor IX NPT) ###

[root.RX350s7ambertest FactorIX_production_NPT]# less mdout_intmva2_gpu4node-4pro_0818-1

          -------------------------------------------------------
          Amber 12 SANDER 2012
          -------------------------------------------------------

| PMEMD implementation of SANDER, Release 12

| Run on 08/16/2013 at 11:01:27

  [-O]verwriting output

File Assignments:
| MDIN: mdin
| MDOUT: mdout_intmva2_gpu4node-4pro_0818-1
| INPCRD: inpcrd
| PARM: prmtop
| RESTRT: restrt
| REFC: refc
| MDVEL: mdvel
| MDEN: mden
| MDCRD: mdcrd
| MDINFO: mdinfo
|LOGFILE: logfile


 Here is the input file:

 Typical Production MD NVT
 &cntrl
  ntx=5, irest=1,
  ntc=2, ntf=2,
  nstlim=10000,
  ntpr=1000, ntwx=1000,
  ntwr=10000,
  dt=0.002, cut=8.,
  ntt=1, tautp=10.0,
  temp0=300.0,
  ntb=2, ntp=1, taup=10.0,
  ioutfm=1,
 /




|--------------------- INFORMATION ----------------------
| GPU (CUDA) Version of PMEMD in use: NVIDIA GPU IN USE.
| Version 12.3
|
| 04/24/2013
|
| Implementation by:
| Ross C. Walker (SDSC)
| Scott Le Grand (nVIDIA)
| Duncan Poole (nVIDIA)
|
| CAUTION: The CUDA code is currently experimental.
| You use it at your own risk. Be sure to
| check ALL results carefully.
|
| Precision model in use:
| [SPFP] - Mixed Single/Double/Fixed Point Precision.
| (Default)
|
|--------------------------------------------------------

|----------------- CITATION INFORMATION -----------------
|
| When publishing work that utilized the CUDA version
| of AMBER, please cite the following in addition to
| the regular AMBER citations:
|
| - Romelia Salomon-Ferrer; Andreas W. Goetz; Duncan
| Poole; Scott Le Grand; Ross C. Walker "Routine
| microsecond molecular dynamics simulations with
| AMBER - Part II: Particle Mesh Ewald", J. Chem.
| Theory Comput., 2012, (In review).
|
| - Andreas W. Goetz; Mark J. Williamson; Dong Xu;
| Duncan Poole; Scott Le Grand; Ross C. Walker
| "Routine microsecond molecular dynamics simulations
| with AMBER - Part I: Generalized Born", J. Chem.
| Theory Comput., 2012, 8 (5), pp1542-1555.
|
| - Scott Le Grand; Andreas W. Goetz; Ross C. Walker
| "SPFP: Speed without compromise - a mixed precision
| model for GPU accelerated molecular dynamics
| simulations.", Comp. Phys. Comm., 2013, 184
| pp374-380, DOI: 10.1016/j.cpc.2012.09.022
|
|--------------------------------------------------------

|------------------- GPU DEVICE INFO --------------------
|
| Task ID: 0
| CUDA Capable Devices Detected: 2
| CUDA Device ID in use: 0
| CUDA Device Name: Tesla K20Xm
| CUDA Device Global Mem Size: 6143 MB
| CUDA Device Num Multiprocessors: 14
| CUDA Device Core Freq: 0.73 GHz
|
|
| Task ID: 1
| CUDA Capable Devices Detected: 2
| CUDA Device ID in use: 1
| CUDA Device Name: Tesla K20Xm
| CUDA Device Global Mem Size: 6143 MB
| CUDA Device Num Multiprocessors: 14
| CUDA Device Core Freq: 0.73 GHz
|
|
| Task ID: 2
| CUDA Capable Devices Detected: 2
| CUDA Device ID in use: 0
| CUDA Device Name: Tesla K20Xm
| CUDA Device Global Mem Size: 6143 MB
| CUDA Device Num Multiprocessors: 14
| CUDA Device Core Freq: 0.73 GHz
|
|
| Task ID: 3
| CUDA Capable Devices Detected: 2
| CUDA Device ID in use: 1
| CUDA Device Name: Tesla K20Xm
| CUDA Device Global Mem Size: 6143 MB
| CUDA Device Num Multiprocessors: 14
| CUDA Device Core Freq: 0.73 GHz
|
|--------------------------------------------------------

| INFO: Axis order optimization will be used.


| Conditional Compilation Defines Used:
| DIRFRC_COMTRANS
| DIRFRC_EFS
| DIRFRC_NOVEC
| MPI
| PUBFFT
| FFTLOADBAL_2PROC

          -------------------------------------------------------
          Amber 12 SANDER 2012
          -------------------------------------------------------

| PMEMD implementation of SANDER, Release 12

| Run on 08/16/2013 at 11:45:26

  [-O]verwriting output

File Assignments:
| MDIN: mdin
| MDOUT: mdout_intmva2_gpu4node-4pro_0818-1
| INPCRD: inpcrd
| PARM: prmtop
| RESTRT: restrt
| REFC: refc
| MDVEL: mdvel
| MDEN: mden
| MDCRD: mdcrd
| MDINFO: mdinfo
|LOGFILE: logfile


 Here is the input file:

 Typical Production MD NVT
 &cntrl
  ntx=5, irest=1,
  ntc=2, ntf=2,
  nstlim=10000,
  ntpr=1000, ntwx=1000,
  ntwr=10000,
  dt=0.002, cut=8.,
  ntt=1, tautp=10.0,
  temp0=300.0,
  ntb=2, ntp=1, taup=10.0,
  ioutfm=1,
 /




|--------------------- INFORMATION ----------------------
| GPU (CUDA) Version of PMEMD in use: NVIDIA GPU IN USE.
| Version 12.3
|
| 04/24/2013
[root.RX350s7ambertest FactorIX_production_NPT]#
[root.RX350s7ambertest FactorIX_production_NPT]# cat mdout_intmva2_gpu4node-4pro_0818-1

          -------------------------------------------------------
          Amber 12 SANDER 2012
          -------------------------------------------------------

| PMEMD implementation of SANDER, Release 12

| Run on 08/16/2013 at 11:45:26

  [-O]verwriting output

File Assignments:
| MDIN: mdin
| MDOUT: mdout_intmva2_gpu4node-4pro_0818-1
| INPCRD: inpcrd
| PARM: prmtop
| RESTRT: restrt
| REFC: refc
| MDVEL: mdvel
| MDEN: mden
| MDCRD: mdcrd
| MDINFO: mdinfo
|LOGFILE: logfile


 Here is the input file:

 Typical Production MD NVT
 &cntrl
  ntx=5, irest=1,
  ntc=2, ntf=2,
  nstlim=10000,
  ntpr=1000, ntwx=1000,
  ntwr=10000,
  dt=0.002, cut=8.,
  ntt=1, tautp=10.0,
  temp0=300.0,
  ntb=2, ntp=1, taup=10.0,
  ioutfm=1,
 /




|--------------------- INFORMATION ----------------------
| GPU (CUDA) Version of PMEMD in use: NVIDIA GPU IN USE.
| Version 12.3
|
| 04/24/2013
|
| Implementation by:
| Ross C. Walker (SDSC)
| Scott Le Grand (nVIDIA)
| Duncan Poole (nVIDIA)
|
| CAUTION: The CUDA code is currently experimental.
| You use it at your own risk. Be sure to
| check ALL results carefully.
|
| Precision model in use:
| [SPFP] - Mixed Single/Double/Fixed Point Precision.
| (Default)
|
|--------------------------------------------------------

|----------------- CITATION INFORMATION -----------------
|
| When publishing work that utilized the CUDA version
| of AMBER, please cite the following in addition to
| the regular AMBER citations:
|
| - Romelia Salomon-Ferrer; Andreas W. Goetz; Duncan
| Poole; Scott Le Grand; Ross C. Walker "Routine
| microsecond molecular dynamics simulations with
| AMBER - Part II: Particle Mesh Ewald", J. Chem.
| Theory Comput., 2012, (In review).
|
| - Andreas W. Goetz; Mark J. Williamson; Dong Xu;
| Duncan Poole; Scott Le Grand; Ross C. Walker
| "Routine microsecond molecular dynamics simulations
| with AMBER - Part I: Generalized Born", J. Chem.
| Theory Comput., 2012, 8 (5), pp1542-1555.
|
| - Scott Le Grand; Andreas W. Goetz; Ross C. Walker
| "SPFP: Speed without compromise - a mixed precision
| model for GPU accelerated molecular dynamics
| simulations.", Comp. Phys. Comm., 2013, 184
| pp374-380, DOI: 10.1016/j.cpc.2012.09.022
|
|--------------------------------------------------------

|------------------- GPU DEVICE INFO --------------------
|
| Task ID: 0
| CUDA Capable Devices Detected: 2
| CUDA Device ID in use: 0
| CUDA Device Name: Tesla K20Xm
| CUDA Device Global Mem Size: 6143 MB
| CUDA Device Num Multiprocessors: 14
| CUDA Device Core Freq: 0.73 GHz
|
|
| Task ID: 1
| CUDA Capable Devices Detected: 2
| CUDA Device ID in use: 1
| CUDA Device Name: Tesla K20Xm
| CUDA Device Global Mem Size: 6143 MB
| CUDA Device Num Multiprocessors: 14
| CUDA Device Core Freq: 0.73 GHz
|
|
| Task ID: 2
| CUDA Capable Devices Detected: 2
| CUDA Device ID in use: 0
| CUDA Device Name: Tesla K20Xm
| CUDA Device Global Mem Size: 6143 MB
| CUDA Device Num Multiprocessors: 14
| CUDA Device Core Freq: 0.73 GHz
|
|
| Task ID: 3
| CUDA Capable Devices Detected: 2
| CUDA Device ID in use: 1
| CUDA Device Name: Tesla K20Xm
| CUDA Device Global Mem Size: 6143 MB
| CUDA Device Num Multiprocessors: 14
| CUDA Device Core Freq: 0.73 GHz
|
|--------------------------------------------------------

| INFO: Axis order optimization will be used.


| Conditional Compilation Defines Used:
| DIRFRC_COMTRANS
| DIRFRC_EFS
| DIRFRC_NOVEC
| MPI
| PUBFFT
| FFTLOADBAL_2PROC
| BINTRAJ
| MKL
| CUDA

| Largest sphere to fit in unit cell has radius = 39.339

| INFO: Old style PARM file read


| Note: 1-4 EEL scale factors were NOT found in the topology file.
| Using default value of 1.2.

| Note: 1-4 VDW scale factors were NOT found in the topology file.
| Using default value of 2.0.
| Duplicated 437 dihedrals

| Duplicated 1846 dihedrals

--------------------------------------------------------------------------------
   1. RESOURCE USE:
--------------------------------------------------------------------------------

 getting new box info from bottom of inpcrd

 NATOM = 90906 NTYPES = 19 NBONH = 87891 MBONA = 3077
 NTHETH = 6433 MTHETA = 4178 NPHIH = 11305 MPHIA = 5519
 NHPARM = 0 NPARM = 0 NNB = 145596 NRES = 28750
 NBONA = 3077 NTHETA = 4178 NPHIA = 5519 NUMBND = 54
 NUMANG = 126 NPTRA = 75 NATYP = 31 NPHB = 1
 IFBOX = 1 NMXRS = 24 IFCAP = 0 NEXTRA = 0
 NCOPY = 0

| Coordinate Index Table dimensions: 28 16 15
| Direct force subcell size = 5.0745 5.2086 5.2452

     BOX TYPE: RECTILINEAR

--------------------------------------------------------------------------------
   2. CONTROL DATA FOR THE RUN
--------------------------------------------------------------------------------

factor IX (ACTIVATED PROTEIN)

General flags:
     imin = 0, nmropt = 0

Nature and format of input:
     ntx = 5, irest = 1, ntrx = 1

Nature and format of output:
     ntxo = 1, ntpr = 1000, ntrx = 1, ntwr = 10000
     iwrap = 0, ntwx = 1000, ntwv = 0, ntwe = 0
     ioutfm = 1, ntwprt = 0, idecomp = 0, rbornstat= 0

Potential function:
     ntf = 2, ntb = 2, igb = 0, nsnb = 25
     ipol = 0, gbsa = 0, iesp = 0
     dielc = 1.00000, cut = 8.00000, intdiel = 1.00000

Frozen or restrained atoms:
     ibelly = 0, ntr = 0

Molecular dynamics:
     nstlim = 10000, nscm = 1000, nrespa = 1
     t = 0.00000, dt = 0.00200, vlimit = -1.00000

Berendsen (weak-coupling) temperature regulation:
     temp0 = 300.00000, tempi = 0.00000, tautp = 10.00000

Pressure regulation:
     ntp = 1
     pres0 = 1.00000, comp = 44.60000, taup = 10.00000

SHAKE:
     ntc = 2, jfastw = 0
     tol = 0.00001

| Intermolecular bonds treatment:
| no_intermolecular_bonds = 1

| Energy averages sample interval:
| ene_avg_sampling = 1000

Ewald parameters:
     verbose = 0, ew_type = 0, nbflag = 1, use_pme = 1
     vdwmeth = 1, eedmeth = 1, netfrc = 1
     Box X = 142.086 Box Y = 83.337 Box Z = 78.678
     Alpha = 90.000 Beta = 90.000 Gamma = 90.000
     NFFT1 = 144 NFFT2 = 84 NFFT3 = 80
     Cutoff= 8.000 Tol =0.100E-04
     Ewald Coefficient = 0.34864
     Interpolation order = 4

| PMEMD ewald parallel performance parameters:
| block_fft = 0
| fft_blk_y_divisor = 2
| excl_recip = 0
| excl_master = 0
| atm_redist_freq = 320

--------------------------------------------------------------------------------
   3. ATOMIC COORDINATES AND VELOCITIES
--------------------------------------------------------------------------------

factor IX (ACTIVATED PROTEIN)
 begin time read from input coords = 2542.675 ps


 Number of triangulated 3-point waters found: 28358

     Sum of charges from parm topology file = 0.00031225
     Forcing neutrality...

| Dynamic Memory, Types Used:
| Reals 3876951
| Integers 3639776

| Nonbonded Pairs Initial Allocation: 5181642

| GPU memory information (estimate):
| KB of GPU memory in use: 96029
| KB of CPU memory in use: 96029

| Running AMBER/MPI version on 4 nodes


--------------------------------------------------------------------------------
   4. RESULTS
--------------------------------------------------------------------------------

 ---------------------------------------------------
 APPROXIMATING switch and d/dx switch using CUBIC SPLINE INTERPOLATION
 using 5000.0 points per unit in tabled values
 TESTING RELATIVE ERROR over r ranging from 0.0 to cutoff
| CHECK switch(x): max rel err = 0.2738E-14 at 2.422500
| CHECK d/dx switch(x): max rel err = 0.8332E-11 at 2.782960
 ---------------------------------------------------
|---------------------------------------------------
| APPROXIMATING direct energy using CUBIC SPLINE INTERPOLATION
| with 50.0 points per unit in tabled values
| Relative Error Limit not exceeded for r .gt. 2.47
| APPROXIMATING direct force using CUBIC SPLINE INTERPOLATION
| with 50.0 points per unit in tabled values
| Relative Error Limit not exceeded for r .gt. 2.89
|---------------------------------------------------
check COM velocity, temp: 0.000018 0.00(Removed)

 NSTEP = 1000 TIME(PS) = 2544.675 TEMP(K) = 296.90 PRESS = -694.5
 Etot = -233748.6862 EKtot = 54523.3398 EPtot = -288272.0261
 BOND = 1137.6172 ANGLE = 2934.5131 DIHED = 2246.0865
 1-4 NB = 1288.3081 1-4 EEL = 15112.1345 VDWAALS = 36457.3797
 EELEC = -347448.0652 EHBOND = 0.0000 RESTRAINT = 0.0000
 EKCMT = 25361.2361 VIRIAL = 39519.1411 VOLUME = 944163.2170
                                                    Density = 0.9747
 ------------------------------------------------------------------------------

check COM velocity, temp: 0.000027 0.00(Removed)

 NSTEP = 2000 TIME(PS) = 2546.675 TEMP(K) = 295.89 PRESS = -968.7
 Etot = -233339.5533 EKtot = 54337.9648 EPtot = -287677.5181
 BOND = 1127.3253 ANGLE = 2924.9643 DIHED = 2215.0624
 1-4 NB = 1298.3079 1-4 EEL = 15114.4109 VDWAALS = 35626.9347
 EELEC = -345984.5237 EHBOND = 0.0000 RESTRAINT = 0.0000
 EKCMT = 25131.5725 VIRIAL = 45093.5005 VOLUME = 954386.4124
                                                    Density = 0.9643
 ------------------------------------------------------------------------------



### "nvidia-smi -l 1" log (node 0): Job doesn't finished but ###

Fri Aug 16 11:01:27 2013 *** test start
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 31C P8 23W / 235W | 14MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 32C P8 19W / 235W | 14MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| No running compute processes found |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:28 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 31C P0 22W / 235W | 84MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 32C P0 21W / 235W | 84MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 68MB |
| 1 30845 pmemd.cuda.MPI 68MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:29 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 32C P0 62W / 235W | 197MB / 6143MB | 54% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 33C P0 61W / 235W | 240MB / 6143MB | 62% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 180MB |
| 1 30845 pmemd.cuda.MPI 224MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:30 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 33C P0 88W / 235W | 197MB / 6143MB | 54% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 34C P0 87W / 235W | 240MB / 6143MB | 60% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 180MB |
| 1 30845 pmemd.cuda.MPI 224MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:31 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 34C P0 96W / 235W | 197MB / 6143MB | 52% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 34C P0 95W / 235W | 240MB / 6143MB | 61% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 180MB |
| 1 30845 pmemd.cuda.MPI 224MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:32 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 34C P0 98W / 235W | 197MB / 6143MB | 53% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 34C P0 97W / 235W | 240MB / 6143MB | 61% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 180MB |
| 1 30845 pmemd.cuda.MPI 224MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:33 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 34C P0 99W / 235W | 197MB / 6143MB | 53% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 35C P0 98W / 235W | 240MB / 6143MB | 61% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 180MB |
| 1 30845 pmemd.cuda.MPI 224MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:34 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 34C P0 100W / 235W | 197MB / 6143MB | 54% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 35C P0 98W / 235W | 240MB / 6143MB | 60% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 180MB |
| 1 30845 pmemd.cuda.MPI 224MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:35 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 35C P0 99W / 235W | 197MB / 6143MB | 53% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 35C P0 98W / 235W | 240MB / 6143MB | 61% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 180MB |
| 1 30845 pmemd.cuda.MPI 224MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:36 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 34C P0 99W / 235W | 197MB / 6143MB | 53% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 35C P0 98W / 235W | 240MB / 6143MB | 61% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 180MB |
| 1 30845 pmemd.cuda.MPI 224MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:37 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 35C P0 100W / 235W | 197MB / 6143MB | 54% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 35C P0 98W / 235W | 240MB / 6143MB | 62% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 180MB |
| 1 30845 pmemd.cuda.MPI 224MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:38 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 35C P0 100W / 235W | 197MB / 6143MB | 53% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 35C P0 98W / 235W | 240MB / 6143MB | 60% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 180MB |
| 1 30845 pmemd.cuda.MPI 224MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:39 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 35C P0 99W / 235W | 197MB / 6143MB | 53% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 35C P0 98W / 235W | 240MB / 6143MB | 60% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 180MB |
| 1 30845 pmemd.cuda.MPI 224MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:40 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 35C P0 99W / 235W | 197MB / 6143MB | 53% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 35C P0 98W / 235W | 240MB / 6143MB | 60% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 180MB |
| 1 30845 pmemd.cuda.MPI 224MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:41 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 35C P0 99W / 235W | 197MB / 6143MB | 53% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 36C P0 98W / 235W | 240MB / 6143MB | 60% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 180MB |
| 1 30845 pmemd.cuda.MPI 224MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:42 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 35C P0 99W / 235W | 197MB / 6143MB | 53% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 36C P0 98W / 235W | 240MB / 6143MB | 60% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 180MB |
| 1 30845 pmemd.cuda.MPI 224MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:43 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 35C P0 100W / 235W | 197MB / 6143MB | 53% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 36C P0 98W / 235W | 240MB / 6143MB | 60% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 180MB |
| 1 30845 pmemd.cuda.MPI 224MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:44 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 35C P0 99W / 235W | 197MB / 6143MB | 54% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 36C P0 98W / 235W | 240MB / 6143MB | 60% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 180MB |
| 1 30845 pmemd.cuda.MPI 224MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:45 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 35C P0 99W / 235W | 197MB / 6143MB | 53% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 36C P0 98W / 235W | 240MB / 6143MB | 60% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 180MB |
| 1 30845 pmemd.cuda.MPI 224MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:46 2013 *** GPU stop and don't work any more
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 34C P0 69W / 235W | 197MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 35C P0 68W / 235W | 240MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 180MB |
| 1 30845 pmemd.cuda.MPI 224MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:47 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 34C P0 59W / 235W | 197MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 34C P0 59W / 235W | 240MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 180MB |
| 1 30845 pmemd.cuda.MPI 224MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:48 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 34C P0 57W / 235W | 197MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 34C P0 56W / 235W | 240MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 180MB |
| 1 30845 pmemd.cuda.MPI 224MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:49 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 34C P0 56W / 235W | 197MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 34C P0 55W / 235W | 240MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 180MB |
| 1 30845 pmemd.cuda.MPI 224MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:50 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 34C P0 55W / 235W | 197MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 34C P0 55W / 235W | 240MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 180MB |
| 1 30845 pmemd.cuda.MPI 224MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:51 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 33C P0 55W / 235W | 197MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 34C P0 55W / 235W | 240MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 30844 pmemd.cuda.MPI 180MB |
| 1 30845 pmemd.cuda.MPI 224MB |
+-----------------------------------------------------------------------------+
^C[root.RX350s7ambertest ~]#




### "nvidia-smi -l 1" log (node 1) ###

Fri Aug 16 11:01:26 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 29C P8 28W / 235W | 14MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 31C P8 23W / 235W | 14MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| No running compute processes found |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:27 2013 *** test start
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 29C P0 21W / 235W | 84MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 31C P0 21W / 235W | 84MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 32138 pmemd.cuda.MPI 68MB |
| 1 32139 pmemd.cuda.MPI 68MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:28 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 30C P0 55W / 235W | 239MB / 6143MB | 54% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 32C P0 57W / 235W | 239MB / 6143MB | 32% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 32138 pmemd.cuda.MPI 223MB |
| 1 32139 pmemd.cuda.MPI 223MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:29 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 31C P0 85W / 235W | 239MB / 6143MB | 59% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 32C P0 87W / 235W | 239MB / 6143MB | 59% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 32138 pmemd.cuda.MPI 223MB |
| 1 32139 pmemd.cuda.MPI 223MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:30 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 31C P0 94W / 235W | 239MB / 6143MB | 60% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 33C P0 96W / 235W | 239MB / 6143MB | 60% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 32138 pmemd.cuda.MPI 223MB |
| 1 32139 pmemd.cuda.MPI 223MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:31 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 32C P0 97W / 235W | 239MB / 6143MB | 59% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 33C P0 98W / 235W | 239MB / 6143MB | 59% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 32138 pmemd.cuda.MPI 223MB |
| 1 32139 pmemd.cuda.MPI 223MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:32 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 32C P0 98W / 235W | 239MB / 6143MB | 61% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 34C P0 99W / 235W | 239MB / 6143MB | 59% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 32138 pmemd.cuda.MPI 223MB |
| 1 32139 pmemd.cuda.MPI 223MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:33 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 32C P0 98W / 235W | 239MB / 6143MB | 59% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 34C P0 100W / 235W | 239MB / 6143MB | 60% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 32138 pmemd.cuda.MPI 223MB |
| 1 32139 pmemd.cuda.MPI 223MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:34 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 32C P0 98W / 235W | 239MB / 6143MB | 59% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 34C P0 99W / 235W | 239MB / 6143MB | 58% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 32138 pmemd.cuda.MPI 223MB |
| 1 32139 pmemd.cuda.MPI 223MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:35 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 32C P0 98W / 235W | 239MB / 6143MB | 59% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 34C P0 100W / 235W | 239MB / 6143MB | 60% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 32138 pmemd.cuda.MPI 223MB |
| 1 32139 pmemd.cuda.MPI 223MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:36 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 33C P0 98W / 235W | 239MB / 6143MB | 60% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 34C P0 100W / 235W | 239MB / 6143MB | 59% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 32138 pmemd.cuda.MPI 223MB |
| 1 32139 pmemd.cuda.MPI 223MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:37 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 32C P0 98W / 235W | 239MB / 6143MB | 58% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 34C P0 100W / 235W | 239MB / 6143MB | 58% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 32138 pmemd.cuda.MPI 223MB |
| 1 32139 pmemd.cuda.MPI 223MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:38 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 33C P0 98W / 235W | 239MB / 6143MB | 58% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 34C P0 100W / 235W | 239MB / 6143MB | 59% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 32138 pmemd.cuda.MPI 223MB |
| 1 32139 pmemd.cuda.MPI 223MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:39 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 33C P0 97W / 235W | 239MB / 6143MB | 55% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 34C P0 99W / 235W | 239MB / 6143MB | 57% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 32138 pmemd.cuda.MPI 223MB |
| 1 32139 pmemd.cuda.MPI 223MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:40 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 33C P0 97W / 235W | 239MB / 6143MB | 59% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 34C P0 100W / 235W | 239MB / 6143MB | 59% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 32138 pmemd.cuda.MPI 223MB |
| 1 32139 pmemd.cuda.MPI 223MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:41 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 33C P0 97W / 235W | 239MB / 6143MB | 58% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 35C P0 100W / 235W | 239MB / 6143MB | 59% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 32138 pmemd.cuda.MPI 223MB |
| 1 32139 pmemd.cuda.MPI 223MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:42 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 33C P0 97W / 235W | 239MB / 6143MB | 58% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 35C P0 100W / 235W | 239MB / 6143MB | 59% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 32138 pmemd.cuda.MPI 223MB |
| 1 32139 pmemd.cuda.MPI 223MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:43 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 33C P0 98W / 235W | 239MB / 6143MB | 58% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 35C P0 100W / 235W | 239MB / 6143MB | 59% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 32138 pmemd.cuda.MPI 223MB |
| 1 32139 pmemd.cuda.MPI 223MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:44 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 33C P0 98W / 235W | 239MB / 6143MB | 60% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 35C P0 100W / 235W | 239MB / 6143MB | 59% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 32138 pmemd.cuda.MPI 223MB |
| 1 32139 pmemd.cuda.MPI 223MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:45 2013 *** GPU stop and don't work any more
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 32C P0 71W / 235W | 239MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 34C P0 72W / 235W | 239MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 32138 pmemd.cuda.MPI 223MB |
| 1 32139 pmemd.cuda.MPI 223MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:46 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 32C P0 60W / 235W | 239MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 33C P0 61W / 235W | 239MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 32138 pmemd.cuda.MPI 223MB |
| 1 32139 pmemd.cuda.MPI 223MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:47 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 32C P0 56W / 235W | 239MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 33C P0 58W / 235W | 239MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 32138 pmemd.cuda.MPI 223MB |
| 1 32139 pmemd.cuda.MPI 223MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:48 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 32C P0 55W / 235W | 239MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 33C P0 57W / 235W | 239MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 32138 pmemd.cuda.MPI 223MB |
| 1 32139 pmemd.cuda.MPI 223MB |
+-----------------------------------------------------------------------------+
Fri Aug 16 11:01:49 2013
+------------------------------------------------------+
| NVIDIA-SMI 5.319.32 Driver Version: 319.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20Xm Off | 0000:84:00.0 Off | Off |
| N/A 31C P0 55W / 235W | 239MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20Xm Off | 0000:85:00.0 Off | Off |
| N/A 33C P0 56W / 235W | 239MB / 6143MB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 32138 pmemd.cuda.MPI 223MB |
| 1 32139 pmemd.cuda.MPI 223MB |
+-----------------------------------------------------------------------------+
^C[root.TX300S7Am ~]#
===========================

Thank you for your support,
Best wishes,

Y. Nakashima

----
-----------------------------------------
Yoshihisa Nakashima
Development Department.I
System Development Devision.II
(Former PC Cluster Development Dev.)
XTC Development Unit
Fujitsu Ltd.
Tel: +81-44-754-3174
EXT:7103-5703
E-mail:(nakashima_y.jp.fujitsu.com)

_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber

Received on Fri Aug 16 2013 - 01:30:04 PDT