Hello, dear Amber authors/users!
I am performing TMD and have amber version 14. When I use a single
node/32 processors for performing TMD simulation, simulations speed up
to 3ns/day, whereas any increase in the nodes/processors slows down the
simulation to 0.2ns/day. As I am running TMD, I cant use pmemd. I am
using the following shell script to run sander parallel version.
#!/bin/bash
#PBS -N Tmd_10ns_0.5force
#PBS -S /bin/bash
#PBS -l nodes=4:ppn=32
#PBS -l walltime=500:00:00
# Script for running Amber 14 pmemd.MPI (Open MPI) on Parallel
export AMBERHOME=/softwares/amber14
source $AMBERHOME/amber.sh
# Initialize the CUDA run-time environment for GPU-based calculations
export CUDA_HOME=/usr/local/cuda-6.5
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CUDA_HOME/lib:$CUDA_HOME/lib64
# mpi libraries using this amber was compiled
#export PATH=/usr/mpi/gcc/openmpi-1.8.8/bin:$PATH
export PATH=/opt/openmpi-torque.mellanox/1.8.2/bin/:$PATH
#export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/mpi/gcc/openmpi-1.8.8/lib
export
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/openmpi-torque.mellanox/1.8.2/lib
# The environment variable $PBS_O_WORKDIR makes it easy to return to the
directory from which you submitted the job
cd $PBS_O_WORKDIR
echo "Current working directory is `pwd`"
echo "Starting run at: `date`"
##################### CHANGE THE SETTINGS HERE
###########################################
export CUDA_VISIBLE_DEVICES="0"
echo "production"
mpirun -np 128 sander.MPI -O -i tgtm.in -c open_3md.rst -p open.prmtop
-ref close_3md.rst -r tmd1.rst -x tmd1.mdcrd -o tmd1.out -inf tmd1.info
echo "Finished run at: `date`"
#########################################################################################
lscpu in my cluster(has 64 nodes each with 2 gpus and 32 cores) shows,
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Thread(s) per core: 1
Core(s) per socket: 16
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Stepping: 2
CPU MHz: 1200.000
BogoMIPS: 4589.24
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 40960K
NUMA node0 CPU(s): 0-15
NUMA node1 CPU(s): 16-31
My TMD input file is:
Ubi targeted MD
&cntrl
imin = 0, nstlim = 5000000, dt = 0.002,
ntx = 1, ntt = 3, gamma_ln = 1.0,
ntc = 2, ntf = 2, ntb = 2, ntp=1,
cut = 10.0, nscm = 0,
ntpr = 1000, ntwx = 1000, ntwr = 1000,
igb = 0, irest=0, nmropt=1,tempi=300.0,
itgtmd=1, tgtmdfrc=1.0,
tgtfitmask=":77-222.CA,N,C,O",
tgtrmsmask=":1-70.CA,N,C,O",
/
&wt
TYPE='TGTRMSD', istep1 =1, istep2 = 400000,
value1 = 37.417, value2 = 35.0,
/
&wt
TYPE='TGTRMSD', istep1 =400000, istep2 = 600000,
value1 = 35.0, value2 = 35.0,
/
&wt
TYPE='TGTRMSD', istep1 =600000, istep2 = 1000000,
value1 = 35.0, value2 = 30.0,
/
&wt
TYPE='TGTRMSD', istep1 =1000000, istep2 = 1200000,
value1 = 30.0, value2 = 30.0,
/
&wt
TYPE='TGTRMSD', istep1 =1200000, istep2 = 1600000,
value1 = 30.0, value2 = 25.0,
/
&wt
TYPE='TGTRMSD', istep1 =1600000, istep2 = 1800000,
value1 = 25.0, value2 = 25.0,
/
&wt
TYPE='TGTRMSD', istep1 =1800000, istep2 = 2200000,
value1 = 25.0, value2 = 20.0,
/
&wt
TYPE='TGTRMSD', istep1 =2200000, istep2 = 2400000,
value1 = 20.0, value2 = 20.0,
/
&wt
TYPE='TGTRMSD', istep1 =2400000, istep2 = 2800000,
value1 = 20.0, value2 = 15.0,
/
&wt
TYPE='TGTRMSD', istep1 =2800000, istep2 = 3000000,
value1 = 15.0, value2 = 15.0,
/
&wt
TYPE='TGTRMSD', istep1 =3000000, istep2 = 3400000,
value1 = 15.0, value2 = 10.0,
/
&wt
TYPE='TGTRMSD', istep1 =3400000, istep2 = 3600000,
value1 = 10.0, value2 = 10.0,
/
&wt
TYPE='TGTRMSD', istep1 =3600000, istep2 = 4000000,
value1 = 10.0, value2 = 5.0,
/
&wt
TYPE='TGTRMSD', istep1 =4000000, istep2 = 4200000,
value1 = 5.0, value2 = 5.0,
/
&wt
TYPE='TGTRMSD', istep1 =4200000, istep2 = 4600000,
value1 = 5.0, value2 = 0.0,
/
&wt
TYPE='TGTRMSD', istep1 =4600000, istep2 = 5000000,
value1 = 0.0, value2 = 0.0,
/
&wt
type="END",
/
Any suggetions to speed up the simulations?
Any help is appreciated and thanks in advance!!
---------------------------------------------------------
Aravind R
National Center for biological sciences
Banglore, India
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Wed Apr 18 2018 - 00:00:03 PDT
