Custom Search
-- --------------------------------------------------------------------------------------- Christina Bergonzo Postdoctoral Researcher Department of Medicinal Chemistry, University of Utah http://home.chpc.utah.edu/~cheatham/ --------------------------------------------------------------------------------------- ------------------------------ Message: 14 Date: Thu, 5 Jan 2017 08:30:50 -0500 From: Daniel Roe <daniel.r.roe.gmail.com> Subject: Re: [AMBER] cpptraj clustering doubt To: AMBER Mailing List <amber.ambermd.org> Message-ID: <CAAC0qOZVuw5H+5TF5_n3eP7VjmbySAns9Qd3Zz6ui6R0YoVwDg.mail.gmail.com> Content-Type: text/plain; charset=UTF-8 See also the clustering script in part 7 of the supporting info here: http://pubs.acs.org/doi/abs/10.1021/jp4125099 -Dan On Thu, Jan 5, 2017 at 7:48 AM, Christina Bergonzo <cbergonzo.gmail.com> wrote: > Hi, > > You can take a look at the supporting information of this paper: > Highly sampled tetranucleotide and tetraloop motifs enable evaluation of > common RNA force fields > <https://scholar.google.com/citations?view_op=view_citation&hl=en&user=d4JoelsAAAAJ&citation_for_view=d4JoelsAAAAJ:In6cVmBjs0IC> > C Bergonzo, NM Henriksen, DR Roe, TE Cheatham > RNA 21 (9), 1578-1590 > > http://rnajournal.cshlp.org/content/suppl/2015/06/19/rna.051102.115.DC1/SuppMaterial.pdf > > There is a CPPTRAJ command which performs combined clustering on 8 > trajectories. > > Hope this helps, > Christina > > On Thu, Jan 5, 2017 at 5:59 AM, Mary Varughese <maryvj1985.gmail.com> wrote: > >> Sir >> >> >> in combined clustering, could i compare more than two trajectories? If >> so is there any difference in the command option >> split ..Frameno... >> >> Also in split_summary.dat is there any useful information. >> >> how could i infer the convergence of the trajectory DBI , pSF >> >> Please give me some idea >> >> >> Thanking you >> >> >> mary varughese >> >> _______________________________________________ >> AMBER mailing list >> AMBER.ambermd.org >> http://lists.ambermd.org/mailman/listinfo/amber >> > > > > -- > --------------------------------------------------------------------------------------- > Christina Bergonzo > Postdoctoral Researcher > Department of Medicinal Chemistry, University of Utah > http://home.chpc.utah.edu/~cheatham/ > --------------------------------------------------------------------------------------- > _______________________________________________ > AMBER mailing list > AMBER.ambermd.org > http://lists.ambermd.org/mailman/listinfo/amber -- ------------------------- Daniel R. Roe Laboratory of Computational Biology National Institutes of Health, NHLBI 5635 Fishers Ln, Rm T900 Rockville MD, 20852 https://www.lobos.nih.gov/lcb ------------------------------ Message: 15 Date: Thu, 5 Jan 2017 08:52:17 -0500 From: Daniel Roe <daniel.r.roe.gmail.com> Subject: Re: [AMBER] 'Diffusion' calculation in Ptraj - could not understand a description of Ptraj code in Amber14 To: AMBER Mailing List <amber.ambermd.org> Message-ID: <CAAC0qOY9H=1AkCJ1oV3GmfViZGJWXsgU-GL1YcvoY3+9j6q-oA.mail.gmail.com> Content-Type: text/plain; charset=UTF-8 Hi, For some reason your message came in all in a single line (i.e. there were no newlines) which made reading your code a little difficult, but I've spotted some problems: On Thu, Jan 5, 2017 at 7:39 AM, wei <lwstudy.sina.com> wrote: > Dear all, > I want to verify that the translational diffusion constant I calculated for a sugar molecule is correct. I found that the magnitudes of mean square displacement (MSD) for one atom I computed using Ptraj/Cpptraj is fairly different from what I obtained from a small code I wrote (I just read in coordinates for all atoms directly then calculate MSDs as follows: > open(11,file='image.crd') > do > read(11,*) > read(11,'(10f8.3)') (x(i),y(i),z(i),i=1,natom) > nframe=nframe+1 > cx(nframe)=x(1) > cy(nframe)=y(1) > cz(nframe)=z(1) > enddo > close(11) This code is incorrect for reading Amber ASCII trajectories. The title line only appears once (see http://ambermd.org/formats.html#trajectory). You need to move the 'read(11,*)' statement to outside the do loop. Also, if your trajectory has any box coordinates they need to be read after the coordinates. A good check to see if your code is doing the right thing is to try and write the data back out and see if you can visualize it. > do t=0,2000 > do i=1,nframe-t > dvx(t+1)=dvx(t+1)+(cx(i+t)-cx(i))**2 > dvy(t+1)=dvy(t+1)+(cy(i+t)-cy(i))**2 > dvz(t+1)=dvz(t+1)+(cz(i+t)-cz(i))**2 > enddo > dvx(t+1)=dvx(t+1)/real(nframe-t) > dvy(t+1)=dvy(t+1)/real(nframe-t) > dvz(t+1)=dvz(t+1)/real(nframe-t) >enddo). This isn't the right calculation for mean-squared displacement. You want the average of the delta of each from from the *initial* frame because you want to know how far a particle moved (i.e. was displaced) from it's initial position, not how far it is moving each frame on average (see also https://en.wikipedia.org/wiki/Mean_squared_displacement). It looks like you're doing some kind of correlation calculation above. > Ptraj input was wrote as: > diffusion :1.C1 2 average output > Then I got (MSD for a specific t):Mine code (239.17)Ptraj (936.91) > Thus I am reading the code related to 'diffusion' command in Ptraj (please find it in AMBER14, 'actions.c'). I feel confused while seeing these lines of comments: > /* * if the particle moved more than half the box, assume * it was imaged and adjust the distance of the total * movement with respect to the original frame... */ if ( state->box[0] > 0.0 ) { if ( delx > state->box[0]/2.0 ) diffusionInfo->deltax[currentAtom] -= state->box[0]; else if ( delx < -state->box[0]/2.0 ) diffusionInfo->deltax[currentAtom] += state->box[0]; if ( dely > state->box[1]/2.0 ) diffusionInfo->deltay[currentAtom] -= state->box[1]; else if ( dely < -state->box[1]/2.0 ) diffusionInfo->deltay[currentAtom] += state->box[1]; if ( delz > state->box[2]/2.0 ) diffusionInfo->deltaz[currentAtom] -= state->box[2]; else if ( delz < -state->box[2]/2.0 ) diffusionInfo->deltaz[currentAtom] += state->box[2]; } This is the code in ptraj that attempts to account for imaging in orthorhombic cells. As I wrote previously, if you don't 'unwrap' an imaged trajectory prior to the diffusion calculation, you have to correct for imaging "jumps", otherwise you get the wrong value for diffusion. Cpptraj does this for both orthorhombic and non-orthorhombic cells. Hope this helps, -Dan > I am just trying to find the reason why I did not compute the same MSDs as Ptraj. I have two simple question:1. How to understand this description--"adjust the distance of the total movement with respect to the original frame"? Does 'total movement' refer to external degrees of freedom of the simulated box?2. Ptraj images atoms for all frames or just for those frames whose particle moved more than half the box? > Any help would be highly appreciated and thanks in advance! > Regards,Liu Wei-------------------------------- > > _______________________________________________ > AMBER mailing list > AMBER.ambermd.org > http://lists.ambermd.org/mailman/listinfo/amber -- ------------------------- Daniel R. Roe Laboratory of Computational Biology National Institutes of Health, NHLBI 5635 Fishers Ln, Rm T900 Rockville MD, 20852 https://www.lobos.nih.gov/lcb ------------------------------ Message: 16 Date: Thu, 5 Jan 2017 09:14:14 -0500 From: Daniel Roe <daniel.r.roe.gmail.com> Subject: Re: [AMBER] Problems about membrane protein simulation To: AMBER Mailing List <amber.ambermd.org> Message-ID: <CAAC0qOYS2qCqA_BD_HdicNruxes6Qei9NDxuUROWbM=ODScKAQ.mail.gmail.com> Content-Type: text/plain; charset=UTF-8 Hi, On Thu, Jan 5, 2017 at 3:14 AM, ?? <wumeng.shanghaitech.edu.cn> wrote: > Hi, > I have followed http://archive.ambermd.org/201603/0160.html to use cpptraj to convert the .rst, but the last step 'trajout ###.pdb' kept in 'Writing ### as PDB' state and reported without any errors. Do you mean cpptraj was hanging and not writing the PDB? Did it exit with an error? Were you running interactively and didn't actually type 'run'. Could you provide some more details? Does something like 'cpptraj -p <topology file> -y heat.rst -x heat.pdb' not work? I suspect your original issue is that the default restart format for Amber16 is NetCDF which VMD does not yet read (although I think support is coming soon). > Actually, it is my first time to use amber to simulate membrane protein, so besides the above problem, I have two more questions that confused me so much, Q1: In my knowledge, the membrane proteins simulate with the charmm force field commonly, I am not sure whether the amber force field is suitable for membrane proteins or not, and need I add the charmm force field to amber? See section '3.4 Lipids' in the Amber 16 manual and references therein, particularly http://pubs.acs.org/doi/abs/10.1021/ct4010307. > If I need, how to do it? Q2: Go back to the original problem, I put restraints on protein and POPC molecules at the same time in 'Heat.in', > >> restraint_wt=10.0 >> restraintmask=':1-486 & !.H=', > > is it acceptable to do so? My thought is to restrain the protein and membrane in the whole heating process(NVT), and then release it by decreasing restraints in the early period of the production MD process(NPT), so if this practice is right or not for the membrane protein? Any suggestions would be greatly appreciated. Thank You Very Much! That restraintmask looks fine, but 10.0 seems like quite a high force constant. I would start at no more than 5. Also, you probably want to gradually reduce the restraint weight in successive runs until you can eliminate it. -Dan > > Sincerely, > Wu Meng > > ########################################################### > Message: 21 > Date: Wed, 4 Jan 2017 17:13:16 +0200 > From: Huang Jing <jing.huang8911.gmail.com> > Subject: Re: [AMBER] Problems about membrane protein simulation > To: AMBER Mailing List <amber.ambermd.org> > Message-ID: > <CAMvERa=tBi4OQN8rDWcULrV6qGXvB1OD+Bn8BCb_73y_wLHD7A.mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > http://archive.ambermd.org/201603/0160.html > this mail could be useful to solve your problem; > > Jing > > On Wed, Jan 4, 2017 at 4:44 PM, Shanghaitech University WuMeng < > wumeng.shanghaitech.edu.cn> wrote: > >> Dear All, >> I am working on a membrane protein simulation. I have referred to "An >> Amber Lipid Force Field Tutorial:Lipid14 Edition"(http://ambermd.org/ >> tutorials/advanced/tutorial16/), the first 'minimization' for my system I >> run is OK, and the next 'Heating' also run successfully, but when I open >> 'Heat.rst' in VMD, nothing displayed in the VMD window, then I tried to use >> ambpdb to transform '.rst' to '.pdb'?there came an error:'Could not read >> restart atoms/time'.(min.rst is OK).It seems there are some mistakes in the >> 'Heat.rst', here is the 'Heat.in': >> #################### >> Heating 100K >> &cntrl >> imin=0, >> ntx=1, >> irest=0, >> ntc=2, >> ntf=2, >> tol=0.0000001, >> nstlim=2500, >> ntt=3, >> gamma_ln=1.0, >> ntr=1, >> ig=-1, >> ntpr=100, >> ntwr=10000, >> ntwx=100, >> dt=0.002, >> nmropt=1, >> ntb=1, >> ntp=0, >> cut=10.0, >> ioutfm=1, >> ntxo=2, >> restraint_wt=10.0 >> restraintmask=':1-486 & !.H=', >> / >> &wt >> type='TEMP0', >> istep1=0, >> istep2=2500, >> value1=0.0, >> value2=100.0 / >> &wt type='END' / >> ###################### >> I add restraints on protein and POPC molecules(RES 1-486), I don't know >> if it is right or not, this is the first time I use amber to simulate >> membrane protein, so maybe somewhere I missed or did in the wrong way. Any >> suggestions or reference resources about membrane protein simulation would >> be greatly appreciated. Thank You Very Much! >> >> >> Sincerely, >> Wu Meng >> _______________________________________________ >> AMBER mailing list >> AMBER.ambermd.org >> http://lists.ambermd.org/mailman/listinfo/amber > _______________________________________________ > AMBER mailing list > AMBER.ambermd.org > http://lists.ambermd.org/mailman/listinfo/amber -- ------------------------- Daniel R. Roe Laboratory of Computational Biology National Institutes of Health, NHLBI 5635 Fishers Ln, Rm T900 Rockville MD, 20852 https://www.lobos.nih.gov/lcb ------------------------------ Message: 17 Date: Thu, 5 Jan 2017 10:09:50 -0500 From: Adrian Roitberg <roitberg.ufl.edu> Subject: [AMBER] VMD/netcdf To: AMBER Mailing List <amber.ambermd.org> Message-ID: <e6533899-1338-a1c1-c863-d79de9595f75.ufl.edu> Content-Type: text/plain; charset=utf-8; format=flowed Dear All Thanks to John Stone (VMD head developer), vmd can now read the default binary netcdf format for amber16 restart files. Just load the prmtop as usual and then load the restart file as netcdf in the file format menu. You need vmd 1.9.3 to do this, and as always, windows does not know how to read netcdf files. adrian -- Dr. Adrian E. Roitberg University of Florida Research Foundation Professor. Department of Chemistry University of Florida roitberg.ufl.edu 352-392-6972 ------------------------------ Message: 18 Date: Thu, 5 Jan 2017 20:53:38 +0530 From: Mary Varughese <maryvj1985.gmail.com> Subject: Re: [AMBER] cpptraj clustering doubt To: AMBER Mailing List <amber.ambermd.org> Message-ID: <CAAjRZU09FzzD+biP8YS5DwBp3LBhup2CnqvUxi_ntWyTMEnn+Q.mail.gmail.com> Content-Type: text/plain; charset=UTF-8 Thank you very much. It was very much helpful especially that script in supp material. Thank you On Thu, Jan 5, 2017 at 7:00 PM, Daniel Roe <daniel.r.roe.gmail.com> wrote: > See also the clustering script in part 7 of the supporting info here: > http://pubs.acs.org/doi/abs/10.1021/jp4125099 > > -Dan > > On Thu, Jan 5, 2017 at 7:48 AM, Christina Bergonzo <cbergonzo.gmail.com> > wrote: > > Hi, > > > > You can take a look at the supporting information of this paper: > > Highly sampled tetranucleotide and tetraloop motifs enable evaluation of > > common RNA force fields > > <https://scholar.google.com/citations?view_op=view_citation&hl=en&user= > d4JoelsAAAAJ&citation_for_view=d4JoelsAAAAJ:In6cVmBjs0IC> > > C Bergonzo, NM Henriksen, DR Roe, TE Cheatham > > RNA 21 (9), 1578-1590 > > > > http://rnajournal.cshlp.org/content/suppl/2015/06/19/rna. > 051102.115.DC1/SuppMaterial.pdf > > > > There is a CPPTRAJ command which performs combined clustering on 8 > > trajectories. > > > > Hope this helps, > > Christina > > > > On Thu, Jan 5, 2017 at 5:59 AM, Mary Varughese <maryvj1985.gmail.com> > wrote: > > > >> Sir > >> > >> > >> in combined clustering, could i compare more than two trajectories? If > >> so is there any difference in the command option > >> split ..Frameno... > >> > >> Also in split_summary.dat is there any useful information. > >> > >> how could i infer the convergence of the trajectory DBI , pSF > >> > >> Please give me some idea > >> > >> > >> Thanking you > >> > >> > >> mary varughese > >> > >> _______________________________________________ > >> AMBER mailing list > >> AMBER.ambermd.org > >> http://lists.ambermd.org/mailman/listinfo/amber > >> > > > > > > > > -- > > ------------------------------------------------------------ > --------------------------- > > Christina Bergonzo > > Postdoctoral Researcher > > Department of Medicinal Chemistry, University of Utah > > http://home.chpc.utah.edu/~cheatham/ > > ------------------------------------------------------------ > --------------------------- > > _______________________________________________ > > AMBER mailing list > > AMBER.ambermd.org > > http://lists.ambermd.org/mailman/listinfo/amber > > > > -- > ------------------------- > Daniel R. Roe > Laboratory of Computational Biology > National Institutes of Health, NHLBI > 5635 Fishers Ln, Rm T900 > Rockville MD, 20852 > https://www.lobos.nih.gov/lcb > > _______________________________________________ > AMBER mailing list > AMBER.ambermd.org > http://lists.ambermd.org/mailman/listinfo/amber > ------------------------------ Message: 19 Date: Thu, 5 Jan 2017 16:25:48 +0000 From: anu chandra <anu80125.gmail.com> Subject: [AMBER] cpptraj PCA of crystal structure To: AMBER Mailing List <amber.ambermd.org> Message-ID: <CADEHHN=_kDwwQ1CznH9Zx5jL_u5Q1zX7eWG97UtzsDZ=1Cbqcw.mail.gmail.com> Content-Type: text/plain; charset=UTF-8 Hello all, I am trying to project the crystal structure of a protein with 210 aminoacids, into top two PC (principle component) space. For this, I have carried out PCA of crystal structure with the following script. Unfortunately, all modes came out as zeros. The crystal structure was obtained from PDB. Am I doing anything wrong here?. ------------------------------------------------------------------------------------------------------ parm 1atm.pdb trajin 1atm.pdb createcrd crd1 run crdaction crd1 matrix covar !.H= name gaccCovar runanalysis diagmatrix gaccCovar vecs 2 name evecs crdaction crd1 projection T1 modes evecs beg 1 end 2 !.H= out T1.dat ----------------------------------------------------------------------------------------------------------- Any help would be highly appreciated Many thanks Anu ------------------------------ Message: 20 Date: Thu, 5 Jan 2017 11:30:14 -0500 From: Daniel Roe <daniel.r.roe.gmail.com> Subject: Re: [AMBER] cpptraj PCA of crystal structure To: AMBER Mailing List <amber.ambermd.org> Message-ID: <CAAC0qOZ8BOxLzXXthu2iFh+usbyre3v9GWoCJUPDqyhKB51q6Q.mail.gmail.com> Content-Type: text/plain; charset=UTF-8 It makes no sense to do principal component analysis on a single trajectory frame (there is no motion to analyze!). The coordinate covariance matrix should be constructed from at least as many frames as you have coordinates, otherwise you will have eigenvectors with zero eigenvalues. Maybe you wanted to do normal mode analysis? In that case you should use the 'nab' program. See the Amber16 manual for details. -Dan On Thu, Jan 5, 2017 at 11:25 AM, anu chandra <anu80125.gmail.com> wrote: > Hello all, > > I am trying to project the crystal structure of a protein with 210 > aminoacids, into top two PC (principle component) space. For this, I have > carried out PCA of crystal structure with the following script. > Unfortunately, all modes came out as zeros. The crystal structure was > obtained from PDB. Am I doing anything wrong here?. > > ------------------------------------------------------------------------------------------------------ > parm 1atm.pdb > > > trajin 1atm.pdb > > createcrd crd1 > run > > crdaction crd1 matrix covar !.H= name gaccCovar > > runanalysis diagmatrix gaccCovar vecs 2 name evecs > > crdaction crd1 projection T1 modes evecs beg 1 end 2 !.H= out T1.dat > ----------------------------------------------------------------------------------------------------------- > > > Any help would be highly appreciated > > > Many thanks > Anu > _______________________________________________ > AMBER mailing list > AMBER.ambermd.org > http://lists.ambermd.org/mailman/listinfo/amber -- ------------------------- Daniel R. Roe Laboratory of Computational Biology National Institutes of Health, NHLBI 5635 Fishers Ln, Rm T900 Rockville MD, 20852 https://www.lobos.nih.gov/lcb ------------------------------ Message: 21 Date: Fri, 6 Jan 2017 00:34:46 +0800 From: jacky zhao <jackyzhao010.gmail.com> Subject: [AMBER] /dev/hfi1_0 device failed to appear after 15 seconds: Connection timed out To: AMBER Mailing List <amber.ambermd.org> Message-ID: <CA+LvrWPMaUC41abbkmh+SqULEHbn0R4L49FXmLjaeVcCy7jCeQ.mail.gmail.com> Content-Type: text/plain; charset=UTF-8 Hi everyone I have compiled Amber16 on my workstation, which have passed amber test without any failed. But when I run Amber16_benchmark_suite, the following information appear: DESKTOP-NODMRU7.lan.1720hfi_wait_for_device: The /dev/hfi1_0 device failed to appear after 15 seconds: Connection timed out. Have you anyone give me some suggestions to solve this problem? Thank you for taking your time. Jacky ------------------------------ Message: 22 Date: Thu, 5 Jan 2017 11:41:00 -0500 From: Susan Chacko <susanc.helix.nih.gov> Subject: Re: [AMBER] Amber16 on K80 GPUs --poor performance on multiple GPUs To: amber.ambermd.org Message-ID: <1074c1a5-d700-6710-1980-4febca064e8f.helix.nih.gov> Content-Type: text/plain; charset=windows-1252; format=flowed I had read that and tried adding the --bind-to-none flag, but it didn't make a difference. I've also tried explicitly allocating CPUs with 'taskset', selecting CPUs that are on the same socket as the GPUs. There are no other jobs running on this K80 node: just my one benchmark run. I have a little additional information to add now. We have both K20xs and K80s in our cluster. For the Factor IX NPT benchmark: K20x, 1 GPU: 31.94 ns/day K20x, 2 GPU: 30.97 ns/day K80, 1 GPU: 30.26 ns/day K80, 2 GPU: 1.13 ns/day Thus, the K20x's behave similar to my benchmarks with Amber14. I only see the big drop in performance with the K80s. The instruction sets for the K20xs are different from the K80s. K20x: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms K80: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm cqm_llc cqm_occup_llc I had originally built on the K20x, so that the executable would run on either type of GPU. I tried rebuilding on the K80, but even with this K80-built executable, I'm still getting ~30 ns/day on 1 K80, and ~1 ns/day on 2 K80s. One thing I noticed is that I see 4 processes running on 2 K80 GPUs, while only 2 processes run on 2 K20 GPUs. i.e. K20x nvidia-smi: +------------------------------------------------------+ | NVIDIA-SMI 352.39 Driver Version: 352.39 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla K20Xm On | 0000:08:00.0 Off | Off | | N/A 33C P0 97W / 235W | 275MiB / 6143MiB | 62% Default | +-------------------------------+----------------------+----------------------+ | 1 Tesla K20Xm On | 0000:27:00.0 Off | Off | | N/A 35C P0 103W / 235W | 333MiB / 6143MiB | 71% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 55048 C ...cal/apps/amber/amber16/bin/pmemd.cuda.MPI 259MiB | | 1 55049 C ...cal/apps/amber/amber16/bin/pmemd.cuda.MPI 317MiB | +-----------------------------------------------------------------------------+ K80x nvidia-smi: +------------------------------------------------------+ | NVIDIA-SMI 352.39 Driver Version: 352.39 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla K80 Off | 0000:83:00.0 Off | 0 | | N/A 34C P8 27W / 149W | 22MiB / 11519MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 1 Tesla K80 Off | 0000:84:00.0 Off | 0 | | N/A 26C P8 31W / 149W | 22MiB / 11519MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 2 Tesla K80 Off | 0000:8A:00.0 Off | 0 | | N/A 75C P0 73W / 149W | 349MiB / 11519MiB | 99% Default | +-------------------------------+----------------------+----------------------+ | 3 Tesla K80 Off | 0000:8B:00.0 Off | 0 | | N/A 49C P0 80W / 149W | 407MiB / 11519MiB | 99% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 2 29646 C ...cal/apps/amber/amber16/bin/pmemd.cuda.MPI 258MiB | | 2 29647 C ...cal/apps/amber/amber16/bin/pmemd.cuda.MPI 64MiB | | 3 29646 C ...cal/apps/amber/amber16/bin/pmemd.cuda.MPI 64MiB | | 3 29647 C ...cal/apps/amber/amber16/bin/pmemd.cuda.MPI 316MiB | +-----------------------------------------------------------------------------+ Is this significant? I'm running the exact same command and executable on both types of GPUs, i.e. mpirun --bind-to none -np 2 `which pmemd.cuda.MPI` -O -i mdin.GPU -o mdout -p prmtop -c inpcrd The CPU processes look fine, i.e. both the K20x and K80s show 2 MPI CPU processes. Susan On 1/4/17 8:05 AM, Ross Walker wrote: > Hi Susan, > > Please see the following: > > http://ambermd.org/gpus/#Max_Perf <http://ambermd.org/gpus/#Max_Perf> > > particularly item 10 with regards to OpenMPI. I reproduced it here for convenience but I'd recommend reading the whole page. > > > 10. If you see that performance when running multiple - multi-GPU runs is bad. That is that say you run 2 x 2GPU jobs and they don't both run at full speed as if the other job was never running then make sure you turn off thread affinity within your MPI implementation or at least set each MPI thread to use a difference core. In my experience MPICH does not have this on by default and so no special settings are needed however both MVAPICH and OpenMPI set thread affinity by default. This would actually be useful if they did it in an intelligent way. However, it seems they pay no attention to load or even other MVAPICH or OpenMPI runs and always just assign from core 0. So 2 x 2 GPU jobs are, rather foolishly, assigned to cores 0 and 1 in both cases. The simplest solution here is to just disable thread affinity as follows: > > MVAPICH: export MV2_ENABLE_AFFINITY=0; mpirun -np 2 ... > OpenMPI: mpirun --bind-to none -np 2 ... > > All the best > Ross > > >> On Jan 3, 2017, at 12:36, Susan Chacko <susanc.helix.nih.gov> wrote: >> >> According to mdout, peer-to-peer support is enabled. >> >> |------------------- GPU DEVICE INFO -------------------- >> | >> | Task ID: 0 >> | CUDA_VISIBLE_DEVICES: not set >> | CUDA Capable Devices Detected: 4 >> | CUDA Device ID in use: 0 >> | CUDA Device Name: Tesla K80 >> | CUDA Device Global Mem Size: 11519 MB >> | CUDA Device Num Multiprocessors: 13 >> | CUDA Device Core Freq: 0.82 GHz >> | >> | >> | Task ID: 1 >> | CUDA_VISIBLE_DEVICES: not set >> | CUDA Capable Devices Detected: 4 >> | CUDA Device ID in use: 1 >> | CUDA Device Name: Tesla K80 >> | CUDA Device Global Mem Size: 11519 MB >> | CUDA Device Num Multiprocessors: 13 >> | CUDA Device Core Freq: 0.82 GHz >> | >> |-------------------------------------------------------- >> >> |---------------- GPU PEER TO PEER INFO ----------------- >> | >> | Peer to Peer support: ENABLED >> >> >> I also downloaded and ran the check_p2p program from the Amber site, and >> got: >> >> ----------- >> >> % ./gpuP2PCheck >> CUDA_VISIBLE_DEVICES is unset. >> CUDA-capable device count: 4 >> GPU0 " Tesla K80" >> GPU1 " Tesla K80" >> GPU2 " Tesla K80" >> GPU3 " Tesla K80" >> >> Two way peer access between: >> GPU0 and GPU1: YES >> GPU0 and GPU2: YES >> GPU0 and GPU3: YES >> GPU1 and GPU2: YES >> GPU1 and GPU3: YES >> GPU2 and GPU3: YES >> >> ----------- >> >> So in theory I should be able to run on up to 4 GPUs. >> I'll try rebuilding with CUDA 8.0 next, as Huang Jing suggested, unless >> anyone else has other ideas. >> >> Susan. >> >> >> On 1/3/17 11:25 AM, Daniel Roe wrote: >>> Hi, >>> >>> See the 'Multi GPU' section in http://ambermd.org/gpus/#Running for >>> some tips. In particular you need to make sure that the GPUs can run >>> with direct peer-to-peer communication to get any kind of speedup for >>> multi GPUs (this is printed somewhere near the top of mdout output). >>> >>> -Dan >>> >>> On Tue, Jan 3, 2017 at 11:00 AM, Susan Chacko <susanc.helix.nih.gov> wrote: >>>> Hi all, >>>> >>>> I successfully built Amber 16 with Intel 2015.1.133, CUDA 7.5, and >>>> OpenMPI 2.0.1. We're running Centos 6.8 and Nvidia drivers 352.39 on >>>> K80x GPUs. >>>> >>>> I ran the benchmark suite. I'm getting approx the same results as shown >>>> on the Amber16 benchmark page for CPUs and 1 GPU >>>> (http://ambermd.org/gpus/benchmarks.htm) >>>> >>>> e.g. >>>> >>>> Factor IX NPT >>>> >>>> Intel E5-2695 v3 . 2.30GHz, 28 cores: 9.58 ns/day >>>> >>>> 1 K80 GPU: 31.2 ns/day >>>> >>>> However, when I attempt to run on 2 K80 GPUs, performance drops >>>> dramatically. >>>> 2 K80 GPUs: 1.19 ns/day >>>> >>>> I'm running the pmemd.cuda_SPFP.MPI executable like this: >>>> cd Amber16_Benchmark_Suite/PME/FactorIX_production_NPT >>>> mpirun -np # /usr/local/apps/amber/amber16/bin/pmemd.cuda_SPFP.MPI -O -i >>>> mdin.GPU -o mdout -p prmtop -c inpcrd >>>> where # is 1 or 2. >>>> Each of the individual GPUs ran this benchmark at ~31.2 ns/day, so I >>>> don't think there is any intrinsic problem with any of GPU hardware. >>>> I get the same drop in performance with pmemd.cuda_DPFP.MPI and >>>> pmemd.cuda_SPXP.MPI >>>> >>>> Is this expected behaviour? I don't see a benchmark for 2 or more K80s >>>> on the Amber16 GPUs benchmark page, so am not sure what to expect. I >>>> also see that the benchmarks on that page were run with Amber16/ Centos >>>> 7 + CUDA 8.0 + MPICH 3.1.4 and are running on later versions of the >>>> Nvidia drivers than we have, but I would not expect those differences to >>>> account for what I'm seeing. >>>> >>>> Any ideas? Is it worth rebuilding with CUDA 8.0, or MPICH instead of >>>> OpenMPI? >>>> >>>> All thoughts and suggestions much appreciated, >>>> Susan. >>>> >>>> >>>> _______________________________________________ >>>> AMBER mailing list >>>> AMBER.ambermd.org >>>> http://lists.ambermd.org/mailman/listinfo/amber >>> >> -- >> Susan Chacko, Ph.D. >> HPC . NIH Staff >> >> >> _______________________________________________ >> AMBER mailing list >> AMBER.ambermd.org >> http://lists.ambermd.org/mailman/listinfo/amber > _______________________________________________ > AMBER mailing list > AMBER.ambermd.org > http://lists.ambermd.org/mailman/listinfo/amber -- Susan Chacko, Ph.D. HPC . NIH Staff ------------------------------ Message: 23 Date: Thu, 5 Jan 2017 18:33:53 +0100 From: erik rp zuiderweg <zuiderwe.umich.edu> Subject: Re: [AMBER] compatibility issues To: david.case.rutgers.edu, AMBER Mailing List <amber.ambermd.org> Message-ID: <4378E42D-E7E0-40FC-A3F5-D3D02E73C7C0.umich.edu> Content-Type: text/plain; charset=utf-8 Making progress ? having my GNU compilers (vs.6.1.0) reinstalled with XCode8.1 on MacOS 10.11.6 and running. Now trying to reinstall AMBER11 Starting in amber11/Ambertools/src I changed the configure file to point to the new compilers. then successfully issued the commands ./configure -macAccelerate gnu ?> OK make clean ?> OK make runs fine, and complies a gazillion things, also fortran codes, but then chokes on gfortran-6.1.0 -c -O0 -ffree-form -o qm2_pm6_hof_module.o _qm2_pm6_hof_module.f _qm2_pm6_hof_module.f:109:41: double precision function hofCorrection() 1 Error: In generic interface 'pm6_correction' at (1) procedures must be either all SUBROUTINEs or all FUNCTIONs Seems to be a compiler incompatibility. Can you help? Or should I just buy Amber16 (OK with me too) Erik Zuiderweg University of Michigan zuiderwe.umich.edu (734) 276 4463 > On Jan 5, 2017, at 4:06 AM, David Case <david.case.rutgers.edu> wrote: > > On Wed, Jan 04, 2017, Erik Zuiderweg wrote: > >> Mi Amber recompilation failed with errors. >> I now need to reinstall Xcode and the compilers ?. > > No getting around this if you upgrade a MacOS to something like Sierra. > > But the rest of Amber installation on Macs has been considerably simplified, > as pointed out at http://ambermd.org/amber_install.html. In particular, > there is no need for macports if you don't want it, and don't need it for > other uses. And the back-and-forth here about python 2 vs 3 is only for those > who care...almost all Amber users should just say "yes" when the configure > script offers to download the correct python. > > ...dac > > > _______________________________________________ > AMBER mailing list > AMBER.ambermd.org > http://lists.ambermd.org/mailman/listinfo/amber ------------------------------ Message: 24 Date: Thu, 5 Jan 2017 12:44:25 -0500 From: Samaneh Ghassabi Kondalaji <saghassabikondalaji.mix.wvu.edu> Subject: [AMBER] Ion Parameters for tip4pfb To: amber.ambermd.org Message-ID: <CAGx1a2x3e-Ta4_jZRw+uHjDoU5jHreiS+rE9M5DfuKB8iw=qKA.mail.gmail.com> Content-Type: text/plain; charset="utf-8" ?Dear Amber developers and users I am attempting to use tip4pfb water model to solvate a peptide. Initially I sourced the leaprc.protein.ff14SB file in addition to loading solvents.lib, frcmod.tip4pfb and atomic_ions.lib files. After solvating the peptide in FB4BOX, using the addions (or addions2) I added 2 Na+ ions without any errors (the Na+ exists in Unit list). However, when I try to save the prmtop and inpcrd files, I get this error (leap.log file is attached to this E-mail): *For atom: .R<Na+ 520>.A<Na+ 1> Could not find vdW (or other) parameters for type: Na+* Now, I understand that I need to source ion parameters using files such as leaprc.water.tip4pewFB4 and frcmod.ionsjc_tip4pewFB4 for FB4BOX water model, except that these files do not exist. Any suggestions on how I can find these parameters would be greatly appreciated. Best Regards samaneh ?P.S. The system is a peptide (ACE LYS NME) with overall charge state of +3 in water. The error message does not appear when I use tip3fb water model with related force field files. -- *Samaneh ghassabi kondalaji* *PhD student in Chemistry* *555 CRL, Chemistry department* *West Virginia University * *Morgantown, USA* *E-mail: **saghassabikondalaji.mix.wvu.edu* <saghassabikondalaji.mix.wvu.edu> -------------- next part -------------- A non-text attachment was scrubbed... Name: leap.log Type: text/x-log Size: 22498 bytes Desc: not available Url : http://lists.ambermd.org/mailman/private/amber/attachments/20170105/10de5043/attachment-0001.bin ------------------------------ Message: 25 Date: Thu, 5 Jan 2017 12:45:05 -0500 From: Hai Nguyen <nhai.qn.gmail.com> Subject: Re: [AMBER] compatibility issues To: AMBER Mailing List <amber.ambermd.org> Cc: David A Case <david.case.rutgers.edu> Message-ID: <CAFNMPM89m7nSWmDzXbUt0wh7GesAW3-GVhSs4eCM7xAANHqXOA.mail.gmail.com> Content-Type: text/plain; charset=UTF-8 Please upgrade to AmberTools16 (free anyway). Using clang: ./configure -macAccelerate clang Using GNU ./configure gnu Hai On Thu, Jan 5, 2017 at 12:33 PM, erik rp zuiderweg <zuiderwe.umich.edu> wrote: > Making progress ? > having my GNU compilers (vs.6.1.0) reinstalled with XCode8.1 on MacOS > 10.11.6 and running. > > Now trying to reinstall AMBER11 > > Starting in amber11/Ambertools/src > I changed the configure file to point to the new compilers. > then successfully issued the commands > ./configure -macAccelerate gnu ?> OK > make clean ?> OK > > make > runs fine, and complies a gazillion things, also fortran codes, but then > chokes on > > gfortran-6.1.0 -c -O0 -ffree-form -o qm2_pm6_hof_module.o > _qm2_pm6_hof_module.f > _qm2_pm6_hof_module.f:109:41: > > double precision function hofCorrection() > 1 > Error: In generic interface 'pm6_correction' at (1) procedures must be > either all SUBROUTINEs or all FUNCTIONs > > Seems to be a compiler incompatibility. > > Can you help? Or should I just buy Amber16 (OK with me too) > > > > > > Erik Zuiderweg > University of Michigan > zuiderwe.umich.edu > (734) 276 4463 > > > On Jan 5, 2017, at 4:06 AM, David Case <david.case.rutgers.edu> wrote: > > > > On Wed, Jan 04, 2017, Erik Zuiderweg wrote: > > > >> Mi Amber recompilation failed with errors. > >> I now need to reinstall Xcode and the compilers ?. > > > > No getting around this if you upgrade a MacOS to something like Sierra. > > > > But the rest of Amber installation on Macs has been considerably > simplified, > > as pointed out at http://ambermd.org/amber_install.html. In particular, > > there is no need for macports if you don't want it, and don't need it for > > other uses. And the back-and-forth here about python 2 vs 3 is only for > those > > who care...almost all Amber users should just say "yes" when the > configure > > script offers to download the correct python. > > > > ...dac > > > > > > _______________________________________________ > > AMBER mailing list > > AMBER.ambermd.org > > http://lists.ambermd.org/mailman/listinfo/amber > > > _______________________________________________ > AMBER mailing list > AMBER.ambermd.org > http://lists.ambermd.org/mailman/listinfo/amber > ------------------------------ Message: 26 Date: Thu, 5 Jan 2017 12:55:23 -0500 From: Jason Swails <jason.swails.gmail.com> Subject: Re: [AMBER] antechamber: why the net charge does not sum up to zero? To: AMBER Mailing List <amber.ambermd.org> Message-ID: <CAEk9e3pwfEV9TJ4-CeZD87uzJqvHV=MihLm6_4B2kD6AJUypMg.mail.gmail.com> Content-Type: text/plain; charset=UTF-8 On Thu, Jan 5, 2017 at 5:52 AM, Azade Yazdan Yar <azade.yazdanyar.gmail.com> wrote: > Hello everyone, > > > > I have a problem similar to here <http://archive.ambermd.org/ > all/2855.html>. > Unfortunately, I did not find an answer to that inquiry. > > I am using antechamber on a pdb file to create the mol2 file. > > The molecule should be charge neutral and I am using ?-c bcc ?nc 0? flags. > However, the sum of the charges in the output.mol2 generated by antechamber > is not 0.0. > > I also tried ?-ek maxcyc=0? flag on several initial.pdb files (point 7, > P.277 of the manual) which all of them lead to a non-zero net charge. The > smallest net charge which I can get is 0.001. > > My question is whether or not this net charge is small enough so I can work > with it or is there is another trick which I can use? > ?The only comment I'll make in addition to what Hannes said is that unless you plan on using the generated mol2 file as a template for a solvent molecule -- that is, unless you plan on including many thousands of copies of this residue in your system -- the round-off error of 0.01 to 0.001 will make little to no difference (values this small get smeared over every atom in the system by sander and pmemd). If you plan on including many thousands of copies, then the net charge may start to become significant (although the net neutralizing plasma will still be able to handle this excess charge for most applications). However, ParmEd has the ability to redistribute this excess charge to yield a set of charges that round to an exact integer for any level of precision you want in the charges. This can be accomplished with a quick Python script that looks like this: import parmed as pmd pmd.load_file('your_file.mol2').fix_charges(precision=4).save('fixed_charges.mol2') This needs to be invoked with the amber.python interpreter to make sure that ParmEd can be found. However, as mentioned before, unless you plan on including thousands of copies of this residue in your system, there is nothing to worry about. HTH, Jason -- Jason M. Swails ------------------------------ Message: 27 Date: Thu, 5 Jan 2017 14:15:30 -0500 From: Susan Chacko <susanc.helix.nih.gov> Subject: Re: [AMBER] Amber16 on K80 GPUs --poor performance on multiple GPUs To: amber.ambermd.org Message-ID: <9b7fd63c-8cf8-87fb-852a-59ae2fb8aff6.helix.nih.gov> Content-Type: text/plain; charset=windows-1252; format=flowed Ah, someone suggested backchannel that I should try setting the CUDA_VISIBLE_DEVICES to ensure that 2 GPUs were being used. deviceQuery says there are 2 devices called 0 and 1 % ./deviceQuery -noprompt | egrep "^Device" Device 0: "Tesla K80" Device 1: "Tesla K80" but nvidia-smi says there are 4. +------------------------------------------------------+ | NVIDIA-SMI 352.39 Driver Version: 352.39 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla K80 Off | 0000:83:00.0 Off | 0 | | N/A 45C P0 57W / 149W | 22MiB / 11519MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 1 Tesla K80 Off | 0000:84:00.0 Off | 0 | | N/A 31C P0 72W / 149W | 22MiB / 11519MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 2 Tesla K80 Off | 0000:8A:00.0 Off | 0 | | N/A 47C P0 59W / 149W | 22MiB / 11519MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 3 Tesla K80 Off | 0000:8B:00.0 Off | 0 | | N/A 36C P0 71W / 149W | 22MiB / 11519MiB | 94% Default | +-------------------------------+----------------------+----------------------+ I tried all combinations of the 4 GPU devices in pairs: CUDA_VISIBLE_DEVICES=0,1 nvidia-smi reports devices 2,3 are being used. Performance: ~1 ns/day CUDA_VISIBLE_DEVICES=0,2 nvidia-smi reports devices 0,2 are being used. Performance: ~44 ns/day CUDA_VISIBLE_DEVICES=0,3 nvidia-smi reports devices 1,2 are being used. Performance: ~44 ns/day CUDA_VISIBLE_DEVICES=1,2 nvidia-smi reports devices 0,3 are being used. Performance: ~44 ns/day CUDA_VISIBLE_DEVICES=1,3 nvidia-smi reports devices 1,3 are being used. Performance: ~44 ns/day CUDA_VISIBLE_DEVICES=2,3 nvidia-smi reports devices 0,1 are being used. Performance: ~1 ns/day Confusing! Anyway, now that I am able to run the benchmark and get 44 ns/day performance, i know there's nothing intrinsically wrong with the build. I'll do some more research into our GPUs and how the GPU devices are labelled. Many thanks to everyone who offered suggestions, either here or backchannel, Susan. On 1/5/17 11:41 AM, Susan Chacko wrote: > I had read that and tried adding the --bind-to-none flag, but it didn't > make a difference. I've also tried explicitly allocating CPUs with > 'taskset', selecting CPUs that are on the same socket as the GPUs. > > There are no other jobs running on this K80 node: just my one benchmark > run. > > I have a little additional information to add now. We have both K20xs > and K80s in our cluster. For the Factor IX NPT benchmark: > > K20x, 1 GPU: 31.94 ns/day > K20x, 2 GPU: 30.97 ns/day > > K80, 1 GPU: 30.26 ns/day > K80, 2 GPU: 1.13 ns/day > > Thus, the K20x's behave similar to my benchmarks with Amber14. I only > see the big drop in performance with the K80s. > > The instruction sets for the K20xs are different from the K80s. > K20x: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat > pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb > rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology > nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est > tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt > tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb > xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid fsgsbase smep > erms > > K80: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat > pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb > rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology > nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est > tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt > tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb > xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase > bmi1 avx2 smep bmi2 erms invpcid cqm cqm_llc cqm_occup_llc > > I had originally built on the K20x, so that the executable would run on > either type of GPU. I tried rebuilding on the K80, but even with this > K80-built executable, I'm still getting ~30 ns/day on 1 K80, and ~1 > ns/day on 2 K80s. > > One thing I noticed is that I see 4 processes running on 2 K80 GPUs, > while only 2 processes run on 2 K20 GPUs. i.e. > > K20x nvidia-smi: > +------------------------------------------------------+ > | NVIDIA-SMI 352.39 Driver Version: 352.39 | > |-------------------------------+----------------------+----------------------+ > | GPU Name Persistence-M| Bus-Id Disp.A | Volatile > Uncorr. ECC | > | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util > Compute M. | > |===============================+======================+======================| > | 0 Tesla K20Xm On | 0000:08:00.0 Off > | Off | > | N/A 33C P0 97W / 235W | 275MiB / 6143MiB | 62% Default | > +-------------------------------+----------------------+----------------------+ > | 1 Tesla K20Xm On | 0000:27:00.0 Off > | Off | > | N/A 35C P0 103W / 235W | 333MiB / 6143MiB | 71% Default | > +-------------------------------+----------------------+----------------------+ > > +-----------------------------------------------------------------------------+ > | Processes: GPU Memory | > | GPU PID Type Process name Usage | > |=============================================================================| > | 0 55048 C ...cal/apps/amber/amber16/bin/pmemd.cuda.MPI > 259MiB | > | 1 55049 C ...cal/apps/amber/amber16/bin/pmemd.cuda.MPI > 317MiB | > +-----------------------------------------------------------------------------+ > > K80x nvidia-smi: > +------------------------------------------------------+ > | NVIDIA-SMI 352.39 Driver Version: 352.39 | > |-------------------------------+----------------------+----------------------+ > | GPU Name Persistence-M| Bus-Id Disp.A | Volatile > Uncorr. ECC | > | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util > Compute M. | > |===============================+======================+======================| > | 0 Tesla K80 Off | 0000:83:00.0 Off > | 0 | > | N/A 34C P8 27W / 149W | 22MiB / 11519MiB | 0% Default | > +-------------------------------+----------------------+----------------------+ > | 1 Tesla K80 Off | 0000:84:00.0 Off > | 0 | > | N/A 26C P8 31W / 149W | 22MiB / 11519MiB | 0% Default | > +-------------------------------+----------------------+----------------------+ > | 2 Tesla K80 Off | 0000:8A:00.0 Off > | 0 | > | N/A 75C P0 73W / 149W | 349MiB / 11519MiB | 99% Default | > +-------------------------------+----------------------+----------------------+ > | 3 Tesla K80 Off | 0000:8B:00.0 Off > | 0 | > | N/A 49C P0 80W / 149W | 407MiB / 11519MiB | 99% Default | > +-------------------------------+----------------------+----------------------+ > > +-----------------------------------------------------------------------------+ > | Processes: GPU Memory | > | GPU PID Type Process name Usage | > |=============================================================================| > | 2 29646 C ...cal/apps/amber/amber16/bin/pmemd.cuda.MPI > 258MiB | > | 2 29647 C ...cal/apps/amber/amber16/bin/pmemd.cuda.MPI > 64MiB | > | 3 29646 C ...cal/apps/amber/amber16/bin/pmemd.cuda.MPI > 64MiB | > | 3 29647 C ...cal/apps/amber/amber16/bin/pmemd.cuda.MPI > 316MiB | > +-----------------------------------------------------------------------------+ > > Is this significant? I'm running the exact same command and executable > on both types of GPUs, i.e. > > mpirun --bind-to none -np 2 `which pmemd.cuda.MPI` -O -i mdin.GPU -o > mdout -p prmtop -c inpcrd > > The CPU processes look fine, i.e. both the K20x and K80s show 2 MPI CPU > processes. > > Susan > > On 1/4/17 8:05 AM, Ross Walker wrote: >> Hi Susan, >> >> Please see the following: >> >> http://ambermd.org/gpus/#Max_Perf <http://ambermd.org/gpus/#Max_Perf> >> >> particularly item 10 with regards to OpenMPI. I reproduced it here for convenience but I'd recommend reading the whole page. >> >> >> 10. If you see that performance when running multiple - multi-GPU runs is bad. That is that say you run 2 x 2GPU jobs and they don't both run at full speed as if the other job was never running then make sure you turn off thread affinity within your MPI implementation or at least set each MPI thread to use a difference core. In my experience MPICH does not have this on by default and so no special settings are needed however both MVAPICH and OpenMPI set thread affinity by default. This would actually be useful if they did it in an intelligent way. However, it seems they pay no attention to load or even other MVAPICH or OpenMPI runs and always just assign from core 0. So 2 x 2 GPU jobs are, rather foolishly, assigned to cores 0 and 1 in both cases. The simplest solution here is to just disable thread affinity as follows: >> >> MVAPICH: export MV2_ENABLE_AFFINITY=0; mpirun -np 2 ... >> OpenMPI: mpirun --bind-to none -np 2 ... >> >> All the best >> Ross >> >> >>> On Jan 3, 2017, at 12:36, Susan Chacko <susanc.helix.nih.gov> wrote: >>> >>> According to mdout, peer-to-peer support is enabled. >>> >>> |------------------- GPU DEVICE INFO -------------------- >>> | >>> | Task ID: 0 >>> | CUDA_VISIBLE_DEVICES: not set >>> | CUDA Capable Devices Detected: 4 >>> | CUDA Device ID in use: 0 >>> | CUDA Device Name: Tesla K80 >>> | CUDA Device Global Mem Size: 11519 MB >>> | CUDA Device Num Multiprocessors: 13 >>> | CUDA Device Core Freq: 0.82 GHz >>> | >>> | >>> | Task ID: 1 >>> | CUDA_VISIBLE_DEVICES: not set >>> | CUDA Capable Devices Detected: 4 >>> | CUDA Device ID in use: 1 >>> | CUDA Device Name: Tesla K80 >>> | CUDA Device Global Mem Size: 11519 MB >>> | CUDA Device Num Multiprocessors: 13 >>> | CUDA Device Core Freq: 0.82 GHz >>> | >>> |-------------------------------------------------------- >>> >>> |---------------- GPU PEER TO PEER INFO ----------------- >>> | >>> | Peer to Peer support: ENABLED >>> >>> >>> I also downloaded and ran the check_p2p program from the Amber site, and >>> got: >>> >>> ----------- >>> >>> % ./gpuP2PCheck >>> CUDA_VISIBLE_DEVICES is unset. >>> CUDA-capable device count: 4 >>> GPU0 " Tesla K80" >>> GPU1 " Tesla K80" >>> GPU2 " Tesla K80" >>> GPU3 " Tesla K80" >>> >>> Two way peer access between: >>> GPU0 and GPU1: YES >>> GPU0 and GPU2: YES >>> GPU0 and GPU3: YES >>> GPU1 and GPU2: YES >>> GPU1 and GPU3: YES >>> GPU2 and GPU3: YES >>> >>> ----------- >>> >>> So in theory I should be able to run on up to 4 GPUs. >>> I'll try rebuilding with CUDA 8.0 next, as Huang Jing suggested, unless >>> anyone else has other ideas. >>> >>> Susan. >>> >>> >>> On 1/3/17 11:25 AM, Daniel Roe wrote: >>>> Hi, >>>> >>>> See the 'Multi GPU' section in http://ambermd.org/gpus/#Running for >>>> some tips. In particular you need to make sure that the GPUs can run >>>> with direct peer-to-peer communication to get any kind of speedup for >>>> multi GPUs (this is printed somewhere near the top of mdout output). >>>> >>>> -Dan >>>> >>>> On Tue, Jan 3, 2017 at 11:00 AM, Susan Chacko <susanc.helix.nih.gov> wrote: >>>>> Hi all, >>>>> >>>>> I successfully built Amber 16 with Intel 2015.1.133, CUDA 7.5, and >>>>> OpenMPI 2.0.1. We're running Centos 6.8 and Nvidia drivers 352.39 on >>>>> K80x GPUs. >>>>> >>>>> I ran the benchmark suite. I'm getting approx the same results as shown >>>>> on the Amber16 benchmark page for CPUs and 1 GPU >>>>> (http://ambermd.org/gpus/benchmarks.htm) >>>>> >>>>> e.g. >>>>> >>>>> Factor IX NPT >>>>> >>>>> Intel E5-2695 v3 . 2.30GHz, 28 cores: 9.58 ns/day >>>>> >>>>> 1 K80 GPU: 31.2 ns/day >>>>> >>>>> However, when I attempt to run on 2 K80 GPUs, performance drops >>>>> dramatically. >>>>> 2 K80 GPUs: 1.19 ns/day >>>>> >>>>> I'm running the pmemd.cuda_SPFP.MPI executable like this: >>>>> cd Amber16_Benchmark_Suite/PME/FactorIX_production_NPT >>>>> mpirun -np # /usr/local/apps/amber/amber16/bin/pmemd.cuda_SPFP.MPI -O -i >>>>> mdin.GPU -o mdout -p prmtop -c inpcrd >>>>> where # is 1 or 2. >>>>> Each of the individual GPUs ran this benchmark at ~31.2 ns/day, so I >>>>> don't think there is any intrinsic problem with any of GPU hardware. >>>>> I get the same drop in performance with pmemd.cuda_DPFP.MPI and >>>>> pmemd.cuda_SPXP.MPI >>>>> >>>>> Is this expected behaviour? I don't see a benchmark for 2 or more K80s >>>>> on the Amber16 GPUs benchmark page, so am not sure what to expect. I >>>>> also see that the benchmarks on that page were run with Amber16/ Centos >>>>> 7 + CUDA 8.0 + MPICH 3.1.4 and are running on later versions of the >>>>> Nvidia drivers than we have, but I would not expect those differences to >>>>> account for what I'm seeing. >>>>> >>>>> Any ideas? Is it worth rebuilding with CUDA 8.0, or MPICH instead of >>>>> OpenMPI? >>>>> >>>>> All thoughts and suggestions much appreciated, >>>>> Susan. >>>>> >>>>> >>>>> _______________________________________________ >>>>> AMBER mailing list >>>>> AMBER.ambermd.org >>>>> http://lists.ambermd.org/mailman/listinfo/amber >>> -- >>> Susan Chacko, Ph.D. >>> HPC . NIH Staff >>> >>> >>> _______________________________________________ >>> AMBER mailing list >>> AMBER.ambermd.org >>> http://lists.ambermd.org/mailman/listinfo/amber >> _______________________________________________ >> AMBER mailing list >> AMBER.ambermd.org >> http://lists.ambermd.org/mailman/listinfo/amber -- Susan Chacko, Ph.D. HPC . NIH Staff ------------------------------ _______________________________________________ AMBER mailing list AMBER.ambermd.org http://lists.ambermd.org/mailman/listinfo/amber End of AMBER Digest, Vol 1806, Issue 1 ************************************** _______________________________________________ AMBER mailing list AMBER.ambermd.org http://lists.ambermd.org/mailman/listinfo/amberReceived on Fri Jan 06 2017 - 22:00:02 PST