I already told you what to do: stop using the GTX 580 as your display
device.
I just checked out what the effect of doing so is: you'll get back about 200
MB. This should be enough to run your system. I would guess this is the
difference between yours and Ross's results. My main work system has a $50
CUDA 1.1 display GPU with 512 MB for this very reason.
Don't believe me? Try this "nvidia-smi -q -a". With a 3.2 driver and
toolkit, this will indicate that the system eats ~190 MB on a 32-bit OS and
~250 MB on a 64-bit OS. In contrast, a non-display GPU only loses about 10
MB.
So why did this work before for you? First, performance costs memory. This
is a massively parallel calculation and it needs lots of memory to keep the
threads in the GPU from colliding into each other. Remember, one $500 GTX
580 performs on par with a $20,000 PC cluster. And that PC cluster will
need far more than 3 GB to keep up. It's the calculation itself that needs
space. I've nearly doubled performance and upped memory usage maybe 30%.
That's just how the cookie crumbles.
Further, AMBER is in some ways a victim of its own success. As more and
more previous corner cases go mainstream, I have had to write more and more
custom kernels to handle these cases in a combinatorial kernel compilation
conundrum. It appears the CUDA runtime loads *all* of them into GPU memory
upon initialization. This is a fixed cost regardless of system size and
it's not possible to calculate how much because it's all under the hood.
This is why a 3 GB GTX 580 can run cellulose and your 1.5 GB GTX 580 is now
stuck stuck in the mud.
All that said, I'm happy to hire myself out as a professional AMBER
consultant to build you a special version of AMBER for your specific cases
with all the offending kernels removed at my usual consulting rate (whatever
the heck that is but it's probably a lot more than buying yourself a $50
display GPU or running your machine headless).
Scott
On Sat, Aug 20, 2011 at 2:15 PM, filip fratev <filipfratev.yahoo.com> wrote:
> Farid,
> It look obvious yes, but can you EXPLAIN how Ross performed Factor IX
> benchmark (90K atoms) using GTX295 ( 896MB per GPU, i.e. nearly 2x less
> )???
> I can perform in a "realistic" case only about 80K atoms. Thus, if it is so
> obvious, please explain me the above and provide some useful tricks/guide
> how and what must be done before pmemd.cuda to be used.....
>
> This is of course not ONLY for you!
>
> Filip
>
>
>
> ________________________________
> From: "Ismail, Mohd F." <farid.ou.edu>
> To: filip fratev <filipfratev.yahoo.com>; AMBER Mailing List <
> amber.ambermd.org>
> Sent: Saturday, August 20, 2011 9:13 PM
> Subject: RE: [AMBER] Major GPU Update Released
>
> Filip,
>
> I think it's obvious, that the new optimization uses more GPU memory than
> the older code. That's why you were getting the out of memory error with
> 1.5GB, and no error with 3GB GPU RAM. You might need to use a smaller
> system with the GTX590 or just use the 580 only for the bigger system that
> you have.
>
>
> --Farid
>
> ________________________________________
> From: filip fratev [filipfratev.yahoo.com]
> Sent: Saturday, August 20, 2011 12:32 PM
> To: Ross Walker
> Cc: AMBER Mailing List
> Subject: Re: [AMBER] Major GPU Update Released
>
> Hi Ross,
> Thanks for your guide. However, as I wrote in my last post, something going
> wrong with cuda.MPI even with 89K atoms. Because, there were no problems
> with the older code and the same input files and settings, I thing and
> report that it could be
> some bug. Indeed it could be also something in my settings. I have no
> problems with Factor IX bench, but the system was very well
> equilibrated, thus this is another indication.
>
> Because you have at least 2 GTX580 with 1.5GB memory could you test please,
> when you have time, whats going wrong?
>
> If some one other can try/help (I saw that there are a lot of people with
> GTX590 and GTX580's) I will be very tankful!
>
> All the best,
> Filip
>
>
> ________________________________
> From: Ross Walker <ross.rosswalker.co.uk>
> To: 'filip fratev' <filipfratev.yahoo.com>; 'AMBER Mailing List' <
> amber.ambermd.org>
> Sent: Saturday, August 20, 2011 6:40 PM
> Subject: RE: [AMBER] Major GPU Update Released
>
> Hi Filip.
>
> Unfortunately some of the new optimizations come at the expense of memory.
> The GPU memory in use is a lower bounds on the amount of memory being used,
> the actual amount may be higher and can change during a run if density
> changes etc. You could switch to NVT which will use less memory. Also using
> things like the Berendsen thermostat should use less memory than langevin.
> Finally avoiding the use of restraints (which always seemed a little bogus
> to me in NPT simulations anyway) could also help reduce memory usage.
>
> You could also ensure that your machine is in runlevel 3 so that no X
> server is running and also try it from a fresh boot into runlevel 3 since it
> is possible there are memory leaks in some of the graphics drivers.
>
> All the best
> Ross
>
> > -----Original Message-----
> > From: filip fratev [mailto:filipfratev.yahoo.com]
> > Sent: Saturday, August 20, 2011 7:17 AM
> > To: filip fratev; AMBER Mailing List
> > Subject: Re: [AMBER] Major GPU Update Released
> >
> > Hi Scott and Ross,
> >
> > > For NPT:
> > > | GPU memory information:
> > > | KB of GPU memory in use: 882413
> > > | KB of CPU memory in use: 104090
> > >
> > >
> > > | GPU memory information:
> > > | KB of GPU memory in use: 1006146
> > > | KB of CPU memory in use: 99724
> >
> > So, the above values are from the older code and for the new are about
> > 2x more? Just to be clear.
> >
> >
> > Indeed that the update is great! I've wrote about that! Just need some
> > more information and help.
> >
> >
> > Regards,
> > Filip
> >
> >
> >
> >
> >
> > ________________________________
> > From: filip fratev <filipfratev.yahoo.com>
> > To: AMBER Mailing List <amber.ambermd.org>
> > Sent: Saturday, August 20, 2011 5:02 PM
> > Subject: Re: [AMBER] Major GPU Update Released
> >
> > I performed some tests last night.
> > All were NPT simulations. The test system was AMD 1090T.3.9Ghz, GTX590
> > (Asus) and 3GB GTX580 (Palit). Suse11.3.
> >
> >
> > JAC:
> > GTX590 1GPU= 32.61 ns/day
> >
> > GTX590 2GPU =42.19 ns/day
> >
> > GTX580 1GPU= 40.73 ns/day
> >
> > GTX580 plus GTX590=50.21 ns/day
> >
> > Factor IX:
> > GTX590 1GPU= 9.54 ns/day
> >
> > GTX590 2GPU =12.24 ns/day
> >
> > GTX580 1GPU= 11.72 ns/day
> >
> > GTX580 plus GTX590=14.69 ns/day
> >
> > Cellulose:
> > GTX580 (3GB) =2.67 ns/day
> >
> >
> >
> > Regards,
> > Filip
> >
> > P.S. My 1.5GB memory issue was still not solved...I will reduce waters
> > from 12A to 10A. Not good but seems the only way for now...hope to
> > work...
> >
> >
> >
> >
> >
> >
> > ________________________________
> > From: Levi Pierce <levipierce.gmail.com>
> > To: AMBER Mailing List <amber.ambermd.org>
> > Sent: Saturday, August 20, 2011 10:20 AM
> > Subject: Re: [AMBER] Major GPU Update Released
> >
> > Had a chance to sit down and test out the new patch. Wow! Very
> > impressive performance boost on a variety of systems I have been
> > running
> > pmemd.cuda on. Great work!
> >
> > On Fri, Aug 19, 2011 at 4:37 PM, Scott Le Grand
> > <varelse2005.gmail.com>wrote:
> >
> > > Use a different gpu foe display I suspect
> > > On Aug 19, 2011 4:09 PM, "filip fratev" <filipfratev.yahoo.com>
> > wrote:
> > > > Hi Ross,
> > > > I compiled the new code and performed many tests and the results
> > are
> > > really impressive! I will post later.
> > > >
> > > > However, I am in a big trouble with my systems (116K atoms) and
> > hope that
> > > you will be able to help me.
> > > > The problem is that with the new code I am not able to simulate
> > these
> > > proteins (116K) with GTX590 (1.5GB per core), because of some memory
> > > issue/bug:
> > > > cudaMalloc GpuBuffer::Allocate failed out of memory
> > > >
> > > > With the older code I had no any problems with same input files and
> > > configuration. I tried both NPT and NVT but the same problem...
> > > > Then I use GTX580 3GB and it works fine. From output you can see
> > that the
> > > requested memory is just 882MB:
> > > > For NPT:
> > > > | GPU memory information:
> > > > | KB of GPU memory in use: 882413
> > > > | KB of CPU memory in use: 104090
> > > >
> > > > and for restrained NVT:
> > > >
> > > > | GPU memory information:
> > > > | KB of GPU memory in use: 1006146
> > > > | KB of CPU memory in use: 99724
> > > > Thus I shouldn’t have any problem.
> > > >
> > > > What could be the issue and how I can solve it?
> > > >
> > > > Regards,
> > > > Filip
> > > >
> > > > Below is the output file (my NPT density.out) and heat.in:
> > > >
> > > > -------------------------------------------------------
> > > > Amber 11 SANDER 2010
> > > > -------------------------------------------------------
> > > >
> > > > | PMEMD implementation of SANDER, Release 11
> > > >
> > > > | Run on 08/20/2011 at 01:42:20
> > > >
> > > > [-O]verwriting output
> > > >
> > > > File Assignments:
> > > > | MDIN:
> > > densityF.in
> > > > | MDOUT:
> > > 0densitytest580Karti.out
> > > > | INPCRD:
> > > heattest.rst
> > > > | PARM:
> > > MyosinWT.prmtop
> > > > | RESTRT:
> > > density1test.rst
> > > > | REFC:
> > > heattest.rst
> > > > | MDVEL:
> > > mdvel
> > > > | MDEN:
> > > mden
> > > > | MDCRD:
> > > density1test.mdcrd
> > > > | MDINFO:
> > > mdinfo
> > > >
> > > >
> > > > Here is the input file:
> > > >
> > > > Ligand9
> > > density
> > > >
> > > &cntrl
> > >
> > > > imin=0,irest=1,
> > > ntx=5,
> > > >
> > > nstlim=5000,dt=0.002,
> > >
> > > > ntc=2,ntf=2, ig=-1,
> > > iwrap=1,
> > > > cut=8.0, ntb=2, ntp=1,
> > > taup=1.0,
> > > > ntpr=5000, ntwx=5000,
> > > ntwr=10000,
> > > > ntt=3,
> > > gamma_ln=2.0,
> > > >
> > > temp0=300.0,
> > >
> > > >
> > > /
> > >
> > > >
> > >
> > >
> > > >
> > >
> > >
> > > >
> > >
> > >
> > > >
> > >
> > >
> > > >
> > > >
> > > > Note: ig = -1. Setting random seed based on wallclock time in
> > > microseconds.
> > > >
> > > > |--------------------- INFORMATION ----------------------
> > > > | GPU (CUDA) Version of PMEMD in use: NVIDIA GPU IN USE.
> > > > | Version 2.2
> > > > |
> > > > | 08/16/2011
> > > > |
> > > > |
> > > > | Implementation by:
> > > > | Ross C. Walker (SDSC)
> > > > | Scott Le Grand (nVIDIA)
> > > > | Duncan Poole (nVIDIA)
> > > > |
> > > > | CAUTION: The CUDA code is currently experimental.
> > > > | You use it at your own risk. Be sure to
> > > > | check ALL results carefully.
> > > > |
> > > > | Precision model in use:
> > > > | [SPDP] - Hybrid Single/Double Precision (Default).
> > > > |
> > > > |--------------------------------------------------------
> > > >
> > > > |------------------- GPU DEVICE INFO --------------------
> > > > |
> > > > | CUDA Capable Devices Detected: 1
> > > > | CUDA Device ID in use: 0
> > > > | CUDA Device Name: GeForce GTX 580
> > > > | CUDA Device Global Mem Size: 3071 MB
> > > > | CUDA Device Num Multiprocessors: 16
> > > > | CUDA Device Core Freq: 1.57 GHz
> > > > |
> > > > |--------------------------------------------------------
> > > >
> > > >
> > > > | Conditional Compilation Defines Used:
> > > > | DIRFRC_COMTRANS
> > > > | DIRFRC_EFS
> > > > | DIRFRC_NOVEC
> > > > | PUBFFT
> > > > | FFTLOADBAL_2PROC
> > > > | BINTRAJ
> > > > | CUDA
> > > >
> > > > | Largest sphere to fit in unit cell has radius = 48.492
> > > >
> > > > | New format PARM file being parsed.
> > > > | Version = 1.000 Date = 05/27/11 Time = 11:50:53
> > > >
> > > > | Note: 1-4 EEL scale factors were NOT found in the topology file.
> > > > | Using default value of 1.2.
> > > >
> > > > | Note: 1-4 VDW scale factors were NOT found in the topology file.
> > > > | Using default value of 2.0.
> > > > | Duplicated 0 dihedrals
> > > >
> > > > | Duplicated 0 dihedrals
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > -----------
> > > > 1. RESOURCE USE:
> > > >
> > >
> > > ---------------------------------------------------------------------
> > -----------
> > > >
> > > > getting new box info from bottom of inpcrd
> > > >
> > > > NATOM = 116271 NTYPES = 21 NBONH = 109977 MBONA = 6423
> > > > NTHETH = 14190 MTHETA = 8659 NPHIH = 27033 MPHIA = 21543
> > > > NHPARM = 0 NPARM = 0 NNB = 207403 NRES = 35368
> > > > NBONA = 6423 NTHETA = 8659 NPHIA = 21543 NUMBND = 59
> > > > NUMANG = 124 NPTRA = 64 NATYP = 40 NPHB = 1
> > > > IFBOX = 2 NMXRS = 43 IFCAP = 0 NEXTRA = 0
> > > > NCOPY = 0
> > > >
> > > > | Coordinate Index Table dimensions: 23 23 23
> > > > | Direct force subcell size = 5.1644 5.1644 5.1644
> > > >
> > > > BOX TYPE: TRUNCATED OCTAHEDRON
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > -----------
> > > > 2. CONTROL DATA FOR THE RUN
> > > >
> > >
> > > ---------------------------------------------------------------------
> > -----------
> > > >
> > > >
> > >
> > >
> > > >
> > > > General flags:
> > > > imin = 0, nmropt = 0
> > > >
> > > > Nature and format of input:
> > > > ntx = 5, irest = 1, ntrx = 1
> > > >
> > > > Nature and format of output:
> > > > ntxo = 1, ntpr = 5000, ntrx = 1,
> > ntwr =
> > > 10000
> > > > iwrap = 1, ntwx = 5000, ntwv = 0, ntwe
> > > = 0
> > > > ioutfm = 0, ntwprt = 0, idecomp = 0,
> > > rbornstat= 0
> > > >
> > > > Potential function:
> > > > ntf = 2, ntb = 2, igb = 0, nsnb
> > > = 25
> > > > ipol = 0, gbsa = 0, iesp = 0
> > > > dielc = 1.00000, cut = 8.00000, intdiel = 1.00000
> > > >
> > > > Frozen or restrained atoms:
> > > > ibelly = 0, ntr = 0
> > > >
> > > > Molecular dynamics:
> > > > nstlim = 5000, nscm = 1000, nrespa = 1
> > > > t = 0.00000, dt = 0.00200, vlimit = -1.00000
> > > >
> > > > Langevin dynamics temperature regulation:
> > > > ig = 974683
> > > > temp0 = 300.00000, tempi = 0.00000, gamma_ln= 2.00000
> > > >
> > > > Pressure regulation:
> > > > ntp = 1
> > > > pres0 = 1.00000, comp = 44.60000, taup = 1.00000
> > > >
> > > > SHAKE:
> > > > ntc = 2, jfastw = 0
> > > > tol = 0.00001
> > > >
> > > > | Intermolecular bonds treatment:
> > > > | no_intermolecular_bonds = 1
> > > >
> > > > | Energy averages sample interval:
> > > > | ene_avg_sampling = 5000
> > > >
> > > > Ewald parameters:
> > > > verbose = 0, ew_type = 0, nbflag = 1,
> > use_pme
> > > = 1
> > > > vdwmeth = 1, eedmeth = 1, netfrc = 1
> > > > Box X = 118.781 Box Y = 118.781 Box Z = 118.781
> > > > Alpha = 109.471 Beta = 109.471 Gamma = 109.471
> > > > NFFT1 = 128 NFFT2 = 128 NFFT3 = 128
> > > > Cutoff= 8.000 Tol =0.100E-04
> > > > Ewald Coefficient = 0.34864
> > > > Interpolation order = 4
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > -----------
> > > > 3. ATOMIC COORDINATES AND VELOCITIES
> > > >
> > >
> > > ---------------------------------------------------------------------
> > -----------
> > > >
> > > >
> > >
> > >
> > > > begin time read from input coords = 10.000 ps
> > > >
> > > >
> > > > Number of triangulated 3-point waters found: 34583
> > > >
> > > > Sum of charges from parm topology file = -0.00000040
> > > > Forcing neutrality...
> > > >
> > > > | Dynamic Memory, Types Used:
> > > > | Reals 3524690
> > > > | Integers 3800219
> > > >
> > > > | Nonbonded Pairs Initial Allocation: 19420163
> > > >
> > > > | GPU memory information:
> > > > | KB of GPU memory in use: 882413
> > > > | KB of CPU memory in use: 104090
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > -----------
> > > > 4. RESULTS
> > > >
> > >
> > > ---------------------------------------------------------------------
> > -----------
> > > >
> > > > ---------------------------------------------------
> > > > APPROXIMATING switch and d/dx switch using CUBIC SPLINE
> > INTERPOLATION
> > > > using 5000.0 points per unit in tabled values
> > > > TESTING RELATIVE ERROR over r ranging from 0.0 to cutoff
> > > > | CHECK switch(x): max rel err = 0.2738E-14 at 2.422500
> > > > | CHECK d/dx switch(x): max rel err = 0.8332E-11 at 2.782960
> > > > ---------------------------------------------------
> > > > |---------------------------------------------------
> > > > | APPROXIMATING direct energy using CUBIC SPLINE INTERPOLATION
> > > > | with 50.0 points per unit in tabled values
> > > > | Relative Error Limit not exceeded for r .gt. 2.47
> > > > | APPROXIMATING direct force using CUBIC SPLINE INTERPOLATION
> > > > | with 50.0 points per unit in tabled values
> > > > | Relative Error Limit not exceeded for r .gt. 2.89
> > > > |---------------------------------------------------
> > > > wrapping first mol.: 38.333512154956900
> > > 54.211771142609109 93.897534410964738
> > > > wrapping first mol.: 38.333512154956900
> > > 54.211771142609109 93.897534410964738
> > > >
> > > > NSTEP = 5000 TIME(PS) = 20.000 TEMP(K)
> > = 300.01 PRESS =
> > > -23.7
> > > > Etot = -281399.8069 EKtot = 71193.9609 EPtot =
> > > -352593.7679
> > > > BOND = 2490.5718 ANGLE = 6429.0655 DIHED =
> > > 8582.5720
> > > > 1-4 NB = 2942.3115 1-4 EEL = 32655.1879 VDWAALS =
> > > 42104.9713
> > > > EELEC = -447798.4479 EHBOND = 0.0000 RESTRAINT =
> > > 0.0000
> > > > EKCMT = 30939.5460 VIRIAL = 31538.3575 VOLUME =
> > > 1170788.5879
> > > > Density =
> > > 1.0106
> > > >
> > >
> > > --------------------------------------------------------------------
> > ----------
> > > >
> > > >
> > > > A V E R A G E S O V E R 1 S T E P S
> > > >
> > > >
> > > > NSTEP = 5000 TIME(PS) = 20.000 TEMP(K)
> > = 300.01 PRESS =
> > > -23.7
> > > > Etot = -281399.8069 EKtot = 71193.9609 EPtot =
> > > -352593.7679
> > > > BOND = 2490.5718 ANGLE = 6429.0655 DIHED =
> > > 8582.5720
> > > > 1-4 NB = 2942.3115 1-4 EEL = 32655.1879 VDWAALS =
> > > 42104.9713
> > > > EELEC = -447798.4479 EHBOND = 0.0000 RESTRAINT =
> > > 0.0000
> > > > EKCMT = 30939.5460 VIRIAL = 31538.3575 VOLUME =
> > > 1170788.5879
> > > > Density =
> > > 1.0106
> > > >
> > >
> > > --------------------------------------------------------------------
> > ----------
> > > >
> > > >
> > > > R M S F L U C T U A T I O N S
> > > >
> > > >
> > > > NSTEP = 5000 TIME(PS) = 20.000 TEMP(K)
> > = 0.00 PRESS
> > > = 0.0
> > > > Etot = 0.0000 EKtot = 0.0000 EPtot =
> > > 0.0000
> > > > BOND = 0.0000 ANGLE = 0.0000 DIHED =
> > > 0.0000
> > > > 1-4 NB = 0.0000 1-4 EEL = 0.0000 VDWAALS =
> > > 0.0000
> > > > EELEC = 0.0000 EHBOND = 0.0000 RESTRAINT =
> > > 0.0000
> > > >
> > >
> > > --------------------------------------------------------------------
> > ----------
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > -----------
> > > > 5. TIMINGS
> > > >
> > >
> > > ---------------------------------------------------------------------
> > -----------
> > > >
> > > > | NonSetup CPU Time in Major Routines:
> > > > |
> > > > | Routine Sec %
> > > > | ------------------------------
> > > > | Nonbond 97.05 92.22
> > > > | Bond 0.00 0.00
> > > > | Angle 0.00 0.00
> > > > | Dihedral 0.00 0.00
> > > > | Shake 2.47 2.34
> > > > | RunMD 5.71 5.43
> > > > | Other 0.00 0.00
> > > > | ------------------------------
> > > > | Total 105.24
> > > >
> > > > | PME Nonbond Pairlist CPU Time:
> > > > |
> > > > | Routine Sec %
> > > > | ---------------------------------
> > > > | Set Up Cit 0.00 0.00
> > > > | Build List 0.00 0.00
> > > > | ---------------------------------
> > > > | Total 0.00 0.00
> > > >
> > > > | PME Direct Force CPU Time:
> > > > |
> > > > | Routine Sec %
> > > > | ---------------------------------
> > > > | NonBonded Calc 0.00 0.00
> > > > | Exclude Masked 0.00 0.00
> > > > | Other 0.00 0.00
> > > > | ---------------------------------
> > > > | Total 0.00 0.00
> > > >
> > > > | PME Reciprocal Force CPU Time:
> > > > |
> > > > | Routine Sec %
> > > > | ---------------------------------
> > > > | 1D bspline 0.00 0.00
> > > > | Grid Charges 0.00 0.00
> > > > | Scalar Sum 0.00 0.00
> > > > | Gradient Sum 0.00 0.00
> > > > | FFT 0.00 0.00
> > > > | ---------------------------------
> > > > | Total 0.00 0.00
> > > >
> > > > | Final Performance Info:
> > > > | -----------------------------------------------------
> > > > | Average timings for last 0 steps:
> > > > | Elapsed(s) = 0.00 Per Step(ms) = +Infinity
> > > > | ns/day = 0.00 seconds/ns = +Infinity
> > > > |
> > > > | Average timings for all steps:
> > > > | Elapsed(s) = 105.26 Per Step(ms) = 21.05
> > > > | ns/day = 8.21 seconds/ns = 10525.53
> > > > | -----------------------------------------------------
> > > >
> > > > | Setup CPU time: 0.90 seconds
> > > > | NonSetup CPU time: 105.24 seconds
> > > > | Total CPU time: 106.13 seconds 0.03 hours
> > > >
> > > > | Setup wall time: 1 seconds
> > > > | NonSetup wall time: 105 seconds
> > > > | Total wall time: 106 seconds 0.03 hours
> > > >
> > > >
> > > > heat Ligand9
> > > > &cntrl
> > > > irest=0, ntx=1,
> > > > nstlim=5000, dt=0.002,
> > > > ntc=2,ntf=2, iwrap=1,
> > > > cut=8.0, ntb=1, ig=-1,
> > > > ntpr=1000, ntwx=1000, ntwr=10000,
> > > > ntt=3, gamma_ln=2.0,
> > > > tempi=0.0, temp0=300.0,
> > > > ioutfm=1, ntr=1,
> > > > ntr=1,
> > > > /
> > > > Group input for restrained atoms
> > > > 2.0
> > > > RES 1 790
> > > > END
> > > > END
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > AMBER mailing list
> > > > AMBER.ambermd.org
> > > > http://lists.ambermd.org/mailman/listinfo/amber
> > > >
> > > >
> > > > _______________________________________________
> > > > AMBER mailing list
> > > > AMBER.ambermd.org
> > > > http://lists.ambermd.org/mailman/listinfo/amber
> > > _______________________________________________
> > > AMBER mailing list
> > > AMBER.ambermd.org
> > > http://lists.ambermd.org/mailman/listinfo/amber
> > >
> >
> >
> >
> > --
> > --
> > ==
> > Levi C.T. Pierce, UCSD Graduate Student
> > McCammon Laboratory
> > http://mccammon.ucsd.edu/
> > w: 858-534-2916
> > _______________________________________________
> > AMBER mailing list
> > AMBER.ambermd.org
> > http://lists.ambermd.org/mailman/listinfo/amber
> > _______________________________________________
> > AMBER mailing list
> > AMBER.ambermd.org
> > http://lists.ambermd.org/mailman/listinfo/amber
> > _______________________________________________
> > AMBER mailing list
> > AMBER.ambermd.org
> > http://lists.ambermd.org/mailman/listinfo/amber
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
>
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Sat Aug 20 2011 - 16:30:02 PDT
