PS this *might* indicate a software bug in cuFFT, but it needs more
characterization... And things are going to get a little stream of
consciousness from here because you're getting unfiltered raw data, so
please don't draw any conclusions towards anything yet - I'm just letting
you guys know what I'm finding out as I find it...
On Sun, Jun 2, 2013 at 9:31 AM, Scott Le Grand <varelse2005.gmail.com>wrote:
> And bingo...
>
> At the very least, the reciprocal sum is intermittently inconsistent...
> This explains the irreproducible behavior...
>
> And here's the level of inconsistency:
> 31989.38940628897399 vs
> 31989.39168370794505
>
> That's error at the level of 1e-7 or a somehow missed single-precision
> transaction somewhere...
>
> The next question is figuring out why... This may or may not ultimately
> explain the crashes you guys are also seeing...
>
>
>
> On Sun, Jun 2, 2013 at 9:07 AM, Scott Le Grand <varelse2005.gmail.com>wrote:
>
>>
>> Observations:
>> 1. The degree to which the reproducibility is broken *does* appear to
>> vary between individual Titan GPUs. One of my Titans breaks within 10K
>> steps on cellulose, the other one made it to 100K steps twice without doing
>> so leading me to believe it could be trusted (until yesterday where I now
>> see it dies between 50K and 100K steps most of the time).
>>
>> 2. GB hasn't broken (yet). So could you run myoglobin for 500K and
>> TRPcage for 1,000,000 steps and let's see if that's universal.
>>
>> 3. Turning on double-precision mode makes my Titan crash rather than run
>> irreproducibly, sigh...
>>
>> So whatever is going on is triggered by something in PME but not GB. So
>> that's either the radix sort, the FFT, the Ewald grid interpolation, or the
>> neighbor list code. Fixing this involves isolating this and figuring out
>> what exactly goes haywire. It could *still* be software at some very small
>> probability but the combination of both 680 and K20c with ECC off running
>> reliably is really pointing towards the Titans just being clocked too
>> fast.
>>
>> So how long with this take? Asking people how long it takes to fix a bug
>> never really works out well. That said, I found the 480 bug within a week
>> and my usual turnaround for a bug with a solid repro is <24 hours.
>>
>> Scott
>>
>> On Sun, Jun 2, 2013 at 7:58 AM, Marek Maly <marek.maly.ujep.cz> wrote:
>>
>>> Hi all,
>>>
>>> here are my results after bugfix 18 application (see attachment).
>>>
>>> In principle I don't see any "drastical" changes.
>>>
>>> FACTOR_IX still perfectly stable/reproducible on both cards,
>>>
>>> JAC tests - problems with finishing AND/OR reproducibility the
>>> same CELLULOSE_NVE although here it seems that my TITAN_1
>>> has no problems with this test (but the same same trend I saw also
>>> before bugfix 18 - see my older 500K steps test).
>>>
>>> But anyway bugfix 18 brought here one change.
>>>
>>> The err
>>>
>>>
>>> #1 ERR writtent in mdout:
>>> ------
>>> | ERROR: max pairlist cutoff must be less than unit cell max sphere
>>> radius!
>>> ------
>>>
>>> was substituted with err/warning ?
>>>
>>> #0 no ERR writtent in mdout, ERR written in standard output (nohup.out)
>>> -----
>>> Nonbond cells need to be recalculated, restart simulation from previous
>>> checkpoint
>>> with a higher value for skinnb.
>>>
>>> -----
>>>
>>> Another thing,
>>>
>>> recently I started on another machine and GTX 580 GPU simulation of
>>> relatively
>>> big system ( 364275 atoms/PME ). The system is composed also from the
>>> "exotic" molecules like polymers. ff12SB, gaff, GLYCAM forcefields used
>>> here. I had problem even with minimization part here, having big energy
>>> on the start:
>>>
>>> -----
>>> NSTEP ENERGY RMS GMAX NAME NUMBER
>>> 1 2.8442E+09 2.1339E+02 1.7311E+04 O 32998
>>>
>>> BOND = 11051.7467 ANGLE = 17720.4706 DIHED =
>>> 18977.7584
>>> VDWAALS = ************* EEL = -1257709.6203 HBOND =
>>> 0.0000
>>> 1-4 VDW = 7253.7412 1-4 EEL = 149867.0207 RESTRAINT =
>>> 0.0000
>>>
>>> ----
>>>
>>> with no chance to minimize the system even with 50 000 steps in both
>>> min cycles (with constrained and unconstrained solute) and hence heating
>>> NVT
>>> crashed immediately even with very small dt. I patched Amber12 here with
>>> the
>>> bugfix 18 and the minimization was done without any problem with common
>>> 5000 steps
>>> (obtaining target Energy -1.4505E+06 while that initial was that written
>>> above).
>>>
>>> So indeed bugfix 18 solved some issues, but unfortunately not those
>>> related to
>>> Titans.
>>>
>>> Here I will try to install cuda 5.5, recompile GPU Amber part with this
>>> new
>>> cuda version and repeat the 100K tests.
>>>
>>> Scott, let us know how finished your experiment with downclocking of
>>> Titan.
>>> Maybe the best choice would be here to flash Titan directly with your
>>> K20c bios :))
>>>
>>> M.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Dne Sat, 01 Jun 2013 21:09:46 +0200 Marek Maly <marek.maly.ujep.cz>
>>> napsal/-a:
>>>
>>>
>>> Hi,
>>>>
>>>> first of all thanks for providing of your test results !
>>>>
>>>> It seems that your results are more or less similar to that of
>>>> mine maybe with the exception of the results on FactorIX tests
>>>> where I had perfect stability and 100% or close to 100% reproducibility.
>>>>
>>>> Anyway the type of errs which you reported are the same which I
>>>> obtained.
>>>>
>>>> So let's see if the bugfix 18 will help here (or at least on NPT tests)
>>>> or not. As I wrote just before few minutes, it seems that it was not
>>>> still
>>>> loaded
>>>> to the given server, although it's description is already present on the
>>>> given
>>>> web page ( see http://ambermd.org/bugfixes12.**html<http://ambermd.org/bugfixes12.html>).
>>>>
>>>> As you can see, this bugfix contains also changes in CPU code although
>>>> the majority is devoted to GPU code, so perhaps the best will be to
>>>> recompile
>>>> whole amber with this patch although this patch would be perhaps applied
>>>> even after just
>>>> GPU configure command ( i.e. ./configure -cuda -noX11 gnu ) but after
>>>> consequent
>>>> building, just the GPU binaries will be updated. Anyway I would rather
>>>> recompile
>>>> whole Amber after this patch.
>>>>
>>>> Regarding to GPU test under linux you may try memtestG80
>>>> (please use the updated/patched version from here
>>>> https://github.com/ihaque/**memtestG80<https://github.com/ihaque/memtestG80>
>>>> )
>>>>
>>>> just use git command like:
>>>>
>>>> git clone https://github.com/ihaque/**memtestG80.git<https://github.com/ihaque/memtestG80.git>PATCHED_MEMTEST-G80
>>>>
>>>> to download all the files and save them into directory named
>>>> PATCHED_MEMTEST-G80.
>>>>
>>>> another possibility is to try perhaps similar (but maybe more up to
>>>> date)
>>>> test
>>>> cuda_memtest ( http://sourceforge.net/**projects/cudagpumemtest/<http://sourceforge.net/projects/cudagpumemtest/>).
>>>>
>>>> regarding ig value: If ig is not present in mdin, the default value is
>>>> used
>>>> (e.g. 71277) if ig=-1 the random seed will be based on the current date
>>>> and time, and hence will be different for every run (not a good variant
>>>> for our testts). I simply deleted eventual ig records from all mdins so
>>>> I
>>>> assume that in each run the default seed 71277 was automatically used.
>>>>
>>>> M.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Dne Sat, 01 Jun 2013 20:26:16 +0200 ET <sketchfoot.gmail.com>
>>>> napsal/-a:
>>>>
>>>> Hi,
>>>>>
>>>>> I've put the graphics card into a machine with the working GTX titan
>>>>> that I
>>>>> mentioned earlier.
>>>>>
>>>>> The Nvidia driver version is: 133.30
>>>>>
>>>>> Amber version is:
>>>>> AmberTools version 13.03
>>>>> Amber version 12.16
>>>>>
>>>>> I ran 50k steps with the amber benchmark using ig=43689 on both cards.
>>>>> For
>>>>> the purpose of discriminating between them, the card I believe (fingers
>>>>> crossed) is working is called GPU-00_TeaNCake, whilst the other one is
>>>>> called GPU-01_008.
>>>>>
>>>>> *When I run the tests on GPU-01_008:*
>>>>>
>>>>> 1) All the tests (across 2x repeats) finish apart from the following
>>>>> which
>>>>> have the errors listed:
>>>>>
>>>>> ------------------------------**--------------
>>>>> CELLULOSE_PRODUCTION_NVE - 408,609 atoms PME
>>>>> Error: unspecified launch failure launching kernel kNLSkinTest
>>>>> cudaFree GpuBuffer::Deallocate failed unspecified launch failure
>>>>>
>>>>> ------------------------------**--------------
>>>>> CELLULOSE_PRODUCTION_NPT - 408,609 atoms PME
>>>>> cudaMemcpy GpuBuffer::Download failed unspecified launch failure
>>>>>
>>>>> ------------------------------**--------------
>>>>> CELLULOSE_PRODUCTION_NVE - 408,609 atoms PME
>>>>> Error: unspecified launch failure launching kernel kNLSkinTest
>>>>> cudaFree GpuBuffer::Deallocate failed unspecified launch failure
>>>>>
>>>>> ------------------------------**--------------
>>>>> CELLULOSE_PRODUCTION_NPT - 408,609 atoms PME
>>>>> cudaMemcpy GpuBuffer::Download failed unspecified launch failure
>>>>> grep: mdinfo.1GTX680: No such file or directory
>>>>>
>>>>>
>>>>>
>>>>> 2) The sdiff logs indicate that reproducibility across the two repeats
>>>>> is
>>>>> as follows:
>>>>>
>>>>> *GB_myoglobin: *Reproducible across 50k steps
>>>>> *GB_nucleosome:* Reproducible till step 7400
>>>>> *GB_TRPCage:* Reproducible across 50k steps
>>>>>
>>>>> *PME_JAC_production_NVE: *No reproducibility shown from step 1,000
>>>>> onwards
>>>>> *PME_JAC_production_NPT*: Reproducible till step 1,000. Also outfile
>>>>> is
>>>>> not written properly - blank gaps appear where something should have
>>>>> been
>>>>> written
>>>>>
>>>>> *PME_FactorIX_production_NVE:* Reproducible across 50k steps
>>>>> *PME_FactorIX_production_NPT:* Reproducible across 50k steps
>>>>>
>>>>> *PME_Cellulose_production_NVE:*** Failure means that both runs do not
>>>>> finish
>>>>> (see point1)
>>>>> *PME_Cellulose_production_NPT: *Failure means that both runs do not
>>>>> finish
>>>>> (see point1)
>>>>>
>>>>> ##############################**##############################**
>>>>> ###########################
>>>>>
>>>>> *When I run the tests on * *GPU-00_TeaNCake:*
>>>>> *
>>>>> *
>>>>> 1) All the tests (across 2x repeats) finish apart from the following
>>>>> which
>>>>> have the errors listed:
>>>>> ------------------------------**-------
>>>>> JAC_PRODUCTION_NPT - 23,558 atoms PME
>>>>> PMEMD Terminated Abnormally!
>>>>> ------------------------------**-------
>>>>>
>>>>>
>>>>> 2) The sdiff logs indicate that reproducibility across the two repeats
>>>>> is
>>>>> as follows:
>>>>>
>>>>> *GB_myoglobin:* Reproducible across 50k steps
>>>>> *GB_nucleosome:* Reproducible across 50k steps
>>>>> *GB_TRPCage:* Reproducible across 50k steps
>>>>>
>>>>> *PME_JAC_production_NVE:* No reproducibility shown from step 10,000
>>>>> onwards
>>>>> *PME_JAC_production_NPT: * No reproducibility shown from step 10,000
>>>>> onwards. Also outfile is not written properly - blank gaps appear where
>>>>> something should have been written. Repeat 2 Crashes with error noted
>>>>> in
>>>>> 1.
>>>>>
>>>>> *PME_FactorIX_production_NVE:* No reproducibility shown from step 9,000
>>>>> onwards
>>>>> *PME_FactorIX_production_NPT: *Reproducible across 50k steps
>>>>>
>>>>> *PME_Cellulose_production_NVE: *No reproducibility shown from step
>>>>> 5,000
>>>>> onwards
>>>>> *PME_Cellulose_production_NPT: ** *No reproducibility shown from step
>>>>> 29,000 onwards. Also outfile is not written properly - blank gaps
>>>>> appear
>>>>> where something should have been written.
>>>>>
>>>>>
>>>>> Out files and sdiff files are included as attatchments
>>>>>
>>>>> ##############################**###################
>>>>>
>>>>> So I'm going to update my nvidia driver to the latest version and patch
>>>>> amber to the latest version and rerun the tests to see if there is any
>>>>> improvement. Could someone let me know if it is necessary to recompile
>>>>> any
>>>>> or all of AMBER after applying the bugfixes?
>>>>>
>>>>> Additionally, I'm going to run memory tests and heaven benchmarks on
>>>>> the
>>>>> cards to check whether they are faulty or not.
>>>>>
>>>>> I'm thinking that there is a mix of hardware error/configuration (esp
>>>>> in
>>>>> the case of GPU-01_008) and amber software error in this situation.
>>>>> What
>>>>> do
>>>>> you guys think?
>>>>>
>>>>> Also am I right in thinking (from what Scott was saying) that all the
>>>>> benchmarks should be reproducible across 50k steps but begin to diverge
>>>>> at
>>>>> around 100K steps? Is there any difference from in setting *ig *to an
>>>>> explicit number to removing it from the mdin file?
>>>>>
>>>>> br,
>>>>> g
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 31 May 2013 23:45, ET <sketchfoot.gmail.com> wrote:
>>>>>
>>>>> I don't need sysadmins, but sysadmins need me as it gives purpose to
>>>>>> their
>>>>>> bureaucratic existence. A encountered evil if working in an
>>>>>> institution
>>>>>> or
>>>>>> comapny IMO. Good science and indiviguality being sacrificed for
>>>>>> standardisation and mediocrity in the intrerests of maintaing a system
>>>>>> that
>>>>>> focusses on maintaining the system and not the objective.
>>>>>>
>>>>>> You need root to move fwd on these things, unfortunately. and ppl with
>>>>>> root are kinda like your parents when you try to borrow money from
>>>>>> them
>>>>>> .
>>>>>> age 12 :D
>>>>>> On May 31, 2013 9:34 PM, "Marek Maly" <marek.maly.ujep.cz> wrote:
>>>>>>
>>>>>> Sorry why do you need sysadmins :)) ?
>>>>>>>
>>>>>>> BTW here is the most recent driver:
>>>>>>>
>>>>>>> http://www.nvidia.com/object/**linux-display-amd64-319.23-**
>>>>>>> driver.html<http://www.nvidia.com/object/linux-display-amd64-319.23-driver.html>
>>>>>>>
>>>>>>> I do not remember anything easier than is to install driver
>>>>>>> (especially
>>>>>>> in case of binary (*.run) installer) :))
>>>>>>>
>>>>>>> M.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Dne Fri, 31 May 2013 22:02:34 +0200 ET <sketchfoot.gmail.com>
>>>>>>> napsal/-a:
>>>>>>>
>>>>>>> > Yup. I know. I replaced a 680 and the everknowing sysadmins are
>>>>>>> reluctant
>>>>>>> > to install drivers not in the repositoery as they are lame. :(
>>>>>>> > On May 31, 2013 7:14 PM, "Marek Maly" <marek.maly.ujep.cz> wrote:
>>>>>>> >>
>>>>>>> >> As I already wrote you,
>>>>>>> >>
>>>>>>> >> the first driver which properly/officially supports Titans, should
>>>>>>> be
>>>>>>> >> 313.26 .
>>>>>>> >>
>>>>>>> >> Anyway I am curious mainly about your 100K repetitive tests with
>>>>>>> >> your Titan SC card. Especially in case of these tests ( JAC_NVE,
>>>>>>> JAC_NPT
>>>>>>> >> and CELLULOSE_NVE ) where
>>>>>>> >> my Titans SC randomly failed or succeeded. In FACTOR_IX_NVE,
>>>>>>> >> FACTOR_IX_NPT
>>>>>>> >> tests both
>>>>>>> >> my cards are perfectly stable (independently from drv. version)
>>>>>>> and
>>>>>>> also
>>>>>>> >> the runs
>>>>>>> >> are perfectly or almost perfectly reproducible.
>>>>>>> >>
>>>>>>> >> Also if your test will crash please report the eventual errs.
>>>>>>> >>
>>>>>>> >> To this moment I have this actual library of errs on my Titans SC
>>>>>>> GPUs.
>>>>>>> >>
>>>>>>> >> #1 ERR writtent in mdout:
>>>>>>> >> ------
>>>>>>> >> | ERROR: max pairlist cutoff must be less than unit cell max
>>>>>>> sphere
>>>>>>> >> radius!
>>>>>>> >> ------
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> #2 no ERR writtent in mdout, ERR written in standard output
>>>>>>> (nohup.out)
>>>>>>> >>
>>>>>>> >> ----
>>>>>>> >> Error: unspecified launch failure launching kernel kNLSkinTest
>>>>>>> >> cudaFree GpuBuffer::Deallocate failed unspecified launch failure
>>>>>>> >> ----
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> #3 no ERR writtent in mdout, ERR written in standard output
>>>>>>> (nohup.out)
>>>>>>> >> ----
>>>>>>> >> cudaMemcpy GpuBuffer::Download failed unspecified launch failure
>>>>>>> >> ----
>>>>>>> >>
>>>>>>> >> Another question, regarding your Titan SC, it is also EVGA as in
>>>>>>> my
>>>>>>> case
>>>>>>> >> or it is another producer ?
>>>>>>> >>
>>>>>>> >> Thanks,
>>>>>>> >>
>>>>>>> >> M.
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> Dne Fri, 31 May 2013 19:17:03 +0200 ET <sketchfoot.gmail.com>
>>>>>>> napsal/-a:
>>>>>>> >>
>>>>>>> >> > Well, this is interesting...
>>>>>>> >> >
>>>>>>> >> > I ran 50k steps on the Titan on the other machine with driver
>>>>>>> 310.44
>>>>>>> >> and
>>>>>>> >> > it
>>>>>>> >> > passed all the GB steps. i.e totally identical results over two
>>>>>>> >> repeats.
>>>>>>> >> > However, it failed all the PME tests after step 1000. I'm going
>>>>>>> to
>>>>>>> > update
>>>>>>> >> > the driver and test it again.
>>>>>>> >> >
>>>>>>> >> > Files included as attachments.
>>>>>>> >> >
>>>>>>> >> > br,
>>>>>>> >> > g
>>>>>>> >> >
>>>>>>> >> >
>>>>>>> >> > On 31 May 2013 16:40, Marek Maly <marek.maly.ujep.cz> wrote:
>>>>>>> >> >
>>>>>>> >> >> One more thing,
>>>>>>> >> >>
>>>>>>> >> >> can you please check under which frequency is running that your
>>>>>>> >> titan ?
>>>>>>> >> >>
>>>>>>> >> >> As the base frequency of normal Titans is 837MHz and the Boost
>>>>>>> one
>>>>>>> is
>>>>>>> >> >> 876MHz I
>>>>>>> >> >> assume that yor GPU is running automatically also under it's
>>>>>>> boot
>>>>>>> >> >> frequency (876MHz).
>>>>>>> >> >> You can find this information e.g. in Amber mdout file.
>>>>>>> >> >>
>>>>>>> >> >> You also mentioned some crashes in your previous email. Your
>>>>>>> ERRs
>>>>>>> >> were
>>>>>>> >> >> something like those here:
>>>>>>> >> >>
>>>>>>> >> >> #1 ERR writtent in mdout:
>>>>>>> >> >> ------
>>>>>>> >> >> | ERROR: max pairlist cutoff must be less than unit cell max
>>>>>>> sphere
>>>>>>> >> >> radius!
>>>>>>> >> >> ------
>>>>>>> >> >>
>>>>>>> >> >>
>>>>>>> >> >> #2 no ERR writtent in mdout, ERR written in standard output
>>>>>>> >> (nohup.out)
>>>>>>> >> >>
>>>>>>> >> >> ----
>>>>>>> >> >> Error: unspecified launch failure launching kernel kNLSkinTest
>>>>>>> >> >> cudaFree GpuBuffer::Deallocate failed unspecified launch
>>>>>>> failure
>>>>>>> >> >> ----
>>>>>>> >> >>
>>>>>>> >> >>
>>>>>>> >> >> #3 no ERR writtent in mdout, ERR written in standard output
>>>>>>> >> (nohup.out)
>>>>>>> >> >> ----
>>>>>>> >> >> cudaMemcpy GpuBuffer::Download failed unspecified launch
>>>>>>> failure
>>>>>>> >> >> ----
>>>>>>> >> >>
>>>>>>> >> >> or you obtained some new/additional errs ?
>>>>>>> >> >>
>>>>>>> >> >>
>>>>>>> >> >>
>>>>>>> >> >> M.
>>>>>>> >> >>
>>>>>>> >> >>
>>>>>>> >> >>
>>>>>>> >> >> Dne Fri, 31 May 2013 17:30:57 +0200 filip fratev
>>>>>>> >> <filipfratev.yahoo.com
>>>>>>> >>
>>>>>>> >> >> napsal/-a:
>>>>>>> >> >>
>>>>>>> >> >> > Hi,
>>>>>>> >> >> > This is what I obtained for 50K tests and "normal" GTXTitan:
>>>>>>> >> >> >
>>>>>>> >> >> > run1:
>>>>>>> >> >> >
>>>>>>> >> >> >
>>>>>>> >> >> >
>>>>>>> >> >>
>>>>>>> >
>>>>>>> ------------------------------**------------------------------**
>>>>>>> ------------------
>>>>>>> >> >> >
>>>>>>> >> >> >
>>>>>>> >> >> > A V E R A G E S O V E R 50 S T E P S
>>>>>>> >> >> >
>>>>>>> >> >> >
>>>>>>> >> >> > NSTEP = 50000 TIME(PS) = 120.020 TEMP(K) =
>>>>>>> 299.87
>>>>>>> >> PRESS
>>>>>>> >> >> > = 0.0
>>>>>>> >> >> > Etot = -443237.1079 EKtot = 257679.9750 EPtot
>>>>>>> =
>>>>>>> >> >> > -700917.0829
>>>>>>> >> >> > BOND = 20193.1856 ANGLE = 53517.5432 DIHED
>>>>>>> =
>>>>>>> >> >> > 23575.4648
>>>>>>> >> >> > 1-4 NB = 21759.5524 1-4 EEL = 742552.5939 VDWAALS
>>>>>>> =
>>>>>>> >> >> > 96286.7714
>>>>>>> >> >> > EELEC = -1658802.1941 EHBOND = 0.0000 RESTRAINT
>>>>>>> =
>>>>>>> >> >> > 0.0000
>>>>>>> >> >> >
>>>>>>> >> >>
>>>>>>> >
>>>>>>> ------------------------------**------------------------------**
>>>>>>> ------------------
>>>>>>> >> >> >
>>>>>>> >> >> >
>>>>>>> >> >> > R M S F L U C T U A T I O N S
>>>>>>> >> >> >
>>>>>>> >> >> >
>>>>>>> >> >> > NSTEP = 50000 TIME(PS) = 120.020 TEMP(K) =
>>>>>>> 0.33
>>>>>>> >> PRESS
>>>>>>> >> >> > = 0.0
>>>>>>> >> >> > Etot = 11.2784 EKtot = 284.8999 EPtot
>>>>>>> =
>>>>>>> >> >> > 289.0773
>>>>>>> >> >> > BOND = 136.3417 ANGLE = 214.0054 DIHED
>>>>>>> =
>>>>>>> >> >> > 59.4893
>>>>>>> >> >> > 1-4 NB = 58.5891 1-4 EEL = 330.5400 VDWAALS
>>>>>>> =
>>>>>>> >> >> > 559.2079
>>>>>>> >> >> > EELEC = 743.8771 EHBOND = 0.0000 RESTRAINT
>>>>>>> =
>>>>>>> >> >> > 0.0000
>>>>>>> >> >> > |E(PBS) = 21.8119
>>>>>>> >> >> >
>>>>>>> >> >>
>>>>>>> >
>>>>>>> ------------------------------**------------------------------**
>>>>>>> ------------------
>>>>>>> >> >> >
>>>>>>> >> >> > run2:
>>>>>>> >> >> >
>>>>>>> >> >>
>>>>>>> >
>>>>>>> ------------------------------**------------------------------**
>>>>>>> ------------------
>>>>>>> >> >> >
>>>>>>> >> >> >
>>>>>>> >> >> > A V E R A G E S O V E R 50 S T E P S
>>>>>>> >> >> >
>>>>>>> >> >> >
>>>>>>> >> >> > NSTEP = 50000 TIME(PS) = 120.020 TEMP(K) =
>>>>>>> 299.89
>>>>>>> >> PRESS
>>>>>>> >> >> > = 0.0
>>>>>>> >> >> > Etot = -443240.0999 EKtot = 257700.0950 EPtot
>>>>>>> =
>>>>>>> >> >> > -700940.1949
>>>>>>> >> >> > BOND = 20241.9174 ANGLE = 53644.6694 DIHED
>>>>>>> =
>>>>>>> >> >> > 23541.3737
>>>>>>> >> >> > 1-4 NB = 21803.1898 1-4 EEL = 742754.2254 VDWAALS
>>>>>>> =
>>>>>>> >> >> > 96298.8308
>>>>>>> >> >> > EELEC = -1659224.4013 EHBOND = 0.0000 RESTRAINT
>>>>>>> =
>>>>>>> >> >> > 0.0000
>>>>>>> >> >> >
>>>>>>> >> >>
>>>>>>> >
>>>>>>> ------------------------------**------------------------------**
>>>>>>> ------------------
>>>>>>> >> >> >
>>>>>>> >> >> >
>>>>>>> >> >> > R M S F L U C T U A T I O N S
>>>>>>> >> >> >
>>>>>>> >> >> >
>>>>>>> >> >> > NSTEP = 50000 TIME(PS) = 120.020 TEMP(K) =
>>>>>>> 0.41
>>>>>>> >> PRESS
>>>>>>> >> >> > = 0.0
>>>>>>> >> >> > Etot = 10.7633 EKtot = 348.2819 EPtot
>>>>>>> =
>>>>>>> >> >> > 353.9918
>>>>>>> >> >> > BOND = 106.5314 ANGLE = 196.7052 DIHED
>>>>>>> =
>>>>>>> >> >> > 69.7476
>>>>>>> >> >> > 1-4 NB = 60.3435 1-4 EEL = 400.7466 VDWAALS
>>>>>>> =
>>>>>>> >> >> > 462.7763
>>>>>>> >> >> > EELEC = 651.9857 EHBOND = 0.0000 RESTRAINT
>>>>>>> =
>>>>>>> >> >> > 0.0000
>>>>>>> >> >> > |E(PBS) = 17.0642
>>>>>>> >> >> >
>>>>>>> >> >>
>>>>>>> >
>>>>>>> ------------------------------**------------------------------**
>>>>>>> ------------------
>>>>>>> >> >> >
>>>>>>> >> >> >
>>>>>>> >> >>
>>>>>>> >
>>>>>>> ------------------------------**------------------------------**
>>>>>>> --------------------
>>>>>>> >> >> >
>>>>>>> >> >> >
>>>>>>> >> >> >
>>>>>>> >> >> >
>>>>>>> >> >> > ______________________________**__
>>>>>>> >> >> > From: Marek Maly <marek.maly.ujep.cz>
>>>>>>> >> >> > To: AMBER Mailing List <amber.ambermd.org>
>>>>>>> >> >> > Sent: Friday, May 31, 2013 3:34 PM
>>>>>>> >> >> > Subject: Re: [AMBER] experiences with EVGA GTX TITAN
>>>>>>> Superclocked
>>>>>>> -
>>>>>>> >> >> > memtestG80 - UNDERclocking in Linux ?
>>>>>>> >> >> >
>>>>>>> >> >> > Hi here are my 100K results for driver 313.30 (and still Cuda
>>>>>>> 5.0).
>>>>>>> >> >> >
>>>>>>> >> >> > The results are rather similar to those obtained
>>>>>>> >> >> > under my original driver 319.17 (see the first table
>>>>>>> >> >> > which I sent in this thread).
>>>>>>> >> >> >
>>>>>>> >> >> > M.
>>>>>>> >> >> >
>>>>>>> >> >> >
>>>>>>> >> >> > Dne Fri, 31 May 2013 12:29:59 +0200 Marek Maly <
>>>>>>> marek.maly.ujep.cz>
>>>>>>> >> >> > napsal/-a:
>>>>>>> >> >> >
>>>>>>> >> >> >> Hi,
>>>>>>> >> >> >>
>>>>>>> >> >> >> please try to run at lest 100K tests twice to verify exact
>>>>>>> >> >> >> reproducibility
>>>>>>> >> >> >> of the results on the given card. If you find in any mdin
>>>>>>> file
>>>>>>> >> ig=-1
>>>>>>> >> >> >> just
>>>>>>> >> >> >> delete it to ensure that you are using the identical random
>>>>>>> seed
>>>>>>> >> for
>>>>>>> >> >> >> both
>>>>>>> >> >> >> runs. You can eventually omit NUCLEOSOME test
>>>>>>> >> >> >> as it is too much time consuming.
>>>>>>> >> >> >>
>>>>>>> >> >> >> Driver 310.44 ?????
>>>>>>> >> >> >>
>>>>>>> >> >> >> As far as I know the proper support for titans is from
>>>>>>> version
>>>>>>> > 313.26
>>>>>>> >> >> >>
>>>>>>> >> >> >> see e.g. here :
>>>>>>> >> >> >>
>>>>>>> >> >>
>>>>>>> >
>>>>>>> http://www.geeks3d.com/**20130306/nvidia-releases-r313-**
>>>>>>> 26-for-linux-with-gtx-titan-**support/<http://www.geeks3d.com/20130306/nvidia-releases-r313-26-for-linux-with-gtx-titan-support/>
>>>>>>> >> >> >>
>>>>>>> >> >> >> BTW: On my site downgrade to drv. 313.30 did not solved the
>>>>>>> >> >> situation, I
>>>>>>> >> >> >> will post
>>>>>>> >> >> >> my results soon here.
>>>>>>> >> >> >>
>>>>>>> >> >> >> M.
>>>>>>> >> >> >>
>>>>>>> >> >> >>
>>>>>>> >> >> >>
>>>>>>> >> >> >>
>>>>>>> >> >> >>
>>>>>>> >> >> >>
>>>>>>> >> >> >>
>>>>>>> >> >> >>
>>>>>>> >> >> >> Dne Fri, 31 May 2013 12:21:21 +0200 ET <
>>>>>>> sketchfoot.gmail.com>
>>>>>>> >> >> napsal/-a:
>>>>>>> >> >> >>
>>>>>>> >> >> >>> ps. I have another install of amber on another computer
>>>>>>> with
>>>>>>> a
>>>>>>> >> >> >>> different
>>>>>>> >> >> >>> Titan and different Driver Version: 310.44.
>>>>>>> >> >> >>>
>>>>>>> >> >> >>> In the interests of thrashing the proverbial horse, I'll
>>>>>>> run
>>>>>>> the
>>>>>>> >> >> >>> benchmark
>>>>>>> >> >> >>> for 50k steps. :P
>>>>>>> >> >> >>>
>>>>>>> >> >> >>> br,
>>>>>>> >> >> >>> g
>>>>>>> >> >> >>>
>>>>>>> >> >> >>>
>>>>>>> >> >> >>> On 31 May 2013 11:17, ET <sketchfoot.gmail.com> wrote:
>>>>>>> >> >> >>>
>>>>>>> >> >> >>>> Hi, I just ran the Amber benchmark for the default (10000
>>>>>>> steps)
>>>>>>> >> >> on my
>>>>>>> >> >> >>>> Titan.
>>>>>>> >> >> >>>>
>>>>>>> >> >> >>>> Using sdiff -sB showed that the two runs were completely
>>>>>>> > identical.
>>>>>>> >> >> >>>> I've
>>>>>>> >> >> >>>> attached compressed files of the mdout & diff files.
>>>>>>> >> >> >>>>
>>>>>>> >> >> >>>> br,
>>>>>>> >> >> >>>> g
>>>>>>> >> >> >>>>
>>>>>>> >> >> >>>>
>>>>>>> >> >> >>>> On 30 May 2013 23:41, Marek Maly <marek.maly.ujep.cz>
>>>>>>> wrote:
>>>>>>> >> >> >>>>
>>>>>>> >> >> >>>>> OK, let's see. The eventual downclocking I see as the
>>>>>>> very
>>>>>>> last
>>>>>>> >> >> >>>>> possibility
>>>>>>> >> >> >>>>> (if I don't decide for RMAing). But now still some other
>>>>>>> >> >> experiments
>>>>>>> >> >> >>>>> are
>>>>>>> >> >> >>>>> available :))
>>>>>>> >> >> >>>>> I just started 100K tests under 313.30 driver. For today
>>>>>>> good
>>>>>>> >> >> night
>>>>>>> >> >> >>>>> ...
>>>>>>> >> >> >>>>>
>>>>>>> >> >> >>>>> M.
>>>>>>> >> >> >>>>>
>>>>>>> >> >> >>>>> Dne Fri, 31 May 2013 00:45:49 +0200 Scott Le Grand
>>>>>>> >> >> >>>>> <varelse2005.gmail.com
>>>>>>> >> >> >>>>> >
>>>>>>> >> >> >>>>> napsal/-a:
>>>>>>> >> >> >>>>>
>>>>>>> >> >> >>>>> > It will be very interesting if this behavior persists
>>>>>>> after
>>>>>>> >> >> >>>>> downclocking.
>>>>>>> >> >> >>>>> >
>>>>>>> >> >> >>>>> > But right now, Titan 0 *looks* hosed and Titan 1
>>>>>>> *looks*
>>>>>>> like
>>>>>>> > it
>>>>>>> >> >> >>>>> needs
>>>>>>> >> >> >>>>> > downclocking...
>>>>>>> >> >> >>>>> > On May 30, 2013 3:20 PM, "Marek Maly"
>>>>>>> <marek.maly.ujep.cz>
>>>>>>> >> >> wrote:
>>>>>>> >> >> >>>>> >
>>>>>>> >> >> >>>>> >> Hi all,
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >> here are my results from the 500K steps 2 x repeated
>>>>>>> > benchmarks
>>>>>>> >> >> >>>>> >> under 319.23 driver and still Cuda 5.0 (see the
>>>>>>> attached
>>>>>>> >> table
>>>>>>> >> >> ).
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >> It is hard to say if the results are better or worse
>>>>>>> than
>>>>>>> in
>>>>>>> > my
>>>>>>> >> >> >>>>> >> previous 100K test under driver 319.17.
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >> While results from Cellulose test were improved and
>>>>>>> the
>>>>>>> > TITAN_1
>>>>>>> >> >> >>>>> card
>>>>>>> >> >> >>>>> >> even
>>>>>>> >> >> >>>>> >> successfully finished all 500K steps moreover with
>>>>>>> exactly
>>>>>>> >> the
>>>>>>> >> >> >>>>> same
>>>>>>> >> >> >>>>> >> final
>>>>>>> >> >> >>>>> >> energy !
>>>>>>> >> >> >>>>> >> (TITAN_0 at least finished more than 100K steps and in
>>>>>>> >> RUN_01
>>>>>>> >> >> even
>>>>>>> >> >> >>>>> more
>>>>>>> >> >> >>>>> >> than 400K steps)
>>>>>>> >> >> >>>>> >> In JAC_NPT test no GPU was able to finish at least
>>>>>>> 100K
>>>>>>> >> steps
>>>>>>> >> >> and
>>>>>>> >> >> >>>>> the
>>>>>>> >> >> >>>>> >> results from JAC_NVE
>>>>>>> >> >> >>>>> >> test are also not too much convincing. FACTOR_IX_NVE
>>>>>>> and
>>>>>>> >> >> >>>>> FACTOR_IX_NPT
>>>>>>> >> >> >>>>> >> were successfully
>>>>>>> >> >> >>>>> >> finished with 100% reproducibility in FACTOR_IX_NPT
>>>>>>> case
>>>>>>> >> (on
>>>>>>> >> >> both
>>>>>>> >> >> >>>>> >> cards)
>>>>>>> >> >> >>>>> >> and almost
>>>>>>> >> >> >>>>> >> 100% reproducibility in case of FACTOR_IX_NVE (again
>>>>>>> 100%
>>>>>>> in
>>>>>>> >> >> case
>>>>>>> >> >> >>>>> of
>>>>>>> >> >> >>>>> >> TITAN_1). TRPCAGE, MYOGLOBIN
>>>>>>> >> >> >>>>> >> again finished without any problem with 100%
>>>>>>> >> reproducibility.
>>>>>>> >> >> >>>>> NUCLEOSOME
>>>>>>> >> >> >>>>> >> test was not done
>>>>>>> >> >> >>>>> >> this time due to high time requirements. If you find
>>>>>>> in
>>>>>>> the
>>>>>>> >> >> table
>>>>>>> >> >> >>>>> >> positive
>>>>>>> >> >> >>>>> >> number finishing with
>>>>>>> >> >> >>>>> >> K (which means "thousands") it means the last number
>>>>>>> of
>>>>>>> step
>>>>>>> >> >> >>>>> written in
>>>>>>> >> >> >>>>> >> mdout before crash.
>>>>>>> >> >> >>>>> >> Below are all the 3 types of detected errs with
>>>>>>> relevant
>>>>>>> >> >> >>>>> systems/rounds
>>>>>>> >> >> >>>>> >> where the given err
>>>>>>> >> >> >>>>> >> appeared.
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >> Now I will try just 100K tests under ETs favourite
>>>>>>> driver
>>>>>>> >> >> version
>>>>>>> >> >> >>>>> 313.30
>>>>>>> >> >> >>>>> >> :)) and then
>>>>>>> >> >> >>>>> >> I will eventually try to experiment with cuda 5.5
>>>>>>> which
>>>>>>> I
>>>>>>> >> >> already
>>>>>>> >> >> >>>>> >> downloaded from the
>>>>>>> >> >> >>>>> >> cuda zone ( I had to become cuda developer for this
>>>>>>> :))
>>>>>>> )
>>>>>>> >> BTW
>>>>>>> >> >> ET
>>>>>>> >> >> >>>>> thanks
>>>>>>> >> >> >>>>> >> for the frequency info !
>>>>>>> >> >> >>>>> >> and I am still ( perhaps not alone :)) ) very curious
>>>>>>> about
>>>>>>> >> >> your 2
>>>>>>> >> >> >>>>> x
>>>>>>> >> >> >>>>> >> repeated Amber benchmark tests with superclocked
>>>>>>> Titan.
>>>>>>> >> Indeed
>>>>>>> >> >> >>>>> that
>>>>>>> >> >> >>>>> I
>>>>>>> >> >> >>>>> am
>>>>>>> >> >> >>>>> >> very curious also about that Ross "hot" patch.
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >> M.
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >> ERRORS DETECTED DURING THE 500K steps tests with
>>>>>>> driver
>>>>>>> >> 319.23
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >> #1 ERR writtent in mdout:
>>>>>>> >> >> >>>>> >> ------
>>>>>>> >> >> >>>>> >> | ERROR: max pairlist cutoff must be less than unit
>>>>>>> cell
>>>>>>> >> max
>>>>>>> >> >> >>>>> sphere
>>>>>>> >> >> >>>>> >> radius!
>>>>>>> >> >> >>>>> >> ------
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >> TITAN_0 ROUND_1 JAC_NPT (at least 5000 steps
>>>>>>> successfully
>>>>>>> > done
>>>>>>> >> >> >>>>> before
>>>>>>> >> >> >>>>> >> crash)
>>>>>>> >> >> >>>>> >> TITAN_0 ROUND_2 JAC_NPT (at least 8000 steps
>>>>>>> successfully
>>>>>>> > done
>>>>>>> >> >> >>>>> before
>>>>>>> >> >> >>>>> >> crash)
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >> #2 no ERR writtent in mdout, ERR written in standard
>>>>>>> output
>>>>>>> >> >> >>>>> (nohup.out)
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >> ----
>>>>>>> >> >> >>>>> >> Error: unspecified launch failure launching kernel
>>>>>>> >> kNLSkinTest
>>>>>>> >> >> >>>>> >> cudaFree GpuBuffer::Deallocate failed unspecified
>>>>>>> launch
>>>>>>> >> >> failure
>>>>>>> >> >> >>>>> >> ----
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >> TITAN_0 ROUND_1 CELLULOSE_NVE (at least 437 000 steps
>>>>>>> >> >> successfully
>>>>>>> >> >> >>>>> done
>>>>>>> >> >> >>>>> >> before crash)
>>>>>>> >> >> >>>>> >> TITAN_0 ROUND_2 JAC_NVE (at least 162 000 steps
>>>>>>> >> successfully
>>>>>>> >> >> done
>>>>>>> >> >> >>>>> >> before
>>>>>>> >> >> >>>>> >> crash)
>>>>>>> >> >> >>>>> >> TITAN_0 ROUND_2 CELLULOSE_NVE (at least 117 000 steps
>>>>>>> >> >> successfully
>>>>>>> >> >> >>>>> done
>>>>>>> >> >> >>>>> >> before crash)
>>>>>>> >> >> >>>>> >> TITAN_1 ROUND_1 JAC_NVE (at least 119 000 steps
>>>>>>> >> successfully
>>>>>>> >> >> done
>>>>>>> >> >> >>>>> >> before
>>>>>>> >> >> >>>>> >> crash)
>>>>>>> >> >> >>>>> >> TITAN_1 ROUND_2 JAC_NVE (at least 43 000 steps
>>>>>>> successfully
>>>>>>> >> >> done
>>>>>>> >> >> >>>>> before
>>>>>>> >> >> >>>>> >> crash)
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >> #3 no ERR writtent in mdout, ERR written in standard
>>>>>>> output
>>>>>>> >> >> >>>>> (nohup.out)
>>>>>>> >> >> >>>>> >> ----
>>>>>>> >> >> >>>>> >> cudaMemcpy GpuBuffer::Download failed unspecified
>>>>>>> launch
>>>>>>> >> >> failure
>>>>>>> >> >> >>>>> >> ----
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >> TITAN_1 ROUND_1 JAC_NPT (at least 77 000 steps
>>>>>>> successfully
>>>>>>> >> >> done
>>>>>>> >> >> >>>>> before
>>>>>>> >> >> >>>>> >> crash)
>>>>>>> >> >> >>>>> >> TITAN_1 ROUND_2 JAC_NPT (at least 58 000 steps
>>>>>>> successfully
>>>>>>> >> >> done
>>>>>>> >> >> >>>>> before
>>>>>>> >> >> >>>>> >> crash)
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >> Dne Thu, 30 May 2013 21:27:17 +0200 Scott Le Grand
>>>>>>> >> >> >>>>> >> <varelse2005.gmail.com>
>>>>>>> >> >> >>>>> >> napsal/-a:
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >> Oops meant to send that to Jason...
>>>>>>> >> >> >>>>> >>>
>>>>>>> >> >> >>>>> >>> Anyway, before we all panic, we need to get K20's
>>>>>>> behavior
>>>>>>> >> >> >>>>> analyzed
>>>>>>> >> >> >>>>> >>> here.
>>>>>>> >> >> >>>>> >>> If it's deterministic, this truly is a hardware
>>>>>>> issue. If
>>>>>>> >> >> not,
>>>>>>> >> >> >>>>> then
>>>>>>> >> >> >>>>> it
>>>>>>> >> >> >>>>> >>> gets interesting because 680 is deterministic as far
>>>>>>> as I
>>>>>>> >> can
>>>>>>> >> >> >>>>> tell...
>>>>>>> >> >> >>>>> >>> On May 30, 2013 12:24 PM, "Scott Le Grand"
>>>>>>> >> >> >>>>> <varelse2005.gmail.com>
>>>>>>> >> >> >>>>> >>> wrote:
>>>>>>> >> >> >>>>> >>>
>>>>>>> >> >> >>>>> >>> If the errors are not deterministically triggered,
>>>>>>> they
>>>>>>> >> >> probably
>>>>>>> >> >> >>>>> >>> won't be
>>>>>>> >> >> >>>>> >>>> fixed by the patch, alas...
>>>>>>> >> >> >>>>> >>>> On May 30, 2013 12:15 PM, "Jason Swails"
>>>>>>> >> >> >>>>> <jason.swails.gmail.com>
>>>>>>> >> >> >>>>> >>>> wrote:
>>>>>>> >> >> >>>>> >>>>
>>>>>>> >> >> >>>>> >>>> Just a reminder to everyone based on what Ross
>>>>>>> said:
>>>>>>> >> there
>>>>>>> >> >> is a
>>>>>>> >> >> >>>>> >>>> pending
>>>>>>> >> >> >>>>> >>>>> patch to pmemd.cuda that will be coming out shortly
>>>>>>> >> (maybe
>>>>>>> >> >> even
>>>>>>> >> >> >>>>> >>>>> within
>>>>>>> >> >> >>>>> >>>>> hours). It's entirely possible that several of
>>>>>>> these
>>>>>>> > errors
>>>>>>> >> >> >>>>> are
>>>>>>> >> >> >>>>> >>>>> fixed
>>>>>>> >> >> >>>>> >>>>> by
>>>>>>> >> >> >>>>> >>>>> this patch.
>>>>>>> >> >> >>>>> >>>>>
>>>>>>> >> >> >>>>> >>>>> All the best,
>>>>>>> >> >> >>>>> >>>>> Jason
>>>>>>> >> >> >>>>> >>>>>
>>>>>>> >> >> >>>>> >>>>>
>>>>>>> >> >> >>>>> >>>>> On Thu, May 30, 2013 at 2:46 PM, filip fratev <
>>>>>>> >> >> >>>>> filipfratev.yahoo.com>
>>>>>>> >> >> >>>>> >>>>> wrote:
>>>>>>> >> >> >>>>> >>>>>
>>>>>>> >> >> >>>>> >>>>> > I have observed the same crashes from time to
>>>>>>> time. I
>>>>>>> > will
>>>>>>> >> >> >>>>> run
>>>>>>> >> >> >>>>> >>>>> cellulose
>>>>>>> >> >> >>>>> >>>>> > nve for 100k and will past results here.
>>>>>>> >> >> >>>>> >>>>> >
>>>>>>> >> >> >>>>> >>>>> > All the best,
>>>>>>> >> >> >>>>> >>>>> > Filip
>>>>>>> >> >> >>>>> >>>>> >
>>>>>>> >> >> >>>>> >>>>> >
>>>>>>> >> >> >>>>> >>>>> >
>>>>>>> >> >> >>>>> >>>>> >
>>>>>>> >> >> >>>>> >>>>> > ______________________________****__
>>>>>>> >> >> >>>>> >>>>> > From: Scott Le Grand <varelse2005.gmail.com>
>>>>>>> >> >> >>>>> >>>>> > To: AMBER Mailing List <amber.ambermd.org>
>>>>>>> >> >> >>>>> >>>>> > Sent: Thursday, May 30, 2013 9:01 PM
>>>>>>> >> >> >>>>> >>>>> > Subject: Re: [AMBER] experiences with EVGA GTX
>>>>>>> TITAN
>>>>>>> >> >> >>>>> Superclocked
>>>>>>> >> >> >>>>> -
>>>>>>> >> >> >>>>> >>>>> > memtestG80 - UNDERclocking in Linux ?
>>>>>>> >> >> >>>>> >>>>> >
>>>>>>> >> >> >>>>> >>>>> >
>>>>>>> >> >> >>>>> >>>>> > Run cellulose nve for 100k iterations twice . If
>>>>>>> the
>>>>>>> >> >> final
>>>>>>> >> >> >>>>> >>>>> energies
>>>>>>> >> >> >>>>> >>>>> don't
>>>>>>> >> >> >>>>> >>>>> > match, you have a hardware issue. No need to
>>>>>>> play
>>>>>>> with
>>>>>>> >> >> ntpr
>>>>>>> >> >> >>>>> or
>>>>>>> >> >> >>>>> any
>>>>>>> >> >> >>>>> >>>>> other
>>>>>>> >> >> >>>>> >>>>> > variable.
>>>>>>> >> >> >>>>> >>>>> > On May 30, 2013 10:58 AM, <pavel.banas.upol.cz>
>>>>>>> wrote:
>>>>>>> >> >> >>>>> >>>>> >
>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>> >> >> >>>>> >>>>> > > Dear all,
>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>> >> >> >>>>> >>>>> > > I would also like to share one of my experience
>>>>>>> with
>>>>>>> >> >> titan
>>>>>>> >> >> >>>>> >>>>> cards. We
>>>>>>> >> >> >>>>> >>>>> have
>>>>>>> >> >> >>>>> >>>>> > > one gtx titan card and with one system (~55k
>>>>>>> atoms,
>>>>>>> > NVT,
>>>>>>> >> >> >>>>> >>>>> RNA+waters)
>>>>>>> >> >> >>>>> >>>>> we
>>>>>>> >> >> >>>>> >>>>> > run
>>>>>>> >> >> >>>>> >>>>> > > into same troubles you are describing. I was
>>>>>>> also
>>>>>>> >> >> playing
>>>>>>> >> >> >>>>> with
>>>>>>> >> >> >>>>> >>>>> ntpr
>>>>>>> >> >> >>>>> >>>>> to
>>>>>>> >> >> >>>>> >>>>> > > figure out what is going on, step by step. I
>>>>>>> >> understand
>>>>>>> >> >> >>>>> that
>>>>>>> >> >> >>>>> the
>>>>>>> >> >> >>>>> >>>>> code
>>>>>>> >> >> >>>>> >>>>> is
>>>>>>> >> >> >>>>> >>>>> > > using different routines for calculation
>>>>>>> >> >> energies+forces or
>>>>>>> >> >> >>>>> only
>>>>>>> >> >> >>>>> >>>>> forces.
>>>>>>> >> >> >>>>> >>>>> > > The
>>>>>>> >> >> >>>>> >>>>> > > simulations of other systems are perfectly
>>>>>>> stable,
>>>>>>> >> >> running
>>>>>>> >> >> >>>>> for
>>>>>>> >> >> >>>>> >>>>> days
>>>>>>> >> >> >>>>> >>>>> and
>>>>>>> >> >> >>>>> >>>>> > > weeks. Only that particular system
>>>>>>> systematically
>>>>>>> >> ends
>>>>>>> >> >> up
>>>>>>> >> >> >>>>> with
>>>>>>> >> >> >>>>> >>>>> this
>>>>>>> >> >> >>>>> >>>>> > error.
>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>> >> >> >>>>> >>>>> > > However, there was one interesting issue. When
>>>>>>> I
>>>>>>> set
>>>>>>> >> >> >>>>> ntpr=1,
>>>>>>> >> >> >>>>> the
>>>>>>> >> >> >>>>> >>>>> error
>>>>>>> >> >> >>>>> >>>>> > > vanished (systematically in multiple runs) and
>>>>>>> the
>>>>>>> >> >> >>>>> simulation
>>>>>>> >> >> >>>>> was
>>>>>>> >> >> >>>>> >>>>> able to
>>>>>>> >> >> >>>>> >>>>> > > run for more than millions of steps (I was not
>>>>>>> let
>>>>>>> it
>>>>>>> >> >> >>>>> running
>>>>>>> >> >> >>>>> for
>>>>>>> >> >> >>>>> >>>>> weeks
>>>>>>> >> >> >>>>> >>>>> > as
>>>>>>> >> >> >>>>> >>>>> > > in the meantime I shifted that simulation to
>>>>>>> other
>>>>>>> >> card
>>>>>>> >> >> -
>>>>>>> >> >> >>>>> need
>>>>>>> >> >> >>>>> >>>>> data,
>>>>>>> >> >> >>>>> >>>>> not
>>>>>>> >> >> >>>>> >>>>> > > testing). All other setting of ntpr failed. As
>>>>>>> I
>>>>>>> read
>>>>>>> >> >> this
>>>>>>> >> >> >>>>> >>>>> discussion, I
>>>>>>> >> >> >>>>> >>>>> > > tried to set ene_avg_sampling=1 with some high
>>>>>>> value
>>>>>>> >> of
>>>>>>> >> >> >>>>> ntpr
>>>>>>> >> >> >>>>> (I
>>>>>>> >> >> >>>>> >>>>> expected
>>>>>>> >> >> >>>>> >>>>> > > that this will shift the code to permanently
>>>>>>> use
>>>>>>> the
>>>>>>> >> >> >>>>> >>>>> force+energies
>>>>>>> >> >> >>>>> >>>>> part
>>>>>>> >> >> >>>>> >>>>> > of
>>>>>>> >> >> >>>>> >>>>> > > the code, similarly to ntpr=1), but the error
>>>>>>> >> occurred
>>>>>>> >> >> >>>>> again.
>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>> >> >> >>>>> >>>>> > > I know it is not very conclusive for finding
>>>>>>> out
>>>>>>> what
>>>>>>> > is
>>>>>>> >> >> >>>>> >>>>> happening,
>>>>>>> >> >> >>>>> >>>>> at
>>>>>>> >> >> >>>>> >>>>> > > least
>>>>>>> >> >> >>>>> >>>>> > > not for me. Do you have any idea, why ntpr=1
>>>>>>> might
>>>>>>> > help?
>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>> >> >> >>>>> >>>>> > > best regards,
>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>> >> >> >>>>> >>>>> > > Pavel
>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>> >> >> >>>>> >>>>> > > --
>>>>>>> >> >> >>>>> >>>>> > > Pavel Banáš
>>>>>>> >> >> >>>>> >>>>> > > pavel.banas.upol.cz
>>>>>>> >> >> >>>>> >>>>> > > Department of Physical Chemistry,
>>>>>>> >> >> >>>>> >>>>> > > Palacky University Olomouc
>>>>>>> >> >> >>>>> >>>>> > > Czech Republic
>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>> >> >> >>>>> >>>>> > > ---------- Původní zpráva ----------
>>>>>>> >> >> >>>>> >>>>> > > Od: Jason Swails <jason.swails.gmail.com>
>>>>>>> >> >> >>>>> >>>>> > > Datum: 29. 5. 2013
>>>>>>> >> >> >>>>> >>>>> > > Předmět: Re: [AMBER] experiences with EVGA GTX
>>>>>>> TITAN
>>>>>>> >> >> >>>>> >>>>> Superclocked -
>>>>>>> >> >> >>>>> >>>>> > > memtestG
>>>>>>> >> >> >>>>> >>>>> > > 80 - UNDERclocking in Linux ?
>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>> >> >> >>>>> >>>>> > > "I'll answer a little bit:
>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>> >> >> >>>>> >>>>> > > NTPR=10 Etot after 2000 steps
>>>>>>> >> >> >>>>> >>>>> > > >
>>>>>>> >> >> >>>>> >>>>> > > > -443256.6711
>>>>>>> >> >> >>>>> >>>>> > > > -443256.6711
>>>>>>> >> >> >>>>> >>>>> > > >
>>>>>>> >> >> >>>>> >>>>> > > > NTPR=200 Etot after 2000 steps
>>>>>>> >> >> >>>>> >>>>> > > >
>>>>>>> >> >> >>>>> >>>>> > > > -443261.0705
>>>>>>> >> >> >>>>> >>>>> > > > -443261.0705
>>>>>>> >> >> >>>>> >>>>> > > >
>>>>>>> >> >> >>>>> >>>>> > > > Any idea why energies should depend on
>>>>>>> frequency
>>>>>>> of
>>>>>>> >> >> >>>>> energy
>>>>>>> >> >> >>>>> >>>>> records
>>>>>>> >> >> >>>>> >>>>> > (NTPR)
>>>>>>> >> >> >>>>> >>>>> > > ?
>>>>>>> >> >> >>>>> >>>>> > > >
>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>> >> >> >>>>> >>>>> > > It is a subtle point, but the answer is
>>>>>>> 'different
>>>>>>> >> code
>>>>>>> >> >> >>>>> paths.'
>>>>>>> >> >> >>>>> >>>>> In
>>>>>>> >> >> >>>>> >>>>> > > general, it is NEVER necessary to compute the
>>>>>>> actual
>>>>>>> >> >> energy
>>>>>>> >> >> >>>>> of a
>>>>>>> >> >> >>>>> >>>>> molecule
>>>>>>> >> >> >>>>> >>>>> > > during the course of standard molecular
>>>>>>> dynamics
>>>>>>> (by
>>>>>>> >> >> >>>>> analogy, it
>>>>>>> >> >> >>>>> >>>>> is
>>>>>>> >> >> >>>>> >>>>> NEVER
>>>>>>> >> >> >>>>> >>>>> > > necessary to compute atomic forces during the
>>>>>>> course
>>>>>>> >> of
>>>>>>> >> >> >>>>> random
>>>>>>> >> >> >>>>> >>>>> Monte
>>>>>>> >> >> >>>>> >>>>> > Carlo
>>>>>>> >> >> >>>>> >>>>> > > sampling).
>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>> >> >> >>>>> >>>>> > > For performance's sake, then, pmemd.cuda
>>>>>>> computes
>>>>>>> >> only
>>>>>>> >> >> the
>>>>>>> >> >> >>>>> force
>>>>>>> >> >> >>>>> >>>>> when
>>>>>>> >> >> >>>>> >>>>> > > energies are not requested, leading to a
>>>>>>> different
>>>>>>> >> >> order of
>>>>>>> >> >> >>>>> >>>>> operations
>>>>>>> >> >> >>>>> >>>>> > for
>>>>>>> >> >> >>>>> >>>>> > > those runs. This difference ultimately causes
>>>>>>> >> >> divergence.
>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>> >> >> >>>>> >>>>> > > To test this, try setting the variable
>>>>>>> >> >> ene_avg_sampling=10
>>>>>>> >> >> >>>>> in
>>>>>>> >> >> >>>>> the
>>>>>>> >> >> >>>>> >>>>> &cntrl
>>>>>>> >> >> >>>>> >>>>> > > section. This will force pmemd.cuda to compute
>>>>>>> >> energies
>>>>>>> >> >> >>>>> every 10
>>>>>>> >> >> >>>>> >>>>> steps
>>>>>>> >> >> >>>>> >>>>> > > (for energy averaging), which will in turn make
>>>>>>> the
>>>>>>> >> >> >>>>> followed
>>>>>>> >> >> >>>>> code
>>>>>>> >> >> >>>>> >>>>> path
>>>>>>> >> >> >>>>> >>>>> > > identical for any multiple-of-10 value of ntpr.
>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>> >> >> >>>>> >>>>> > > --
>>>>>>> >> >> >>>>> >>>>> > > Jason M. Swails
>>>>>>> >> >> >>>>> >>>>> > > Quantum Theory Project,
>>>>>>> >> >> >>>>> >>>>> > > University of Florida
>>>>>>> >> >> >>>>> >>>>> > > Ph.D. Candidate
>>>>>>> >> >> >>>>> >>>>> > > 352-392-4032
>>>>>>> >> >> >>>>> >>>>> > > ______________________________**
>>>>>>> **_________________
>>>>>>> >> >> >>>>> >>>>> > > AMBER mailing list
>>>>>>> >> >> >>>>> >>>>> > > AMBER.ambermd.org
>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>> >> >> >>>>> >>>>> http://lists.ambermd.org/****
>>>>>>> mailman/listinfo/amber<http://lists.ambermd.org/**mailman/listinfo/amber>
>>>>>>> <
>>>>>>> >> >> >>>>> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>> >
>>>>>>> >> >> >>>>> >>>>> "
>>>>>>> >> >> >>>>> >>>>> > > ______________________________**
>>>>>>> **_________________
>>>>>>> >> >> >>>>> >>>>> > > AMBER mailing list
>>>>>>> >> >> >>>>> >>>>> > > AMBER.ambermd.org
>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>> >> >> >>>>> >>>>> http://lists.ambermd.org/****
>>>>>>> mailman/listinfo/amber<http://lists.ambermd.org/**mailman/listinfo/amber>
>>>>>>> <
>>>>>>> >> >> >>>>> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>> >
>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>> >> >> >>>>> >>>>> > ______________________________**
>>>>>>> **_________________
>>>>>>> >> >> >>>>> >>>>> > AMBER mailing list
>>>>>>> >> >> >>>>> >>>>> > AMBER.ambermd.org
>>>>>>> >> >> >>>>> >>>>> >
>>>>>>> >> >> >>>>> >>>>> http://lists.ambermd.org/****
>>>>>>> mailman/listinfo/amber<http://lists.ambermd.org/**mailman/listinfo/amber>
>>>>>>> <
>>>>>>> >> >> >>>>> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>> >
>>>>>>> >> >> >>>>> >>>>> > ______________________________**
>>>>>>> **_________________
>>>>>>> >> >> >>>>> >>>>> > AMBER mailing list
>>>>>>> >> >> >>>>> >>>>> > AMBER.ambermd.org
>>>>>>> >> >> >>>>> >>>>> >
>>>>>>> >> >> >>>>> >>>>> http://lists.ambermd.org/****
>>>>>>> mailman/listinfo/amber<http://lists.ambermd.org/**mailman/listinfo/amber>
>>>>>>> <
>>>>>>> >> >> >>>>> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>> >
>>>>>>> >> >> >>>>> >>>>> >
>>>>>>> >> >> >>>>> >>>>>
>>>>>>> >> >> >>>>> >>>>>
>>>>>>> >> >> >>>>> >>>>>
>>>>>>> >> >> >>>>> >>>>> --
>>>>>>> >> >> >>>>> >>>>> Jason M. Swails
>>>>>>> >> >> >>>>> >>>>> Quantum Theory Project,
>>>>>>> >> >> >>>>> >>>>> University of Florida
>>>>>>> >> >> >>>>> >>>>> Ph.D. Candidate
>>>>>>> >> >> >>>>> >>>>> 352-392-4032
>>>>>>> >> >> >>>>> >>>>> ______________________________**
>>>>>>> **_________________
>>>>>>> >> >> >>>>> >>>>> AMBER mailing list
>>>>>>> >> >> >>>>> >>>>> AMBER.ambermd.org
>>>>>>> >> >> >>>>> >>>>> http://lists.ambermd.org/****
>>>>>>> mailman/listinfo/amber<http://lists.ambermd.org/**mailman/listinfo/amber>
>>>>>>> <
>>>>>>> >> >> >>>>> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>> >
>>>>>>> >> >> >>>>> >>>>>
>>>>>>> >> >> >>>>> >>>>>
>>>>>>> >> >> >>>>> >>>> ______________________________**
>>>>>>> **_________________
>>>>>>> >> >> >>>>> >>> AMBER mailing list
>>>>>>> >> >> >>>>> >>> AMBER.ambermd.org
>>>>>>> >> >> >>>>> >>> http://lists.ambermd.org/****mailman/listinfo/amber<http://lists.ambermd.org/**mailman/listinfo/amber>
>>>>>>> <
>>>>>>> >> >> >>>>> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>> >
>>>>>>> >> >> >>>>> >>>
>>>>>>> >> >> >>>>> >>> __________ Informace od ESET NOD32 Antivirus, verze
>>>>>>> >> databaze
>>>>>>> >> >> 8394
>>>>>>> >> >> >>>>> >>> (20130530) __________
>>>>>>> >> >> >>>>> >>>
>>>>>>> >> >> >>>>> >>> Tuto zpravu proveril ESET NOD32 Antivirus.
>>>>>>> >> >> >>>>> >>>
>>>>>>> >> >> >>>>> >>> http://www.eset.cz
>>>>>>> >> >> >>>>> >>>
>>>>>>> >> >> >>>>> >>>
>>>>>>> >> >> >>>>> >>>
>>>>>>> >> >> >>>>> >>>
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >> --
>>>>>>> >> >> >>>>> >> Tato zpráva byla vytvořena převratným poštovním
>>>>>>> klientem
>>>>>>> > Opery:
>>>>>>> >> >> >>>>> >> http://www.opera.com/mail/
>>>>>>> >> >> >>>>> >> ______________________________**_________________
>>>>>>> >> >> >>>>> >> AMBER mailing list
>>>>>>> >> >> >>>>> >> AMBER.ambermd.org
>>>>>>> >> >> >>>>> >> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> >>
>>>>>>> >> >> >>>>> > ______________________________**_________________
>>>>>>> >> >> >>>>> > AMBER mailing list
>>>>>>> >> >> >>>>> > AMBER.ambermd.org
>>>>>>> >> >> >>>>> > http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>> >> >> >>>>> >
>>>>>>> >> >> >>>>> > __________ Informace od ESET NOD32 Antivirus, verze
>>>>>>> databaze
>>>>>>> >> >> 8394
>>>>>>> >> >> >>>>> > (20130530) __________
>>>>>>> >> >> >>>>> >
>>>>>>> >> >> >>>>> > Tuto zpravu proveril ESET NOD32 Antivirus.
>>>>>>> >> >> >>>>> >
>>>>>>> >> >> >>>>> > http://www.eset.cz
>>>>>>> >> >> >>>>> >
>>>>>>> >> >> >>>>> >
>>>>>>> >> >> >>>>> >
>>>>>>> >> >> >>>>>
>>>>>>> >> >> >>>>>
>>>>>>> >> >> >>>>> --
>>>>>>> >> >> >>>>> Tato zpráva byla vytvořena převratným poštovním klientem
>>>>>>> Opery:
>>>>>>> >> >> >>>>> http://www.opera.com/mail/
>>>>>>> >> >> >>>>>
>>>>>>> >> >> >>>>> ______________________________**_________________
>>>>>>> >> >> >>>>> AMBER mailing list
>>>>>>> >> >> >>>>> AMBER.ambermd.org
>>>>>>> >> >> >>>>> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>> >> >> >>>>>
>>>>>>> >> >> >>>>
>>>>>>> >> >> >>>>
>>>>>>> >> >> >>> ______________________________**_________________
>>>>>>> >> >> >>> AMBER mailing list
>>>>>>> >> >> >>> AMBER.ambermd.org
>>>>>>> >> >> >>> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>> >> >> >>>
>>>>>>> >> >> >>> __________ Informace od ESET NOD32 Antivirus, verze
>>>>>>> databaze
>>>>>>> 8395
>>>>>>> >> >> >>> (20130531) __________
>>>>>>> >> >> >>>
>>>>>>> >> >> >>> Tuto zpravu proveril ESET NOD32 Antivirus.
>>>>>>> >> >> >>>
>>>>>>> >> >> >>> http://www.eset.cz
>>>>>>> >> >> >>>
>>>>>>> >> >> >>>
>>>>>>> >> >> >>>
>>>>>>> >> >> >>
>>>>>>> >> >> >>
>>>>>>> >> >> >
>>>>>>> >> >> >
>>>>>>> >> >>
>>>>>>> >> >>
>>>>>>> >> >> --
>>>>>>> >> >> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
>>>>>>> >> >> http://www.opera.com/mail/
>>>>>>> >> >>
>>>>>>> >> >> ______________________________**_________________
>>>>>>> >> >> AMBER mailing list
>>>>>>> >> >> AMBER.ambermd.org
>>>>>>> >> >> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>> >> >>
>>>>>>> >> >
>>>>>>> >> >
>>>>>>> >> >
>>>>>>> >> > __________ Informace od ESET NOD32 Antivirus, verze databaze
>>>>>>> 8397
>>>>>>> >> > (20130531) __________
>>>>>>> >> >
>>>>>>> >> > Tuto zpravu proveril ESET NOD32 Antivirus.
>>>>>>> >> >
>>>>>>> >> > GB_out_plus_diff_Files.tar.gz - poskozeny archiv
>>>>>>> >> > GB_out_plus_diff_Files.tar.gz > GZIP >
>>>>>>> >> GB_out_plus_diff_Files.tar
>>>>>>> >> > - poskozeny archiv
>>>>>>> >> > GB_out_plus_diff_Files.tar.gz > GZIP >
>>>>>>> >> > GB_out_plus_diff_Files.tar > TAR >
>>>>>>> GB_out_plus_diff_Files.tar.gz -
>>>>>>> >> > poskozeny archiv
>>>>>>> >> > GB_out_plus_diff_Files.tar.gz > GZIP >
>>>>>>> >> > GB_out_plus_diff_Files.tar > TAR >
>>>>>>> GB_out_plus_diff_Files.tar.gz >
>>>>>>> >> GZIP
>>>>>>> >> > > GB_out_plus_diff_Files.tar - poskozeny archiv
>>>>>>> >> > GB_out_plus_diff_Files.tar.gz > GZIP >
>>>>>>> >> > GB_out_plus_diff_Files.tar > TAR >
>>>>>>> GB_out_plus_diff_Files.tar.gz >
>>>>>>> >> GZIP
>>>>>>> >> > > GB_out_plus_diff_Files.tar > TAR >
>>>>>>> GB_nucleosome-sim3.mdout-full -
>>>>>>> >> > vyskytl se problem pri cteni archivu
>>>>>>> >> > PME_out_plus_diff_Files.tar.gz - poskozeny archiv
>>>>>>> >> > PME_out_plus_diff_Files.tar.gz > GZIP >
>>>>>>> >> > PME_out_plus_diff_Files.tar - poskozeny archiv
>>>>>>> >> > PME_out_plus_diff_Files.tar.gz > GZIP >
>>>>>>> >> > PME_out_plus_diff_Files.tar > TAR >
>>>>>>> PME_out_plus_diff_Files.tar.gz -
>>>>>>> >> > poskozeny archiv
>>>>>>> >> > PME_out_plus_diff_Files.tar.gz > GZIP >
>>>>>>> >> > PME_out_plus_diff_Files.tar > TAR >
>>>>>>> PME_out_plus_diff_Files.tar.gz >
>>>>>>> >> > GZIP > PME_out_plus_diff_Files.tar - poskozeny archiv
>>>>>>> >> > PME_out_plus_diff_Files.tar.gz > GZIP >
>>>>>>> >> > PME_out_plus_diff_Filestar > TAR >
>>>>>>> PME_out_plus_diff_Files.tar.gz
>>>>>>> >
>>>>>>> >> GZIP
>>>>>>> >> > > PME_out_plus_diff_Files.tar > TAR >
>>>>>>> >> > PME_JAC_production_NPT-sim3.**mdout-full - vyskytl se problem
>>>>>>> pri
>>>>>>> cteni
>>>>>>> >> > archivu
>>>>>>> >> >
>>>>>>> >> > http://www.eset.cz
>>>>>>> >> >
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> --
>>>>>>> >> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
>>>>>>> >> http://www.opera.com/mail/
>>>>>>> >>
>>>>>>> >> ______________________________**_________________
>>>>>>> >> AMBER mailing list
>>>>>>> >> AMBER.ambermd.org
>>>>>>> >> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>> > ______________________________**_________________
>>>>>>> > AMBER mailing list
>>>>>>> > AMBER.ambermd.org
>>>>>>> > http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>> >
>>>>>>> > __________ Informace od ESET NOD32 Antivirus, verze databaze 8398
>>>>>>> > (20130531) __________
>>>>>>> >
>>>>>>> > Tuto zpravu proveril ESET NOD32 Antivirus.
>>>>>>> >
>>>>>>> > http://www.eset.cz
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
>>>>>>> http://www.opera.com/mail/
>>>>>>>
>>>>>>> ______________________________**_________________
>>>>>>> AMBER mailing list
>>>>>>> AMBER.ambermd.org
>>>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> __________ Informace od ESET NOD32 Antivirus, verze databaze 8401
>>>>> (20130601) __________
>>>>>
>>>>> Tuto zpravu proveril ESET NOD32 Antivirus.
>>>>>
>>>>> http://www.eset.cz
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> --
>>> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
>>> http://www.opera.com/mail/
>>>
>>> _______________________________________________
>>> AMBER mailing list
>>> AMBER.ambermd.org
>>> http://lists.ambermd.org/mailman/listinfo/amber
>>>
>>>
>>
>
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Sun Jun 02 2013 - 10:00:03 PDT
