PPS My testbed also repros crashing so at least I have things isolated...
On Sun, Jun 2, 2013 at 9:33 AM, Scott Le Grand <varelse2005.gmail.com>wrote:
> PS this *might* indicate a software bug in cuFFT, but it needs more
> characterization... And things are going to get a little stream of
> consciousness from here because you're getting unfiltered raw data, so
> please don't draw any conclusions towards anything yet - I'm just letting
> you guys know what I'm finding out as I find it...
>
>
>
> On Sun, Jun 2, 2013 at 9:31 AM, Scott Le Grand <varelse2005.gmail.com>wrote:
>
>> And bingo...
>>
>> At the very least, the reciprocal sum is intermittently inconsistent...
>> This explains the irreproducible behavior...
>>
>> And here's the level of inconsistency:
>> 31989.38940628897399 vs
>> 31989.39168370794505
>>
>> That's error at the level of 1e-7 or a somehow missed single-precision
>> transaction somewhere...
>>
>> The next question is figuring out why... This may or may not ultimately
>> explain the crashes you guys are also seeing...
>>
>>
>>
>> On Sun, Jun 2, 2013 at 9:07 AM, Scott Le Grand <varelse2005.gmail.com>wrote:
>>
>>>
>>> Observations:
>>> 1. The degree to which the reproducibility is broken *does* appear to
>>> vary between individual Titan GPUs. One of my Titans breaks within 10K
>>> steps on cellulose, the other one made it to 100K steps twice without doing
>>> so leading me to believe it could be trusted (until yesterday where I now
>>> see it dies between 50K and 100K steps most of the time).
>>>
>>> 2. GB hasn't broken (yet). So could you run myoglobin for 500K and
>>> TRPcage for 1,000,000 steps and let's see if that's universal.
>>>
>>> 3. Turning on double-precision mode makes my Titan crash rather than run
>>> irreproducibly, sigh...
>>>
>>> So whatever is going on is triggered by something in PME but not GB. So
>>> that's either the radix sort, the FFT, the Ewald grid interpolation, or the
>>> neighbor list code. Fixing this involves isolating this and figuring out
>>> what exactly goes haywire. It could *still* be software at some very small
>>> probability but the combination of both 680 and K20c with ECC off running
>>> reliably is really pointing towards the Titans just being clocked too
>>> fast.
>>>
>>> So how long with this take? Asking people how long it takes to fix a
>>> bug never really works out well. That said, I found the 480 bug within a
>>> week and my usual turnaround for a bug with a solid repro is <24 hours.
>>>
>>> Scott
>>>
>>> On Sun, Jun 2, 2013 at 7:58 AM, Marek Maly <marek.maly.ujep.cz> wrote:
>>>
>>>> Hi all,
>>>>
>>>> here are my results after bugfix 18 application (see attachment).
>>>>
>>>> In principle I don't see any "drastical" changes.
>>>>
>>>> FACTOR_IX still perfectly stable/reproducible on both cards,
>>>>
>>>> JAC tests - problems with finishing AND/OR reproducibility the
>>>> same CELLULOSE_NVE although here it seems that my TITAN_1
>>>> has no problems with this test (but the same same trend I saw also
>>>> before bugfix 18 - see my older 500K steps test).
>>>>
>>>> But anyway bugfix 18 brought here one change.
>>>>
>>>> The err
>>>>
>>>>
>>>> #1 ERR writtent in mdout:
>>>> ------
>>>> | ERROR: max pairlist cutoff must be less than unit cell max sphere
>>>> radius!
>>>> ------
>>>>
>>>> was substituted with err/warning ?
>>>>
>>>> #0 no ERR writtent in mdout, ERR written in standard output (nohup.out)
>>>> -----
>>>> Nonbond cells need to be recalculated, restart simulation from previous
>>>> checkpoint
>>>> with a higher value for skinnb.
>>>>
>>>> -----
>>>>
>>>> Another thing,
>>>>
>>>> recently I started on another machine and GTX 580 GPU simulation of
>>>> relatively
>>>> big system ( 364275 atoms/PME ). The system is composed also from the
>>>> "exotic" molecules like polymers. ff12SB, gaff, GLYCAM forcefields used
>>>> here. I had problem even with minimization part here, having big energy
>>>> on the start:
>>>>
>>>> -----
>>>> NSTEP ENERGY RMS GMAX NAME
>>>> NUMBER
>>>> 1 2.8442E+09 2.1339E+02 1.7311E+04 O 32998
>>>>
>>>> BOND = 11051.7467 ANGLE = 17720.4706 DIHED =
>>>> 18977.7584
>>>> VDWAALS = ************* EEL = -1257709.6203 HBOND =
>>>> 0.0000
>>>> 1-4 VDW = 7253.7412 1-4 EEL = 149867.0207 RESTRAINT =
>>>> 0.0000
>>>>
>>>> ----
>>>>
>>>> with no chance to minimize the system even with 50 000 steps in both
>>>> min cycles (with constrained and unconstrained solute) and hence
>>>> heating NVT
>>>> crashed immediately even with very small dt. I patched Amber12 here
>>>> with the
>>>> bugfix 18 and the minimization was done without any problem with common
>>>> 5000 steps
>>>> (obtaining target Energy -1.4505E+06 while that initial was that
>>>> written above).
>>>>
>>>> So indeed bugfix 18 solved some issues, but unfortunately not those
>>>> related to
>>>> Titans.
>>>>
>>>> Here I will try to install cuda 5.5, recompile GPU Amber part with
>>>> this new
>>>> cuda version and repeat the 100K tests.
>>>>
>>>> Scott, let us know how finished your experiment with downclocking of
>>>> Titan.
>>>> Maybe the best choice would be here to flash Titan directly with your
>>>> K20c bios :))
>>>>
>>>> M.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Dne Sat, 01 Jun 2013 21:09:46 +0200 Marek Maly <marek.maly.ujep.cz>
>>>> napsal/-a:
>>>>
>>>>
>>>> Hi,
>>>>>
>>>>> first of all thanks for providing of your test results !
>>>>>
>>>>> It seems that your results are more or less similar to that of
>>>>> mine maybe with the exception of the results on FactorIX tests
>>>>> where I had perfect stability and 100% or close to 100%
>>>>> reproducibility.
>>>>>
>>>>> Anyway the type of errs which you reported are the same which I
>>>>> obtained.
>>>>>
>>>>> So let's see if the bugfix 18 will help here (or at least on NPT tests)
>>>>> or not. As I wrote just before few minutes, it seems that it was not
>>>>> still
>>>>> loaded
>>>>> to the given server, although it's description is already present on
>>>>> the
>>>>> given
>>>>> web page ( see http://ambermd.org/bugfixes12.**html<http://ambermd.org/bugfixes12.html>).
>>>>>
>>>>> As you can see, this bugfix contains also changes in CPU code although
>>>>> the majority is devoted to GPU code, so perhaps the best will be to
>>>>> recompile
>>>>> whole amber with this patch although this patch would be perhaps
>>>>> applied
>>>>> even after just
>>>>> GPU configure command ( i.e. ./configure -cuda -noX11 gnu ) but after
>>>>> consequent
>>>>> building, just the GPU binaries will be updated. Anyway I would rather
>>>>> recompile
>>>>> whole Amber after this patch.
>>>>>
>>>>> Regarding to GPU test under linux you may try memtestG80
>>>>> (please use the updated/patched version from here
>>>>> https://github.com/ihaque/**memtestG80<https://github.com/ihaque/memtestG80>
>>>>> )
>>>>>
>>>>> just use git command like:
>>>>>
>>>>> git clone https://github.com/ihaque/**memtestG80.git<https://github.com/ihaque/memtestG80.git>PATCHED_MEMTEST-G80
>>>>>
>>>>> to download all the files and save them into directory named
>>>>> PATCHED_MEMTEST-G80.
>>>>>
>>>>> another possibility is to try perhaps similar (but maybe more up to
>>>>> date)
>>>>> test
>>>>> cuda_memtest ( http://sourceforge.net/**projects/cudagpumemtest/<http://sourceforge.net/projects/cudagpumemtest/>).
>>>>>
>>>>> regarding ig value: If ig is not present in mdin, the default value is
>>>>> used
>>>>> (e.g. 71277) if ig=-1 the random seed will be based on the current date
>>>>> and time, and hence will be different for every run (not a good variant
>>>>> for our testts). I simply deleted eventual ig records from all mdins
>>>>> so I
>>>>> assume that in each run the default seed 71277 was automatically used.
>>>>>
>>>>> M.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Dne Sat, 01 Jun 2013 20:26:16 +0200 ET <sketchfoot.gmail.com>
>>>>> napsal/-a:
>>>>>
>>>>> Hi,
>>>>>>
>>>>>> I've put the graphics card into a machine with the working GTX titan
>>>>>> that I
>>>>>> mentioned earlier.
>>>>>>
>>>>>> The Nvidia driver version is: 133.30
>>>>>>
>>>>>> Amber version is:
>>>>>> AmberTools version 13.03
>>>>>> Amber version 12.16
>>>>>>
>>>>>> I ran 50k steps with the amber benchmark using ig=43689 on both cards.
>>>>>> For
>>>>>> the purpose of discriminating between them, the card I believe
>>>>>> (fingers
>>>>>> crossed) is working is called GPU-00_TeaNCake, whilst the other one is
>>>>>> called GPU-01_008.
>>>>>>
>>>>>> *When I run the tests on GPU-01_008:*
>>>>>>
>>>>>> 1) All the tests (across 2x repeats) finish apart from the following
>>>>>> which
>>>>>> have the errors listed:
>>>>>>
>>>>>> ------------------------------**--------------
>>>>>> CELLULOSE_PRODUCTION_NVE - 408,609 atoms PME
>>>>>> Error: unspecified launch failure launching kernel kNLSkinTest
>>>>>> cudaFree GpuBuffer::Deallocate failed unspecified launch failure
>>>>>>
>>>>>> ------------------------------**--------------
>>>>>> CELLULOSE_PRODUCTION_NPT - 408,609 atoms PME
>>>>>> cudaMemcpy GpuBuffer::Download failed unspecified launch failure
>>>>>>
>>>>>> ------------------------------**--------------
>>>>>> CELLULOSE_PRODUCTION_NVE - 408,609 atoms PME
>>>>>> Error: unspecified launch failure launching kernel kNLSkinTest
>>>>>> cudaFree GpuBuffer::Deallocate failed unspecified launch failure
>>>>>>
>>>>>> ------------------------------**--------------
>>>>>> CELLULOSE_PRODUCTION_NPT - 408,609 atoms PME
>>>>>> cudaMemcpy GpuBuffer::Download failed unspecified launch failure
>>>>>> grep: mdinfo.1GTX680: No such file or directory
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2) The sdiff logs indicate that reproducibility across the two
>>>>>> repeats
>>>>>> is
>>>>>> as follows:
>>>>>>
>>>>>> *GB_myoglobin: *Reproducible across 50k steps
>>>>>> *GB_nucleosome:* Reproducible till step 7400
>>>>>> *GB_TRPCage:* Reproducible across 50k steps
>>>>>>
>>>>>> *PME_JAC_production_NVE: *No reproducibility shown from step 1,000
>>>>>> onwards
>>>>>> *PME_JAC_production_NPT*: Reproducible till step 1,000. Also outfile
>>>>>> is
>>>>>> not written properly - blank gaps appear where something should have
>>>>>> been
>>>>>> written
>>>>>>
>>>>>> *PME_FactorIX_production_NVE:* Reproducible across 50k steps
>>>>>> *PME_FactorIX_production_NPT:* Reproducible across 50k steps
>>>>>>
>>>>>> *PME_Cellulose_production_NVE:*** Failure means that both runs do not
>>>>>> finish
>>>>>> (see point1)
>>>>>> *PME_Cellulose_production_NPT: *Failure means that both runs do not
>>>>>> finish
>>>>>> (see point1)
>>>>>>
>>>>>> ##############################**##############################**
>>>>>> ###########################
>>>>>>
>>>>>> *When I run the tests on * *GPU-00_TeaNCake:*
>>>>>> *
>>>>>> *
>>>>>> 1) All the tests (across 2x repeats) finish apart from the following
>>>>>> which
>>>>>> have the errors listed:
>>>>>> ------------------------------**-------
>>>>>> JAC_PRODUCTION_NPT - 23,558 atoms PME
>>>>>> PMEMD Terminated Abnormally!
>>>>>> ------------------------------**-------
>>>>>>
>>>>>>
>>>>>> 2) The sdiff logs indicate that reproducibility across the two
>>>>>> repeats
>>>>>> is
>>>>>> as follows:
>>>>>>
>>>>>> *GB_myoglobin:* Reproducible across 50k steps
>>>>>> *GB_nucleosome:* Reproducible across 50k steps
>>>>>> *GB_TRPCage:* Reproducible across 50k steps
>>>>>>
>>>>>> *PME_JAC_production_NVE:* No reproducibility shown from step 10,000
>>>>>> onwards
>>>>>> *PME_JAC_production_NPT: * No reproducibility shown from step 10,000
>>>>>> onwards. Also outfile is not written properly - blank gaps appear
>>>>>> where
>>>>>> something should have been written. Repeat 2 Crashes with error noted
>>>>>> in
>>>>>> 1.
>>>>>>
>>>>>> *PME_FactorIX_production_NVE:* No reproducibility shown from step
>>>>>> 9,000
>>>>>> onwards
>>>>>> *PME_FactorIX_production_NPT: *Reproducible across 50k steps
>>>>>>
>>>>>> *PME_Cellulose_production_NVE: *No reproducibility shown from step
>>>>>> 5,000
>>>>>> onwards
>>>>>> *PME_Cellulose_production_NPT: ** *No reproducibility shown from step
>>>>>> 29,000 onwards. Also outfile is not written properly - blank gaps
>>>>>> appear
>>>>>> where something should have been written.
>>>>>>
>>>>>>
>>>>>> Out files and sdiff files are included as attatchments
>>>>>>
>>>>>> ##############################**###################
>>>>>>
>>>>>> So I'm going to update my nvidia driver to the latest version and
>>>>>> patch
>>>>>> amber to the latest version and rerun the tests to see if there is any
>>>>>> improvement. Could someone let me know if it is necessary to recompile
>>>>>> any
>>>>>> or all of AMBER after applying the bugfixes?
>>>>>>
>>>>>> Additionally, I'm going to run memory tests and heaven benchmarks on
>>>>>> the
>>>>>> cards to check whether they are faulty or not.
>>>>>>
>>>>>> I'm thinking that there is a mix of hardware error/configuration (esp
>>>>>> in
>>>>>> the case of GPU-01_008) and amber software error in this situation.
>>>>>> What
>>>>>> do
>>>>>> you guys think?
>>>>>>
>>>>>> Also am I right in thinking (from what Scott was saying) that all the
>>>>>> benchmarks should be reproducible across 50k steps but begin to
>>>>>> diverge
>>>>>> at
>>>>>> around 100K steps? Is there any difference from in setting *ig *to an
>>>>>> explicit number to removing it from the mdin file?
>>>>>>
>>>>>> br,
>>>>>> g
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 31 May 2013 23:45, ET <sketchfoot.gmail.com> wrote:
>>>>>>
>>>>>> I don't need sysadmins, but sysadmins need me as it gives purpose to
>>>>>>> their
>>>>>>> bureaucratic existence. A encountered evil if working in an
>>>>>>> institution
>>>>>>> or
>>>>>>> comapny IMO. Good science and indiviguality being sacrificed for
>>>>>>> standardisation and mediocrity in the intrerests of maintaing a
>>>>>>> system
>>>>>>> that
>>>>>>> focusses on maintaining the system and not the objective.
>>>>>>>
>>>>>>> You need root to move fwd on these things, unfortunately. and ppl
>>>>>>> with
>>>>>>> root are kinda like your parents when you try to borrow money from
>>>>>>> them
>>>>>>> .
>>>>>>> age 12 :D
>>>>>>> On May 31, 2013 9:34 PM, "Marek Maly" <marek.maly.ujep.cz> wrote:
>>>>>>>
>>>>>>> Sorry why do you need sysadmins :)) ?
>>>>>>>>
>>>>>>>> BTW here is the most recent driver:
>>>>>>>>
>>>>>>>> http://www.nvidia.com/object/**linux-display-amd64-319.23-**
>>>>>>>> driver.html<http://www.nvidia.com/object/linux-display-amd64-319.23-driver.html>
>>>>>>>>
>>>>>>>> I do not remember anything easier than is to install driver
>>>>>>>> (especially
>>>>>>>> in case of binary (*.run) installer) :))
>>>>>>>>
>>>>>>>> M.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Dne Fri, 31 May 2013 22:02:34 +0200 ET <sketchfoot.gmail.com>
>>>>>>>> napsal/-a:
>>>>>>>>
>>>>>>>> > Yup. I know. I replaced a 680 and the everknowing sysadmins are
>>>>>>>> reluctant
>>>>>>>> > to install drivers not in the repositoery as they are lame. :(
>>>>>>>> > On May 31, 2013 7:14 PM, "Marek Maly" <marek.maly.ujep.cz> wrote:
>>>>>>>> >>
>>>>>>>> >> As I already wrote you,
>>>>>>>> >>
>>>>>>>> >> the first driver which properly/officially supports Titans,
>>>>>>>> should
>>>>>>>> be
>>>>>>>> >> 313.26 .
>>>>>>>> >>
>>>>>>>> >> Anyway I am curious mainly about your 100K repetitive tests with
>>>>>>>> >> your Titan SC card. Especially in case of these tests ( JAC_NVE,
>>>>>>>> JAC_NPT
>>>>>>>> >> and CELLULOSE_NVE ) where
>>>>>>>> >> my Titans SC randomly failed or succeeded. In FACTOR_IX_NVE,
>>>>>>>> >> FACTOR_IX_NPT
>>>>>>>> >> tests both
>>>>>>>> >> my cards are perfectly stable (independently from drv. version)
>>>>>>>> and
>>>>>>>> also
>>>>>>>> >> the runs
>>>>>>>> >> are perfectly or almost perfectly reproducible.
>>>>>>>> >>
>>>>>>>> >> Also if your test will crash please report the eventual errs.
>>>>>>>> >>
>>>>>>>> >> To this moment I have this actual library of errs on my Titans SC
>>>>>>>> GPUs.
>>>>>>>> >>
>>>>>>>> >> #1 ERR writtent in mdout:
>>>>>>>> >> ------
>>>>>>>> >> | ERROR: max pairlist cutoff must be less than unit cell max
>>>>>>>> sphere
>>>>>>>> >> radius!
>>>>>>>> >> ------
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> #2 no ERR writtent in mdout, ERR written in standard output
>>>>>>>> (nohup.out)
>>>>>>>> >>
>>>>>>>> >> ----
>>>>>>>> >> Error: unspecified launch failure launching kernel kNLSkinTest
>>>>>>>> >> cudaFree GpuBuffer::Deallocate failed unspecified launch failure
>>>>>>>> >> ----
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> #3 no ERR writtent in mdout, ERR written in standard output
>>>>>>>> (nohup.out)
>>>>>>>> >> ----
>>>>>>>> >> cudaMemcpy GpuBuffer::Download failed unspecified launch failure
>>>>>>>> >> ----
>>>>>>>> >>
>>>>>>>> >> Another question, regarding your Titan SC, it is also EVGA as in
>>>>>>>> my
>>>>>>>> case
>>>>>>>> >> or it is another producer ?
>>>>>>>> >>
>>>>>>>> >> Thanks,
>>>>>>>> >>
>>>>>>>> >> M.
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> Dne Fri, 31 May 2013 19:17:03 +0200 ET <sketchfoot.gmail.com>
>>>>>>>> napsal/-a:
>>>>>>>> >>
>>>>>>>> >> > Well, this is interesting...
>>>>>>>> >> >
>>>>>>>> >> > I ran 50k steps on the Titan on the other machine with driver
>>>>>>>> 310.44
>>>>>>>> >> and
>>>>>>>> >> > it
>>>>>>>> >> > passed all the GB steps. i.e totally identical results over two
>>>>>>>> >> repeats.
>>>>>>>> >> > However, it failed all the PME tests after step 1000. I'm
>>>>>>>> going to
>>>>>>>> > update
>>>>>>>> >> > the driver and test it again.
>>>>>>>> >> >
>>>>>>>> >> > Files included as attachments.
>>>>>>>> >> >
>>>>>>>> >> > br,
>>>>>>>> >> > g
>>>>>>>> >> >
>>>>>>>> >> >
>>>>>>>> >> > On 31 May 2013 16:40, Marek Maly <marek.maly.ujep.cz> wrote:
>>>>>>>> >> >
>>>>>>>> >> >> One more thing,
>>>>>>>> >> >>
>>>>>>>> >> >> can you please check under which frequency is running that
>>>>>>>> your
>>>>>>>> >> titan ?
>>>>>>>> >> >>
>>>>>>>> >> >> As the base frequency of normal Titans is 837MHz and the Boost
>>>>>>>> one
>>>>>>>> is
>>>>>>>> >> >> 876MHz I
>>>>>>>> >> >> assume that yor GPU is running automatically also under it's
>>>>>>>> boot
>>>>>>>> >> >> frequency (876MHz).
>>>>>>>> >> >> You can find this information e.g. in Amber mdout file.
>>>>>>>> >> >>
>>>>>>>> >> >> You also mentioned some crashes in your previous email. Your
>>>>>>>> ERRs
>>>>>>>> >> were
>>>>>>>> >> >> something like those here:
>>>>>>>> >> >>
>>>>>>>> >> >> #1 ERR writtent in mdout:
>>>>>>>> >> >> ------
>>>>>>>> >> >> | ERROR: max pairlist cutoff must be less than unit cell max
>>>>>>>> sphere
>>>>>>>> >> >> radius!
>>>>>>>> >> >> ------
>>>>>>>> >> >>
>>>>>>>> >> >>
>>>>>>>> >> >> #2 no ERR writtent in mdout, ERR written in standard output
>>>>>>>> >> (nohup.out)
>>>>>>>> >> >>
>>>>>>>> >> >> ----
>>>>>>>> >> >> Error: unspecified launch failure launching kernel kNLSkinTest
>>>>>>>> >> >> cudaFree GpuBuffer::Deallocate failed unspecified launch
>>>>>>>> failure
>>>>>>>> >> >> ----
>>>>>>>> >> >>
>>>>>>>> >> >>
>>>>>>>> >> >> #3 no ERR writtent in mdout, ERR written in standard output
>>>>>>>> >> (nohup.out)
>>>>>>>> >> >> ----
>>>>>>>> >> >> cudaMemcpy GpuBuffer::Download failed unspecified launch
>>>>>>>> failure
>>>>>>>> >> >> ----
>>>>>>>> >> >>
>>>>>>>> >> >> or you obtained some new/additional errs ?
>>>>>>>> >> >>
>>>>>>>> >> >>
>>>>>>>> >> >>
>>>>>>>> >> >> M.
>>>>>>>> >> >>
>>>>>>>> >> >>
>>>>>>>> >> >>
>>>>>>>> >> >> Dne Fri, 31 May 2013 17:30:57 +0200 filip fratev
>>>>>>>> >> <filipfratev.yahoo.com
>>>>>>>> >>
>>>>>>>> >> >> napsal/-a:
>>>>>>>> >> >>
>>>>>>>> >> >> > Hi,
>>>>>>>> >> >> > This is what I obtained for 50K tests and "normal" GTXTitan:
>>>>>>>> >> >> >
>>>>>>>> >> >> > run1:
>>>>>>>> >> >> >
>>>>>>>> >> >> >
>>>>>>>> >> >> >
>>>>>>>> >> >>
>>>>>>>> >
>>>>>>>> ------------------------------**------------------------------**
>>>>>>>> ------------------
>>>>>>>> >> >> >
>>>>>>>> >> >> >
>>>>>>>> >> >> > A V E R A G E S O V E R 50 S T E P S
>>>>>>>> >> >> >
>>>>>>>> >> >> >
>>>>>>>> >> >> > NSTEP = 50000 TIME(PS) = 120.020 TEMP(K) =
>>>>>>>> 299.87
>>>>>>>> >> PRESS
>>>>>>>> >> >> > = 0.0
>>>>>>>> >> >> > Etot = -443237.1079 EKtot = 257679.9750 EPtot
>>>>>>>> =
>>>>>>>> >> >> > -700917.0829
>>>>>>>> >> >> > BOND = 20193.1856 ANGLE = 53517.5432 DIHED
>>>>>>>> =
>>>>>>>> >> >> > 23575.4648
>>>>>>>> >> >> > 1-4 NB = 21759.5524 1-4 EEL = 742552.5939 VDWAALS
>>>>>>>> =
>>>>>>>> >> >> > 96286.7714
>>>>>>>> >> >> > EELEC = -1658802.1941 EHBOND = 0.0000
>>>>>>>> RESTRAINT
>>>>>>>> =
>>>>>>>> >> >> > 0.0000
>>>>>>>> >> >> >
>>>>>>>> >> >>
>>>>>>>> >
>>>>>>>> ------------------------------**------------------------------**
>>>>>>>> ------------------
>>>>>>>> >> >> >
>>>>>>>> >> >> >
>>>>>>>> >> >> > R M S F L U C T U A T I O N S
>>>>>>>> >> >> >
>>>>>>>> >> >> >
>>>>>>>> >> >> > NSTEP = 50000 TIME(PS) = 120.020 TEMP(K) =
>>>>>>>> 0.33
>>>>>>>> >> PRESS
>>>>>>>> >> >> > = 0.0
>>>>>>>> >> >> > Etot = 11.2784 EKtot = 284.8999 EPtot
>>>>>>>> =
>>>>>>>> >> >> > 289.0773
>>>>>>>> >> >> > BOND = 136.3417 ANGLE = 214.0054 DIHED
>>>>>>>> =
>>>>>>>> >> >> > 59.4893
>>>>>>>> >> >> > 1-4 NB = 58.5891 1-4 EEL = 330.5400 VDWAALS
>>>>>>>> =
>>>>>>>> >> >> > 559.2079
>>>>>>>> >> >> > EELEC = 743.8771 EHBOND = 0.0000
>>>>>>>> RESTRAINT
>>>>>>>> =
>>>>>>>> >> >> > 0.0000
>>>>>>>> >> >> > |E(PBS) = 21.8119
>>>>>>>> >> >> >
>>>>>>>> >> >>
>>>>>>>> >
>>>>>>>> ------------------------------**------------------------------**
>>>>>>>> ------------------
>>>>>>>> >> >> >
>>>>>>>> >> >> > run2:
>>>>>>>> >> >> >
>>>>>>>> >> >>
>>>>>>>> >
>>>>>>>> ------------------------------**------------------------------**
>>>>>>>> ------------------
>>>>>>>> >> >> >
>>>>>>>> >> >> >
>>>>>>>> >> >> > A V E R A G E S O V E R 50 S T E P S
>>>>>>>> >> >> >
>>>>>>>> >> >> >
>>>>>>>> >> >> > NSTEP = 50000 TIME(PS) = 120.020 TEMP(K) =
>>>>>>>> 299.89
>>>>>>>> >> PRESS
>>>>>>>> >> >> > = 0.0
>>>>>>>> >> >> > Etot = -443240.0999 EKtot = 257700.0950 EPtot
>>>>>>>> =
>>>>>>>> >> >> > -700940.1949
>>>>>>>> >> >> > BOND = 20241.9174 ANGLE = 53644.6694 DIHED
>>>>>>>> =
>>>>>>>> >> >> > 23541.3737
>>>>>>>> >> >> > 1-4 NB = 21803.1898 1-4 EEL = 742754.2254 VDWAALS
>>>>>>>> =
>>>>>>>> >> >> > 96298.8308
>>>>>>>> >> >> > EELEC = -1659224.4013 EHBOND = 0.0000
>>>>>>>> RESTRAINT
>>>>>>>> =
>>>>>>>> >> >> > 0.0000
>>>>>>>> >> >> >
>>>>>>>> >> >>
>>>>>>>> >
>>>>>>>> ------------------------------**------------------------------**
>>>>>>>> ------------------
>>>>>>>> >> >> >
>>>>>>>> >> >> >
>>>>>>>> >> >> > R M S F L U C T U A T I O N S
>>>>>>>> >> >> >
>>>>>>>> >> >> >
>>>>>>>> >> >> > NSTEP = 50000 TIME(PS) = 120.020 TEMP(K) =
>>>>>>>> 0.41
>>>>>>>> >> PRESS
>>>>>>>> >> >> > = 0.0
>>>>>>>> >> >> > Etot = 10.7633 EKtot = 348.2819 EPtot
>>>>>>>> =
>>>>>>>> >> >> > 353.9918
>>>>>>>> >> >> > BOND = 106.5314 ANGLE = 196.7052 DIHED
>>>>>>>> =
>>>>>>>> >> >> > 69.7476
>>>>>>>> >> >> > 1-4 NB = 60.3435 1-4 EEL = 400.7466 VDWAALS
>>>>>>>> =
>>>>>>>> >> >> > 462.7763
>>>>>>>> >> >> > EELEC = 651.9857 EHBOND = 0.0000
>>>>>>>> RESTRAINT
>>>>>>>> =
>>>>>>>> >> >> > 0.0000
>>>>>>>> >> >> > |E(PBS) = 17.0642
>>>>>>>> >> >> >
>>>>>>>> >> >>
>>>>>>>> >
>>>>>>>> ------------------------------**------------------------------**
>>>>>>>> ------------------
>>>>>>>> >> >> >
>>>>>>>> >> >> >
>>>>>>>> >> >>
>>>>>>>> >
>>>>>>>> ------------------------------**------------------------------**
>>>>>>>> --------------------
>>>>>>>> >> >> >
>>>>>>>> >> >> >
>>>>>>>> >> >> >
>>>>>>>> >> >> >
>>>>>>>> >> >> > ______________________________**__
>>>>>>>> >> >> > From: Marek Maly <marek.maly.ujep.cz>
>>>>>>>> >> >> > To: AMBER Mailing List <amber.ambermd.org>
>>>>>>>> >> >> > Sent: Friday, May 31, 2013 3:34 PM
>>>>>>>> >> >> > Subject: Re: [AMBER] experiences with EVGA GTX TITAN
>>>>>>>> Superclocked
>>>>>>>> -
>>>>>>>> >> >> > memtestG80 - UNDERclocking in Linux ?
>>>>>>>> >> >> >
>>>>>>>> >> >> > Hi here are my 100K results for driver 313.30 (and still
>>>>>>>> Cuda
>>>>>>>> 5.0).
>>>>>>>> >> >> >
>>>>>>>> >> >> > The results are rather similar to those obtained
>>>>>>>> >> >> > under my original driver 319.17 (see the first table
>>>>>>>> >> >> > which I sent in this thread).
>>>>>>>> >> >> >
>>>>>>>> >> >> > M.
>>>>>>>> >> >> >
>>>>>>>> >> >> >
>>>>>>>> >> >> > Dne Fri, 31 May 2013 12:29:59 +0200 Marek Maly <
>>>>>>>> marek.maly.ujep.cz>
>>>>>>>> >> >> > napsal/-a:
>>>>>>>> >> >> >
>>>>>>>> >> >> >> Hi,
>>>>>>>> >> >> >>
>>>>>>>> >> >> >> please try to run at lest 100K tests twice to verify exact
>>>>>>>> >> >> >> reproducibility
>>>>>>>> >> >> >> of the results on the given card. If you find in any mdin
>>>>>>>> file
>>>>>>>> >> ig=-1
>>>>>>>> >> >> >> just
>>>>>>>> >> >> >> delete it to ensure that you are using the identical random
>>>>>>>> seed
>>>>>>>> >> for
>>>>>>>> >> >> >> both
>>>>>>>> >> >> >> runs. You can eventually omit NUCLEOSOME test
>>>>>>>> >> >> >> as it is too much time consuming.
>>>>>>>> >> >> >>
>>>>>>>> >> >> >> Driver 310.44 ?????
>>>>>>>> >> >> >>
>>>>>>>> >> >> >> As far as I know the proper support for titans is from
>>>>>>>> version
>>>>>>>> > 313.26
>>>>>>>> >> >> >>
>>>>>>>> >> >> >> see e.g. here :
>>>>>>>> >> >> >>
>>>>>>>> >> >>
>>>>>>>> >
>>>>>>>> http://www.geeks3d.com/**20130306/nvidia-releases-r313-**
>>>>>>>> 26-for-linux-with-gtx-titan-**support/<http://www.geeks3d.com/20130306/nvidia-releases-r313-26-for-linux-with-gtx-titan-support/>
>>>>>>>> >> >> >>
>>>>>>>> >> >> >> BTW: On my site downgrade to drv. 313.30 did not solved the
>>>>>>>> >> >> situation, I
>>>>>>>> >> >> >> will post
>>>>>>>> >> >> >> my results soon here.
>>>>>>>> >> >> >>
>>>>>>>> >> >> >> M.
>>>>>>>> >> >> >>
>>>>>>>> >> >> >>
>>>>>>>> >> >> >>
>>>>>>>> >> >> >>
>>>>>>>> >> >> >>
>>>>>>>> >> >> >>
>>>>>>>> >> >> >>
>>>>>>>> >> >> >>
>>>>>>>> >> >> >> Dne Fri, 31 May 2013 12:21:21 +0200 ET <
>>>>>>>> sketchfoot.gmail.com>
>>>>>>>> >> >> napsal/-a:
>>>>>>>> >> >> >>
>>>>>>>> >> >> >>> ps. I have another install of amber on another computer
>>>>>>>> with
>>>>>>>> a
>>>>>>>> >> >> >>> different
>>>>>>>> >> >> >>> Titan and different Driver Version: 310.44.
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>> In the interests of thrashing the proverbial horse, I'll
>>>>>>>> run
>>>>>>>> the
>>>>>>>> >> >> >>> benchmark
>>>>>>>> >> >> >>> for 50k steps. :P
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>> br,
>>>>>>>> >> >> >>> g
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>> On 31 May 2013 11:17, ET <sketchfoot.gmail.com> wrote:
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>>> Hi, I just ran the Amber benchmark for the default (10000
>>>>>>>> steps)
>>>>>>>> >> >> on my
>>>>>>>> >> >> >>>> Titan.
>>>>>>>> >> >> >>>>
>>>>>>>> >> >> >>>> Using sdiff -sB showed that the two runs were completely
>>>>>>>> > identical.
>>>>>>>> >> >> >>>> I've
>>>>>>>> >> >> >>>> attached compressed files of the mdout & diff files.
>>>>>>>> >> >> >>>>
>>>>>>>> >> >> >>>> br,
>>>>>>>> >> >> >>>> g
>>>>>>>> >> >> >>>>
>>>>>>>> >> >> >>>>
>>>>>>>> >> >> >>>> On 30 May 2013 23:41, Marek Maly <marek.maly.ujep.cz>
>>>>>>>> wrote:
>>>>>>>> >> >> >>>>
>>>>>>>> >> >> >>>>> OK, let's see. The eventual downclocking I see as the
>>>>>>>> very
>>>>>>>> last
>>>>>>>> >> >> >>>>> possibility
>>>>>>>> >> >> >>>>> (if I don't decide for RMAing). But now still some other
>>>>>>>> >> >> experiments
>>>>>>>> >> >> >>>>> are
>>>>>>>> >> >> >>>>> available :))
>>>>>>>> >> >> >>>>> I just started 100K tests under 313.30 driver. For today
>>>>>>>> good
>>>>>>>> >> >> night
>>>>>>>> >> >> >>>>> ...
>>>>>>>> >> >> >>>>>
>>>>>>>> >> >> >>>>> M.
>>>>>>>> >> >> >>>>>
>>>>>>>> >> >> >>>>> Dne Fri, 31 May 2013 00:45:49 +0200 Scott Le Grand
>>>>>>>> >> >> >>>>> <varelse2005.gmail.com
>>>>>>>> >> >> >>>>> >
>>>>>>>> >> >> >>>>> napsal/-a:
>>>>>>>> >> >> >>>>>
>>>>>>>> >> >> >>>>> > It will be very interesting if this behavior persists
>>>>>>>> after
>>>>>>>> >> >> >>>>> downclocking.
>>>>>>>> >> >> >>>>> >
>>>>>>>> >> >> >>>>> > But right now, Titan 0 *looks* hosed and Titan 1
>>>>>>>> *looks*
>>>>>>>> like
>>>>>>>> > it
>>>>>>>> >> >> >>>>> needs
>>>>>>>> >> >> >>>>> > downclocking...
>>>>>>>> >> >> >>>>> > On May 30, 2013 3:20 PM, "Marek Maly"
>>>>>>>> <marek.maly.ujep.cz>
>>>>>>>> >> >> wrote:
>>>>>>>> >> >> >>>>> >
>>>>>>>> >> >> >>>>> >> Hi all,
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >> here are my results from the 500K steps 2 x repeated
>>>>>>>> > benchmarks
>>>>>>>> >> >> >>>>> >> under 319.23 driver and still Cuda 5.0 (see the
>>>>>>>> attached
>>>>>>>> >> table
>>>>>>>> >> >> ).
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >> It is hard to say if the results are better or worse
>>>>>>>> than
>>>>>>>> in
>>>>>>>> > my
>>>>>>>> >> >> >>>>> >> previous 100K test under driver 319.17.
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >> While results from Cellulose test were improved and
>>>>>>>> the
>>>>>>>> > TITAN_1
>>>>>>>> >> >> >>>>> card
>>>>>>>> >> >> >>>>> >> even
>>>>>>>> >> >> >>>>> >> successfully finished all 500K steps moreover with
>>>>>>>> exactly
>>>>>>>> >> the
>>>>>>>> >> >> >>>>> same
>>>>>>>> >> >> >>>>> >> final
>>>>>>>> >> >> >>>>> >> energy !
>>>>>>>> >> >> >>>>> >> (TITAN_0 at least finished more than 100K steps and
>>>>>>>> in
>>>>>>>> >> RUN_01
>>>>>>>> >> >> even
>>>>>>>> >> >> >>>>> more
>>>>>>>> >> >> >>>>> >> than 400K steps)
>>>>>>>> >> >> >>>>> >> In JAC_NPT test no GPU was able to finish at least
>>>>>>>> 100K
>>>>>>>> >> steps
>>>>>>>> >> >> and
>>>>>>>> >> >> >>>>> the
>>>>>>>> >> >> >>>>> >> results from JAC_NVE
>>>>>>>> >> >> >>>>> >> test are also not too much convincing. FACTOR_IX_NVE
>>>>>>>> and
>>>>>>>> >> >> >>>>> FACTOR_IX_NPT
>>>>>>>> >> >> >>>>> >> were successfully
>>>>>>>> >> >> >>>>> >> finished with 100% reproducibility in FACTOR_IX_NPT
>>>>>>>> case
>>>>>>>> >> (on
>>>>>>>> >> >> both
>>>>>>>> >> >> >>>>> >> cards)
>>>>>>>> >> >> >>>>> >> and almost
>>>>>>>> >> >> >>>>> >> 100% reproducibility in case of FACTOR_IX_NVE (again
>>>>>>>> 100%
>>>>>>>> in
>>>>>>>> >> >> case
>>>>>>>> >> >> >>>>> of
>>>>>>>> >> >> >>>>> >> TITAN_1). TRPCAGE, MYOGLOBIN
>>>>>>>> >> >> >>>>> >> again finished without any problem with 100%
>>>>>>>> >> reproducibility.
>>>>>>>> >> >> >>>>> NUCLEOSOME
>>>>>>>> >> >> >>>>> >> test was not done
>>>>>>>> >> >> >>>>> >> this time due to high time requirements. If you find
>>>>>>>> in
>>>>>>>> the
>>>>>>>> >> >> table
>>>>>>>> >> >> >>>>> >> positive
>>>>>>>> >> >> >>>>> >> number finishing with
>>>>>>>> >> >> >>>>> >> K (which means "thousands") it means the last number
>>>>>>>> of
>>>>>>>> step
>>>>>>>> >> >> >>>>> written in
>>>>>>>> >> >> >>>>> >> mdout before crash.
>>>>>>>> >> >> >>>>> >> Below are all the 3 types of detected errs with
>>>>>>>> relevant
>>>>>>>> >> >> >>>>> systems/rounds
>>>>>>>> >> >> >>>>> >> where the given err
>>>>>>>> >> >> >>>>> >> appeared.
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >> Now I will try just 100K tests under ETs favourite
>>>>>>>> driver
>>>>>>>> >> >> version
>>>>>>>> >> >> >>>>> 313.30
>>>>>>>> >> >> >>>>> >> :)) and then
>>>>>>>> >> >> >>>>> >> I will eventually try to experiment with cuda 5.5
>>>>>>>> which
>>>>>>>> I
>>>>>>>> >> >> already
>>>>>>>> >> >> >>>>> >> downloaded from the
>>>>>>>> >> >> >>>>> >> cuda zone ( I had to become cuda developer for this
>>>>>>>> :))
>>>>>>>> )
>>>>>>>> >> BTW
>>>>>>>> >> >> ET
>>>>>>>> >> >> >>>>> thanks
>>>>>>>> >> >> >>>>> >> for the frequency info !
>>>>>>>> >> >> >>>>> >> and I am still ( perhaps not alone :)) ) very curious
>>>>>>>> about
>>>>>>>> >> >> your 2
>>>>>>>> >> >> >>>>> x
>>>>>>>> >> >> >>>>> >> repeated Amber benchmark tests with superclocked
>>>>>>>> Titan.
>>>>>>>> >> Indeed
>>>>>>>> >> >> >>>>> that
>>>>>>>> >> >> >>>>> I
>>>>>>>> >> >> >>>>> am
>>>>>>>> >> >> >>>>> >> very curious also about that Ross "hot" patch.
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >> M.
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >> ERRORS DETECTED DURING THE 500K steps tests with
>>>>>>>> driver
>>>>>>>> >> 319.23
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >> #1 ERR writtent in mdout:
>>>>>>>> >> >> >>>>> >> ------
>>>>>>>> >> >> >>>>> >> | ERROR: max pairlist cutoff must be less than unit
>>>>>>>> cell
>>>>>>>> >> max
>>>>>>>> >> >> >>>>> sphere
>>>>>>>> >> >> >>>>> >> radius!
>>>>>>>> >> >> >>>>> >> ------
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >> TITAN_0 ROUND_1 JAC_NPT (at least 5000 steps
>>>>>>>> successfully
>>>>>>>> > done
>>>>>>>> >> >> >>>>> before
>>>>>>>> >> >> >>>>> >> crash)
>>>>>>>> >> >> >>>>> >> TITAN_0 ROUND_2 JAC_NPT (at least 8000 steps
>>>>>>>> successfully
>>>>>>>> > done
>>>>>>>> >> >> >>>>> before
>>>>>>>> >> >> >>>>> >> crash)
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >> #2 no ERR writtent in mdout, ERR written in standard
>>>>>>>> output
>>>>>>>> >> >> >>>>> (nohup.out)
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >> ----
>>>>>>>> >> >> >>>>> >> Error: unspecified launch failure launching kernel
>>>>>>>> >> kNLSkinTest
>>>>>>>> >> >> >>>>> >> cudaFree GpuBuffer::Deallocate failed unspecified
>>>>>>>> launch
>>>>>>>> >> >> failure
>>>>>>>> >> >> >>>>> >> ----
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >> TITAN_0 ROUND_1 CELLULOSE_NVE (at least 437 000 steps
>>>>>>>> >> >> successfully
>>>>>>>> >> >> >>>>> done
>>>>>>>> >> >> >>>>> >> before crash)
>>>>>>>> >> >> >>>>> >> TITAN_0 ROUND_2 JAC_NVE (at least 162 000 steps
>>>>>>>> >> successfully
>>>>>>>> >> >> done
>>>>>>>> >> >> >>>>> >> before
>>>>>>>> >> >> >>>>> >> crash)
>>>>>>>> >> >> >>>>> >> TITAN_0 ROUND_2 CELLULOSE_NVE (at least 117 000 steps
>>>>>>>> >> >> successfully
>>>>>>>> >> >> >>>>> done
>>>>>>>> >> >> >>>>> >> before crash)
>>>>>>>> >> >> >>>>> >> TITAN_1 ROUND_1 JAC_NVE (at least 119 000 steps
>>>>>>>> >> successfully
>>>>>>>> >> >> done
>>>>>>>> >> >> >>>>> >> before
>>>>>>>> >> >> >>>>> >> crash)
>>>>>>>> >> >> >>>>> >> TITAN_1 ROUND_2 JAC_NVE (at least 43 000 steps
>>>>>>>> successfully
>>>>>>>> >> >> done
>>>>>>>> >> >> >>>>> before
>>>>>>>> >> >> >>>>> >> crash)
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >> #3 no ERR writtent in mdout, ERR written in standard
>>>>>>>> output
>>>>>>>> >> >> >>>>> (nohup.out)
>>>>>>>> >> >> >>>>> >> ----
>>>>>>>> >> >> >>>>> >> cudaMemcpy GpuBuffer::Download failed unspecified
>>>>>>>> launch
>>>>>>>> >> >> failure
>>>>>>>> >> >> >>>>> >> ----
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >> TITAN_1 ROUND_1 JAC_NPT (at least 77 000 steps
>>>>>>>> successfully
>>>>>>>> >> >> done
>>>>>>>> >> >> >>>>> before
>>>>>>>> >> >> >>>>> >> crash)
>>>>>>>> >> >> >>>>> >> TITAN_1 ROUND_2 JAC_NPT (at least 58 000 steps
>>>>>>>> successfully
>>>>>>>> >> >> done
>>>>>>>> >> >> >>>>> before
>>>>>>>> >> >> >>>>> >> crash)
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >> Dne Thu, 30 May 2013 21:27:17 +0200 Scott Le Grand
>>>>>>>> >> >> >>>>> >> <varelse2005.gmail.com>
>>>>>>>> >> >> >>>>> >> napsal/-a:
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >> Oops meant to send that to Jason...
>>>>>>>> >> >> >>>>> >>>
>>>>>>>> >> >> >>>>> >>> Anyway, before we all panic, we need to get K20's
>>>>>>>> behavior
>>>>>>>> >> >> >>>>> analyzed
>>>>>>>> >> >> >>>>> >>> here.
>>>>>>>> >> >> >>>>> >>> If it's deterministic, this truly is a hardware
>>>>>>>> issue. If
>>>>>>>> >> >> not,
>>>>>>>> >> >> >>>>> then
>>>>>>>> >> >> >>>>> it
>>>>>>>> >> >> >>>>> >>> gets interesting because 680 is deterministic as far
>>>>>>>> as I
>>>>>>>> >> can
>>>>>>>> >> >> >>>>> tell...
>>>>>>>> >> >> >>>>> >>> On May 30, 2013 12:24 PM, "Scott Le Grand"
>>>>>>>> >> >> >>>>> <varelse2005.gmail.com>
>>>>>>>> >> >> >>>>> >>> wrote:
>>>>>>>> >> >> >>>>> >>>
>>>>>>>> >> >> >>>>> >>> If the errors are not deterministically triggered,
>>>>>>>> they
>>>>>>>> >> >> probably
>>>>>>>> >> >> >>>>> >>> won't be
>>>>>>>> >> >> >>>>> >>>> fixed by the patch, alas...
>>>>>>>> >> >> >>>>> >>>> On May 30, 2013 12:15 PM, "Jason Swails"
>>>>>>>> >> >> >>>>> <jason.swails.gmail.com>
>>>>>>>> >> >> >>>>> >>>> wrote:
>>>>>>>> >> >> >>>>> >>>>
>>>>>>>> >> >> >>>>> >>>> Just a reminder to everyone based on what Ross
>>>>>>>> said:
>>>>>>>> >> there
>>>>>>>> >> >> is a
>>>>>>>> >> >> >>>>> >>>> pending
>>>>>>>> >> >> >>>>> >>>>> patch to pmemd.cuda that will be coming out
>>>>>>>> shortly
>>>>>>>> >> (maybe
>>>>>>>> >> >> even
>>>>>>>> >> >> >>>>> >>>>> within
>>>>>>>> >> >> >>>>> >>>>> hours). It's entirely possible that several of
>>>>>>>> these
>>>>>>>> > errors
>>>>>>>> >> >> >>>>> are
>>>>>>>> >> >> >>>>> >>>>> fixed
>>>>>>>> >> >> >>>>> >>>>> by
>>>>>>>> >> >> >>>>> >>>>> this patch.
>>>>>>>> >> >> >>>>> >>>>>
>>>>>>>> >> >> >>>>> >>>>> All the best,
>>>>>>>> >> >> >>>>> >>>>> Jason
>>>>>>>> >> >> >>>>> >>>>>
>>>>>>>> >> >> >>>>> >>>>>
>>>>>>>> >> >> >>>>> >>>>> On Thu, May 30, 2013 at 2:46 PM, filip fratev <
>>>>>>>> >> >> >>>>> filipfratev.yahoo.com>
>>>>>>>> >> >> >>>>> >>>>> wrote:
>>>>>>>> >> >> >>>>> >>>>>
>>>>>>>> >> >> >>>>> >>>>> > I have observed the same crashes from time to
>>>>>>>> time. I
>>>>>>>> > will
>>>>>>>> >> >> >>>>> run
>>>>>>>> >> >> >>>>> >>>>> cellulose
>>>>>>>> >> >> >>>>> >>>>> > nve for 100k and will past results here.
>>>>>>>> >> >> >>>>> >>>>> >
>>>>>>>> >> >> >>>>> >>>>> > All the best,
>>>>>>>> >> >> >>>>> >>>>> > Filip
>>>>>>>> >> >> >>>>> >>>>> >
>>>>>>>> >> >> >>>>> >>>>> >
>>>>>>>> >> >> >>>>> >>>>> >
>>>>>>>> >> >> >>>>> >>>>> >
>>>>>>>> >> >> >>>>> >>>>> > ______________________________****__
>>>>>>>> >> >> >>>>> >>>>> > From: Scott Le Grand <varelse2005.gmail.com>
>>>>>>>> >> >> >>>>> >>>>> > To: AMBER Mailing List <amber.ambermd.org>
>>>>>>>> >> >> >>>>> >>>>> > Sent: Thursday, May 30, 2013 9:01 PM
>>>>>>>> >> >> >>>>> >>>>> > Subject: Re: [AMBER] experiences with EVGA GTX
>>>>>>>> TITAN
>>>>>>>> >> >> >>>>> Superclocked
>>>>>>>> >> >> >>>>> -
>>>>>>>> >> >> >>>>> >>>>> > memtestG80 - UNDERclocking in Linux ?
>>>>>>>> >> >> >>>>> >>>>> >
>>>>>>>> >> >> >>>>> >>>>> >
>>>>>>>> >> >> >>>>> >>>>> > Run cellulose nve for 100k iterations twice .
>>>>>>>> If
>>>>>>>> the
>>>>>>>> >> >> final
>>>>>>>> >> >> >>>>> >>>>> energies
>>>>>>>> >> >> >>>>> >>>>> don't
>>>>>>>> >> >> >>>>> >>>>> > match, you have a hardware issue. No need to
>>>>>>>> play
>>>>>>>> with
>>>>>>>> >> >> ntpr
>>>>>>>> >> >> >>>>> or
>>>>>>>> >> >> >>>>> any
>>>>>>>> >> >> >>>>> >>>>> other
>>>>>>>> >> >> >>>>> >>>>> > variable.
>>>>>>>> >> >> >>>>> >>>>> > On May 30, 2013 10:58 AM, <pavel.banas.upol.cz>
>>>>>>>> wrote:
>>>>>>>> >> >> >>>>> >>>>> >
>>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>>> >> >> >>>>> >>>>> > > Dear all,
>>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>>> >> >> >>>>> >>>>> > > I would also like to share one of my
>>>>>>>> experience
>>>>>>>> with
>>>>>>>> >> >> titan
>>>>>>>> >> >> >>>>> >>>>> cards. We
>>>>>>>> >> >> >>>>> >>>>> have
>>>>>>>> >> >> >>>>> >>>>> > > one gtx titan card and with one system (~55k
>>>>>>>> atoms,
>>>>>>>> > NVT,
>>>>>>>> >> >> >>>>> >>>>> RNA+waters)
>>>>>>>> >> >> >>>>> >>>>> we
>>>>>>>> >> >> >>>>> >>>>> > run
>>>>>>>> >> >> >>>>> >>>>> > > into same troubles you are describing. I was
>>>>>>>> also
>>>>>>>> >> >> playing
>>>>>>>> >> >> >>>>> with
>>>>>>>> >> >> >>>>> >>>>> ntpr
>>>>>>>> >> >> >>>>> >>>>> to
>>>>>>>> >> >> >>>>> >>>>> > > figure out what is going on, step by step. I
>>>>>>>> >> understand
>>>>>>>> >> >> >>>>> that
>>>>>>>> >> >> >>>>> the
>>>>>>>> >> >> >>>>> >>>>> code
>>>>>>>> >> >> >>>>> >>>>> is
>>>>>>>> >> >> >>>>> >>>>> > > using different routines for calculation
>>>>>>>> >> >> energies+forces or
>>>>>>>> >> >> >>>>> only
>>>>>>>> >> >> >>>>> >>>>> forces.
>>>>>>>> >> >> >>>>> >>>>> > > The
>>>>>>>> >> >> >>>>> >>>>> > > simulations of other systems are perfectly
>>>>>>>> stable,
>>>>>>>> >> >> running
>>>>>>>> >> >> >>>>> for
>>>>>>>> >> >> >>>>> >>>>> days
>>>>>>>> >> >> >>>>> >>>>> and
>>>>>>>> >> >> >>>>> >>>>> > > weeks. Only that particular system
>>>>>>>> systematically
>>>>>>>> >> ends
>>>>>>>> >> >> up
>>>>>>>> >> >> >>>>> with
>>>>>>>> >> >> >>>>> >>>>> this
>>>>>>>> >> >> >>>>> >>>>> > error.
>>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>>> >> >> >>>>> >>>>> > > However, there was one interesting issue.
>>>>>>>> When I
>>>>>>>> set
>>>>>>>> >> >> >>>>> ntpr=1,
>>>>>>>> >> >> >>>>> the
>>>>>>>> >> >> >>>>> >>>>> error
>>>>>>>> >> >> >>>>> >>>>> > > vanished (systematically in multiple runs) and
>>>>>>>> the
>>>>>>>> >> >> >>>>> simulation
>>>>>>>> >> >> >>>>> was
>>>>>>>> >> >> >>>>> >>>>> able to
>>>>>>>> >> >> >>>>> >>>>> > > run for more than millions of steps (I was not
>>>>>>>> let
>>>>>>>> it
>>>>>>>> >> >> >>>>> running
>>>>>>>> >> >> >>>>> for
>>>>>>>> >> >> >>>>> >>>>> weeks
>>>>>>>> >> >> >>>>> >>>>> > as
>>>>>>>> >> >> >>>>> >>>>> > > in the meantime I shifted that simulation to
>>>>>>>> other
>>>>>>>> >> card
>>>>>>>> >> >> -
>>>>>>>> >> >> >>>>> need
>>>>>>>> >> >> >>>>> >>>>> data,
>>>>>>>> >> >> >>>>> >>>>> not
>>>>>>>> >> >> >>>>> >>>>> > > testing). All other setting of ntpr failed.
>>>>>>>> As I
>>>>>>>> read
>>>>>>>> >> >> this
>>>>>>>> >> >> >>>>> >>>>> discussion, I
>>>>>>>> >> >> >>>>> >>>>> > > tried to set ene_avg_sampling=1 with some high
>>>>>>>> value
>>>>>>>> >> of
>>>>>>>> >> >> >>>>> ntpr
>>>>>>>> >> >> >>>>> (I
>>>>>>>> >> >> >>>>> >>>>> expected
>>>>>>>> >> >> >>>>> >>>>> > > that this will shift the code to permanently
>>>>>>>> use
>>>>>>>> the
>>>>>>>> >> >> >>>>> >>>>> force+energies
>>>>>>>> >> >> >>>>> >>>>> part
>>>>>>>> >> >> >>>>> >>>>> > of
>>>>>>>> >> >> >>>>> >>>>> > > the code, similarly to ntpr=1), but the error
>>>>>>>> >> occurred
>>>>>>>> >> >> >>>>> again.
>>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>>> >> >> >>>>> >>>>> > > I know it is not very conclusive for finding
>>>>>>>> out
>>>>>>>> what
>>>>>>>> > is
>>>>>>>> >> >> >>>>> >>>>> happening,
>>>>>>>> >> >> >>>>> >>>>> at
>>>>>>>> >> >> >>>>> >>>>> > > least
>>>>>>>> >> >> >>>>> >>>>> > > not for me. Do you have any idea, why ntpr=1
>>>>>>>> might
>>>>>>>> > help?
>>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>>> >> >> >>>>> >>>>> > > best regards,
>>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>>> >> >> >>>>> >>>>> > > Pavel
>>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>>> >> >> >>>>> >>>>> > > --
>>>>>>>> >> >> >>>>> >>>>> > > Pavel Banáš
>>>>>>>> >> >> >>>>> >>>>> > > pavel.banas.upol.cz
>>>>>>>> >> >> >>>>> >>>>> > > Department of Physical Chemistry,
>>>>>>>> >> >> >>>>> >>>>> > > Palacky University Olomouc
>>>>>>>> >> >> >>>>> >>>>> > > Czech Republic
>>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>>> >> >> >>>>> >>>>> > > ---------- Původní zpráva ----------
>>>>>>>> >> >> >>>>> >>>>> > > Od: Jason Swails <jason.swails.gmail.com>
>>>>>>>> >> >> >>>>> >>>>> > > Datum: 29. 5. 2013
>>>>>>>> >> >> >>>>> >>>>> > > Předmět: Re: [AMBER] experiences with EVGA GTX
>>>>>>>> TITAN
>>>>>>>> >> >> >>>>> >>>>> Superclocked -
>>>>>>>> >> >> >>>>> >>>>> > > memtestG
>>>>>>>> >> >> >>>>> >>>>> > > 80 - UNDERclocking in Linux ?
>>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>>> >> >> >>>>> >>>>> > > "I'll answer a little bit:
>>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>>> >> >> >>>>> >>>>> > > NTPR=10 Etot after 2000 steps
>>>>>>>> >> >> >>>>> >>>>> > > >
>>>>>>>> >> >> >>>>> >>>>> > > > -443256.6711
>>>>>>>> >> >> >>>>> >>>>> > > > -443256.6711
>>>>>>>> >> >> >>>>> >>>>> > > >
>>>>>>>> >> >> >>>>> >>>>> > > > NTPR=200 Etot after 2000 steps
>>>>>>>> >> >> >>>>> >>>>> > > >
>>>>>>>> >> >> >>>>> >>>>> > > > -443261.0705
>>>>>>>> >> >> >>>>> >>>>> > > > -443261.0705
>>>>>>>> >> >> >>>>> >>>>> > > >
>>>>>>>> >> >> >>>>> >>>>> > > > Any idea why energies should depend on
>>>>>>>> frequency
>>>>>>>> of
>>>>>>>> >> >> >>>>> energy
>>>>>>>> >> >> >>>>> >>>>> records
>>>>>>>> >> >> >>>>> >>>>> > (NTPR)
>>>>>>>> >> >> >>>>> >>>>> > > ?
>>>>>>>> >> >> >>>>> >>>>> > > >
>>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>>> >> >> >>>>> >>>>> > > It is a subtle point, but the answer is
>>>>>>>> 'different
>>>>>>>> >> code
>>>>>>>> >> >> >>>>> paths.'
>>>>>>>> >> >> >>>>> >>>>> In
>>>>>>>> >> >> >>>>> >>>>> > > general, it is NEVER necessary to compute the
>>>>>>>> actual
>>>>>>>> >> >> energy
>>>>>>>> >> >> >>>>> of a
>>>>>>>> >> >> >>>>> >>>>> molecule
>>>>>>>> >> >> >>>>> >>>>> > > during the course of standard molecular
>>>>>>>> dynamics
>>>>>>>> (by
>>>>>>>> >> >> >>>>> analogy, it
>>>>>>>> >> >> >>>>> >>>>> is
>>>>>>>> >> >> >>>>> >>>>> NEVER
>>>>>>>> >> >> >>>>> >>>>> > > necessary to compute atomic forces during the
>>>>>>>> course
>>>>>>>> >> of
>>>>>>>> >> >> >>>>> random
>>>>>>>> >> >> >>>>> >>>>> Monte
>>>>>>>> >> >> >>>>> >>>>> > Carlo
>>>>>>>> >> >> >>>>> >>>>> > > sampling).
>>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>>> >> >> >>>>> >>>>> > > For performance's sake, then, pmemd.cuda
>>>>>>>> computes
>>>>>>>> >> only
>>>>>>>> >> >> the
>>>>>>>> >> >> >>>>> force
>>>>>>>> >> >> >>>>> >>>>> when
>>>>>>>> >> >> >>>>> >>>>> > > energies are not requested, leading to a
>>>>>>>> different
>>>>>>>> >> >> order of
>>>>>>>> >> >> >>>>> >>>>> operations
>>>>>>>> >> >> >>>>> >>>>> > for
>>>>>>>> >> >> >>>>> >>>>> > > those runs. This difference ultimately causes
>>>>>>>> >> >> divergence.
>>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>>> >> >> >>>>> >>>>> > > To test this, try setting the variable
>>>>>>>> >> >> ene_avg_sampling=10
>>>>>>>> >> >> >>>>> in
>>>>>>>> >> >> >>>>> the
>>>>>>>> >> >> >>>>> >>>>> &cntrl
>>>>>>>> >> >> >>>>> >>>>> > > section. This will force pmemd.cuda to compute
>>>>>>>> >> energies
>>>>>>>> >> >> >>>>> every 10
>>>>>>>> >> >> >>>>> >>>>> steps
>>>>>>>> >> >> >>>>> >>>>> > > (for energy averaging), which will in turn
>>>>>>>> make
>>>>>>>> the
>>>>>>>> >> >> >>>>> followed
>>>>>>>> >> >> >>>>> code
>>>>>>>> >> >> >>>>> >>>>> path
>>>>>>>> >> >> >>>>> >>>>> > > identical for any multiple-of-10 value of
>>>>>>>> ntpr.
>>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>>> >> >> >>>>> >>>>> > > --
>>>>>>>> >> >> >>>>> >>>>> > > Jason M. Swails
>>>>>>>> >> >> >>>>> >>>>> > > Quantum Theory Project,
>>>>>>>> >> >> >>>>> >>>>> > > University of Florida
>>>>>>>> >> >> >>>>> >>>>> > > Ph.D. Candidate
>>>>>>>> >> >> >>>>> >>>>> > > 352-392-4032
>>>>>>>> >> >> >>>>> >>>>> > > ______________________________**
>>>>>>>> **_________________
>>>>>>>> >> >> >>>>> >>>>> > > AMBER mailing list
>>>>>>>> >> >> >>>>> >>>>> > > AMBER.ambermd.org
>>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>>> >> >> >>>>> >>>>> http://lists.ambermd.org/****
>>>>>>>> mailman/listinfo/amber<http://lists.ambermd.org/**mailman/listinfo/amber>
>>>>>>>> <
>>>>>>>> >> >> >>>>> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>>> >
>>>>>>>> >> >> >>>>> >>>>> "
>>>>>>>> >> >> >>>>> >>>>> > > ______________________________**
>>>>>>>> **_________________
>>>>>>>> >> >> >>>>> >>>>> > > AMBER mailing list
>>>>>>>> >> >> >>>>> >>>>> > > AMBER.ambermd.org
>>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>>> >> >> >>>>> >>>>> http://lists.ambermd.org/****
>>>>>>>> mailman/listinfo/amber<http://lists.ambermd.org/**mailman/listinfo/amber>
>>>>>>>> <
>>>>>>>> >> >> >>>>> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>>> >
>>>>>>>> >> >> >>>>> >>>>> > >
>>>>>>>> >> >> >>>>> >>>>> > ______________________________**
>>>>>>>> **_________________
>>>>>>>> >> >> >>>>> >>>>> > AMBER mailing list
>>>>>>>> >> >> >>>>> >>>>> > AMBER.ambermd.org
>>>>>>>> >> >> >>>>> >>>>> >
>>>>>>>> >> >> >>>>> >>>>> http://lists.ambermd.org/****
>>>>>>>> mailman/listinfo/amber<http://lists.ambermd.org/**mailman/listinfo/amber>
>>>>>>>> <
>>>>>>>> >> >> >>>>> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>>> >
>>>>>>>> >> >> >>>>> >>>>> > ______________________________**
>>>>>>>> **_________________
>>>>>>>> >> >> >>>>> >>>>> > AMBER mailing list
>>>>>>>> >> >> >>>>> >>>>> > AMBER.ambermd.org
>>>>>>>> >> >> >>>>> >>>>> >
>>>>>>>> >> >> >>>>> >>>>> http://lists.ambermd.org/****
>>>>>>>> mailman/listinfo/amber<http://lists.ambermd.org/**mailman/listinfo/amber>
>>>>>>>> <
>>>>>>>> >> >> >>>>> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>>> >
>>>>>>>> >> >> >>>>> >>>>> >
>>>>>>>> >> >> >>>>> >>>>>
>>>>>>>> >> >> >>>>> >>>>>
>>>>>>>> >> >> >>>>> >>>>>
>>>>>>>> >> >> >>>>> >>>>> --
>>>>>>>> >> >> >>>>> >>>>> Jason M. Swails
>>>>>>>> >> >> >>>>> >>>>> Quantum Theory Project,
>>>>>>>> >> >> >>>>> >>>>> University of Florida
>>>>>>>> >> >> >>>>> >>>>> Ph.D. Candidate
>>>>>>>> >> >> >>>>> >>>>> 352-392-4032
>>>>>>>> >> >> >>>>> >>>>> ______________________________**
>>>>>>>> **_________________
>>>>>>>> >> >> >>>>> >>>>> AMBER mailing list
>>>>>>>> >> >> >>>>> >>>>> AMBER.ambermd.org
>>>>>>>> >> >> >>>>> >>>>> http://lists.ambermd.org/****
>>>>>>>> mailman/listinfo/amber<http://lists.ambermd.org/**mailman/listinfo/amber>
>>>>>>>> <
>>>>>>>> >> >> >>>>> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>>> >
>>>>>>>> >> >> >>>>> >>>>>
>>>>>>>> >> >> >>>>> >>>>>
>>>>>>>> >> >> >>>>> >>>> ______________________________**
>>>>>>>> **_________________
>>>>>>>> >> >> >>>>> >>> AMBER mailing list
>>>>>>>> >> >> >>>>> >>> AMBER.ambermd.org
>>>>>>>> >> >> >>>>> >>> http://lists.ambermd.org/****mailman/listinfo/amber<http://lists.ambermd.org/**mailman/listinfo/amber>
>>>>>>>> <
>>>>>>>> >> >> >>>>> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>>> >
>>>>>>>> >> >> >>>>> >>>
>>>>>>>> >> >> >>>>> >>> __________ Informace od ESET NOD32 Antivirus, verze
>>>>>>>> >> databaze
>>>>>>>> >> >> 8394
>>>>>>>> >> >> >>>>> >>> (20130530) __________
>>>>>>>> >> >> >>>>> >>>
>>>>>>>> >> >> >>>>> >>> Tuto zpravu proveril ESET NOD32 Antivirus.
>>>>>>>> >> >> >>>>> >>>
>>>>>>>> >> >> >>>>> >>> http://www.eset.cz
>>>>>>>> >> >> >>>>> >>>
>>>>>>>> >> >> >>>>> >>>
>>>>>>>> >> >> >>>>> >>>
>>>>>>>> >> >> >>>>> >>>
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >> --
>>>>>>>> >> >> >>>>> >> Tato zpráva byla vytvořena převratným poštovním
>>>>>>>> klientem
>>>>>>>> > Opery:
>>>>>>>> >> >> >>>>> >> http://www.opera.com/mail/
>>>>>>>> >> >> >>>>> >> ______________________________**_________________
>>>>>>>> >> >> >>>>> >> AMBER mailing list
>>>>>>>> >> >> >>>>> >> AMBER.ambermd.org
>>>>>>>> >> >> >>>>> >> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> >>
>>>>>>>> >> >> >>>>> > ______________________________**_________________
>>>>>>>> >> >> >>>>> > AMBER mailing list
>>>>>>>> >> >> >>>>> > AMBER.ambermd.org
>>>>>>>> >> >> >>>>> > http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>>> >> >> >>>>> >
>>>>>>>> >> >> >>>>> > __________ Informace od ESET NOD32 Antivirus, verze
>>>>>>>> databaze
>>>>>>>> >> >> 8394
>>>>>>>> >> >> >>>>> > (20130530) __________
>>>>>>>> >> >> >>>>> >
>>>>>>>> >> >> >>>>> > Tuto zpravu proveril ESET NOD32 Antivirus.
>>>>>>>> >> >> >>>>> >
>>>>>>>> >> >> >>>>> > http://www.eset.cz
>>>>>>>> >> >> >>>>> >
>>>>>>>> >> >> >>>>> >
>>>>>>>> >> >> >>>>> >
>>>>>>>> >> >> >>>>>
>>>>>>>> >> >> >>>>>
>>>>>>>> >> >> >>>>> --
>>>>>>>> >> >> >>>>> Tato zpráva byla vytvořena převratným poštovním klientem
>>>>>>>> Opery:
>>>>>>>> >> >> >>>>> http://www.opera.com/mail/
>>>>>>>> >> >> >>>>>
>>>>>>>> >> >> >>>>> ______________________________**_________________
>>>>>>>> >> >> >>>>> AMBER mailing list
>>>>>>>> >> >> >>>>> AMBER.ambermd.org
>>>>>>>> >> >> >>>>> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>>> >> >> >>>>>
>>>>>>>> >> >> >>>>
>>>>>>>> >> >> >>>>
>>>>>>>> >> >> >>> ______________________________**_________________
>>>>>>>> >> >> >>> AMBER mailing list
>>>>>>>> >> >> >>> AMBER.ambermd.org
>>>>>>>> >> >> >>> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>> __________ Informace od ESET NOD32 Antivirus, verze
>>>>>>>> databaze
>>>>>>>> 8395
>>>>>>>> >> >> >>> (20130531) __________
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>> Tuto zpravu proveril ESET NOD32 Antivirus.
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>> http://www.eset.cz
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>
>>>>>>>> >> >> >>
>>>>>>>> >> >> >
>>>>>>>> >> >> >
>>>>>>>> >> >>
>>>>>>>> >> >>
>>>>>>>> >> >> --
>>>>>>>> >> >> Tato zpráva byla vytvořena převratným poštovním klientem
>>>>>>>> Opery:
>>>>>>>> >> >> http://www.opera.com/mail/
>>>>>>>> >> >>
>>>>>>>> >> >> ______________________________**_________________
>>>>>>>> >> >> AMBER mailing list
>>>>>>>> >> >> AMBER.ambermd.org
>>>>>>>> >> >> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>>> >> >>
>>>>>>>> >> >
>>>>>>>> >> >
>>>>>>>> >> >
>>>>>>>> >> > __________ Informace od ESET NOD32 Antivirus, verze databaze
>>>>>>>> 8397
>>>>>>>> >> > (20130531) __________
>>>>>>>> >> >
>>>>>>>> >> > Tuto zpravu proveril ESET NOD32 Antivirus.
>>>>>>>> >> >
>>>>>>>> >> > GB_out_plus_diff_Files.tar.gz - poskozeny archiv
>>>>>>>> >> > GB_out_plus_diff_Files.tar.gz > GZIP >
>>>>>>>> >> GB_out_plus_diff_Files.tar
>>>>>>>> >> > - poskozeny archiv
>>>>>>>> >> > GB_out_plus_diff_Files.tar.gz > GZIP >
>>>>>>>> >> > GB_out_plus_diff_Files.tar > TAR >
>>>>>>>> GB_out_plus_diff_Files.tar.gz -
>>>>>>>> >> > poskozeny archiv
>>>>>>>> >> > GB_out_plus_diff_Files.tar.gz > GZIP >
>>>>>>>> >> > GB_out_plus_diff_Files.tar > TAR >
>>>>>>>> GB_out_plus_diff_Files.tar.gz >
>>>>>>>> >> GZIP
>>>>>>>> >> > > GB_out_plus_diff_Files.tar - poskozeny archiv
>>>>>>>> >> > GB_out_plus_diff_Files.tar.gz > GZIP >
>>>>>>>> >> > GB_out_plus_diff_Files.tar > TAR >
>>>>>>>> GB_out_plus_diff_Files.tar.gz >
>>>>>>>> >> GZIP
>>>>>>>> >> > > GB_out_plus_diff_Files.tar > TAR >
>>>>>>>> GB_nucleosome-sim3.mdout-full -
>>>>>>>> >> > vyskytl se problem pri cteni archivu
>>>>>>>> >> > PME_out_plus_diff_Files.tar.gz - poskozeny archiv
>>>>>>>> >> > PME_out_plus_diff_Files.tar.gz > GZIP >
>>>>>>>> >> > PME_out_plus_diff_Files.tar - poskozeny archiv
>>>>>>>> >> > PME_out_plus_diff_Files.tar.gz > GZIP >
>>>>>>>> >> > PME_out_plus_diff_Files.tar > TAR >
>>>>>>>> PME_out_plus_diff_Files.tar.gz -
>>>>>>>> >> > poskozeny archiv
>>>>>>>> >> > PME_out_plus_diff_Files.tar.gz > GZIP >
>>>>>>>> >> > PME_out_plus_diff_Files.tar > TAR >
>>>>>>>> PME_out_plus_diff_Files.tar.gz >
>>>>>>>> >> > GZIP > PME_out_plus_diff_Files.tar - poskozeny archiv
>>>>>>>> >> > PME_out_plus_diff_Files.tar.gz > GZIP >
>>>>>>>> >> > PME_out_plus_diff_Filestar > TAR >
>>>>>>>> PME_out_plus_diff_Files.tar.gz
>>>>>>>> >
>>>>>>>> >> GZIP
>>>>>>>> >> > > PME_out_plus_diff_Files.tar > TAR >
>>>>>>>> >> > PME_JAC_production_NPT-sim3.**mdout-full - vyskytl se problem
>>>>>>>> pri
>>>>>>>> cteni
>>>>>>>> >> > archivu
>>>>>>>> >> >
>>>>>>>> >> > http://www.eset.cz
>>>>>>>> >> >
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> --
>>>>>>>> >> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
>>>>>>>> >> http://www.opera.com/mail/
>>>>>>>> >>
>>>>>>>> >> ______________________________**_________________
>>>>>>>> >> AMBER mailing list
>>>>>>>> >> AMBER.ambermd.org
>>>>>>>> >> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>>> > ______________________________**_________________
>>>>>>>> > AMBER mailing list
>>>>>>>> > AMBER.ambermd.org
>>>>>>>> > http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>>> >
>>>>>>>> > __________ Informace od ESET NOD32 Antivirus, verze databaze 8398
>>>>>>>> > (20130531) __________
>>>>>>>> >
>>>>>>>> > Tuto zpravu proveril ESET NOD32 Antivirus.
>>>>>>>> >
>>>>>>>> > http://www.eset.cz
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
>>>>>>>> http://www.opera.com/mail/
>>>>>>>>
>>>>>>>> ______________________________**_________________
>>>>>>>> AMBER mailing list
>>>>>>>> AMBER.ambermd.org
>>>>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<http://lists.ambermd.org/mailman/listinfo/amber>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> __________ Informace od ESET NOD32 Antivirus, verze databaze 8401
>>>>>> (20130601) __________
>>>>>>
>>>>>> Tuto zpravu proveril ESET NOD32 Antivirus.
>>>>>>
>>>>>> http://www.eset.cz
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
>>>> http://www.opera.com/mail/
>>>>
>>>> _______________________________________________
>>>> AMBER mailing list
>>>> AMBER.ambermd.org
>>>> http://lists.ambermd.org/mailman/listinfo/amber
>>>>
>>>>
>>>
>>
>
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Sun Jun 02 2013 - 10:00:03 PDT
