OK, here is for the moment the immediate "snap"
of my nohup.out file during CUDA 5.5 running tests:
---------------
JAC_PRODUCTION_NVE - 23,558 atoms PME
-------------------------------------
1 x GTX_TITAN: | ns/day = 113.05 seconds/ns =
764.27
JAC_PRODUCTION_NPT - 23,558 atoms PME
-------------------------------------
1 x GTX_TITAN: Nonbond cells need to be recalculated, restart
simulation from previous checkpoint
with a higher value for skinnb.
| ns/day = 86.34 seconds/ns = 1000.68
FACTOR_IX_PRODUCTION_NVE - 90,906 atoms PME
-------------------------------------------
1 x GTX_TITAN: | ns/day = 30.21 seconds/ns =
2860.07
---------------
so it seems that going to CUDA 5.5 version it self will not
solve the Titan problem.
M.
Dne Sun, 02 Jun 2013 19:15:23 +0200 Scott Le Grand <varelse2005.gmail.com>
napsal/-a:
> Hey Marek,
> No explanation whatsoever! That's why this is *stil*l a bug! I'm just
> being thorough and giving every perp at the scene of the crime a thorough
> shakedown...
>
> Ever read/watch one of those locked door murder mysteries? Same thing,
> different context :-)...
>
> Scott
>
>
>
> On Sun, Jun 2, 2013 at 9:44 AM, Marek Maly <marek.maly.ujep.cz> wrote:
>
>> Hi Scott thanks for the update !
>>
>> Anyway any explanation regarding "cuFFT hypothesis" why there are no
>> problems
>> with GTX 580, GTX 680 or even K20c ???
>>
>>
>> meanwhile I also tried to recompile GPU part of Amber with
>> cuda 5.5 installed before, I have obtained these errs
>> already in configure phase:
>>
>> --------
>> [root.dyn-138-272 amber12]# ./configure -cuda -noX11 gnu
>> Checking for updates...
>> Checking for available patches online. This may take a few seconds...
>>
>> Available AmberTools 13 patches:
>>
>> No patches available
>>
>> Available Amber 12 patches:
>>
>> No patches available
>> Searching for python2... Found python2.6: /usr/bin/python2.6
>> Error: Unsupported CUDA version 5.5 detected.
>> AMBER requires CUDA version == 4.2 .or. 5.0
>> Configure failed due to the errors above!
>> ---------
>>
>> so it seems that Amber is possible to compile only with CUDA 4.2 or 5.0
>> at
>> the moment:
>>
>> and this part of configure2 file has to be edited:
>>
>>
>> -----------
>> nvcc="$CUDA_HOME/bin/nvcc"
>> sm35flags='-gencode arch=compute_35,code=sm_35'
>> sm30flags='-gencode arch=compute_30,code=sm_30'
>> sm20flags='-gencode arch=compute_20,code=sm_20'
>> sm13flags='-gencode arch=compute_13,code=sm_13'
>> nvccflags="$sm13flags $sm20flags"
>> cudaversion=`$nvcc --version | grep 'release' | cut -d' ' -f5 | cut
>> -d',' -f1`
>> if [ "$cudaversion" == "5.0" ]; then
>> echo "CUDA Version $cudaversion detected"
>> nvccflags="$nvccflags $sm30flags $sm35flags"
>> elif [ "$cudaversion" == "4.2" ]; then
>> echo "CUDA Version $cudaversion detected"
>> nvccflags="$nvccflags $sm30flags"
>> else
>> echo "Error: Unsupported CUDA version $cudaversion detected."
>> echo "AMBER requires CUDA version == 4.2 .or. 5.0"
>> exit 1
>> fi
>> nvcc="$nvcc $nvccflags"
>>
>> fi
>>
>> -----------
>>
>> would it be just OK to change
>> "if [ "$cudaversion" == "5.0" ]; then"
>>
>> to
>>
>> "if [ "$cudaversion" == "5.5" ]; then"
>>
>>
>> or some more flags etc. should be defined here to proceed successfully ?
>>
>>
>> BTW it seems Scott, that you are on the way to isolate the problem soon
>> so maybe it's better to wait and not to loose time with cuda 5.5
>> experiments.
>>
>> I just thought that cuda 5.5 might be more "friendly" to Titans :)) e.g.
>> in terms of cuFFT function ....
>>
>>
>> I will keep fingers crossed :))
>>
>> M.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Dne Sun, 02 Jun 2013 18:33:52 +0200 Scott Le Grand
>> <varelse2005.gmail.com>
>> napsal/-a:
>>
>> > PS this *might* indicate a software bug in cuFFT, but it needs more
>> > characterization... And things are going to get a little stream of
>> > consciousness from here because you're getting unfiltered raw data, so
>> > please don't draw any conclusions towards anything yet - I'm just
>> letting
>> > you guys know what I'm finding out as I find it...
>> >
>> >
>> >
>> > On Sun, Jun 2, 2013 at 9:31 AM, Scott Le Grand
>> > <varelse2005.gmail.com>wrote:
>> >
>> >> And bingo...
>> >>
>> >> At the very least, the reciprocal sum is intermittently
>> inconsistent...
>> >> This explains the irreproducible behavior...
>> >>
>> >> And here's the level of inconsistency:
>> >> 31989.38940628897399 vs
>> >> 31989.39168370794505
>> >>
>> >> That's error at the level of 1e-7 or a somehow missed
>> single-precision
>> >> transaction somewhere...
>> >>
>> >> The next question is figuring out why... This may or may not
>> ultimately
>> >> explain the crashes you guys are also seeing...
>> >>
>> >>
>> >>
>> >> On Sun, Jun 2, 2013 at 9:07 AM, Scott Le Grand
>> >> <varelse2005.gmail.com>wrote:
>> >>
>> >>>
>> >>> Observations:
>> >>> 1. The degree to which the reproducibility is broken *does* appear
>> to
>> >>> vary between individual Titan GPUs. One of my Titans breaks within
>> 10K
>> >>> steps on cellulose, the other one made it to 100K steps twice
>> without
>> >>> doing
>> >>> so leading me to believe it could be trusted (until yesterday where
>> I
>> >>> now
>> >>> see it dies between 50K and 100K steps most of the time).
>> >>>
>> >>> 2. GB hasn't broken (yet). So could you run myoglobin for 500K and
>> >>> TRPcage for 1,000,000 steps and let's see if that's universal.
>> >>>
>> >>> 3. Turning on double-precision mode makes my Titan crash rather than
>> >>> run
>> >>> irreproducibly, sigh...
>> >>>
>> >>> So whatever is going on is triggered by something in PME but not GB.
>> >>> So
>> >>> that's either the radix sort, the FFT, the Ewald grid interpolation,
>> >>> or the
>> >>> neighbor list code. Fixing this involves isolating this and
>> figuring
>> >>> out
>> >>> what exactly goes haywire. It could *still* be software at some
>> very
>> >>> small
>> >>> probability but the combination of both 680 and K20c with ECC off
>> >>> running
>> >>> reliably is really pointing towards the Titans just being clocked
>> too
>> >>> fast.
>> >>>
>> >>> So how long with this take? Asking people how long it takes to fix
>> a
>> >>> bug
>> >>> never really works out well. That said, I found the 480 bug within
>> a
>> >>> week
>> >>> and my usual turnaround for a bug with a solid repro is <24 hours.
>> >>>
>> >>> Scott
>> >>>
>> >>> On Sun, Jun 2, 2013 at 7:58 AM, Marek Maly <marek.maly.ujep.cz>
>> wrote:
>> >>>
>> >>>> Hi all,
>> >>>>
>> >>>> here are my results after bugfix 18 application (see attachment).
>> >>>>
>> >>>> In principle I don't see any "drastical" changes.
>> >>>>
>> >>>> FACTOR_IX still perfectly stable/reproducible on both cards,
>> >>>>
>> >>>> JAC tests - problems with finishing AND/OR reproducibility the
>> >>>> same CELLULOSE_NVE although here it seems that my TITAN_1
>> >>>> has no problems with this test (but the same same trend I saw also
>> >>>> before bugfix 18 - see my older 500K steps test).
>> >>>>
>> >>>> But anyway bugfix 18 brought here one change.
>> >>>>
>> >>>> The err
>> >>>>
>> >>>>
>> >>>> #1 ERR writtent in mdout:
>> >>>> ------
>> >>>> | ERROR: max pairlist cutoff must be less than unit cell max
>> sphere
>> >>>> radius!
>> >>>> ------
>> >>>>
>> >>>> was substituted with err/warning ?
>> >>>>
>> >>>> #0 no ERR writtent in mdout, ERR written in standard output
>> >>>> (nohup.out)
>> >>>> -----
>> >>>> Nonbond cells need to be recalculated, restart simulation from
>> >>>> previous
>> >>>> checkpoint
>> >>>> with a higher value for skinnb.
>> >>>>
>> >>>> -----
>> >>>>
>> >>>> Another thing,
>> >>>>
>> >>>> recently I started on another machine and GTX 580 GPU simulation of
>> >>>> relatively
>> >>>> big system ( 364275 atoms/PME ). The system is composed also from
>> the
>> >>>> "exotic" molecules like polymers. ff12SB, gaff, GLYCAM forcefields
>> >>>> used
>> >>>> here. I had problem even with minimization part here, having big
>> >>>> energy
>> >>>> on the start:
>> >>>>
>> >>>> -----
>> >>>> NSTEP ENERGY RMS GMAX NAME
>> >>>> NUMBER
>> >>>> 1 2.8442E+09 2.1339E+02 1.7311E+04 O
>> >>>> 32998
>> >>>>
>> >>>> BOND = 11051.7467 ANGLE = 17720.4706 DIHED =
>> >>>> 18977.7584
>> >>>> VDWAALS = ************* EEL = -1257709.6203 HBOND =
>> >>>> 0.0000
>> >>>> 1-4 VDW = 7253.7412 1-4 EEL = 149867.0207 RESTRAINT =
>> >>>> 0.0000
>> >>>>
>> >>>> ----
>> >>>>
>> >>>> with no chance to minimize the system even with 50 000 steps in
>> both
>> >>>> min cycles (with constrained and unconstrained solute) and hence
>> >>>> heating
>> >>>> NVT
>> >>>> crashed immediately even with very small dt. I patched Amber12 here
>> >>>> with
>> >>>> the
>> >>>> bugfix 18 and the minimization was done without any problem with
>> >>>> common
>> >>>> 5000 steps
>> >>>> (obtaining target Energy -1.4505E+06 while that initial was that
>> >>>> written
>> >>>> above).
>> >>>>
>> >>>> So indeed bugfix 18 solved some issues, but unfortunately not those
>> >>>> related to
>> >>>> Titans.
>> >>>>
>> >>>> Here I will try to install cuda 5.5, recompile GPU Amber part with
>> >>>> this
>> >>>> new
>> >>>> cuda version and repeat the 100K tests.
>> >>>>
>> >>>> Scott, let us know how finished your experiment with downclocking
>> of
>> >>>> Titan.
>> >>>> Maybe the best choice would be here to flash Titan directly with
>> your
>> >>>> K20c bios :))
>> >>>>
>> >>>> M.
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> Dne Sat, 01 Jun 2013 21:09:46 +0200 Marek Maly <marek.maly.ujep.cz>
>> >>>> napsal/-a:
>> >>>>
>> >>>>
>> >>>> Hi,
>> >>>>>
>> >>>>> first of all thanks for providing of your test results !
>> >>>>>
>> >>>>> It seems that your results are more or less similar to that of
>> >>>>> mine maybe with the exception of the results on FactorIX tests
>> >>>>> where I had perfect stability and 100% or close to 100%
>> >>>>> reproducibility.
>> >>>>>
>> >>>>> Anyway the type of errs which you reported are the same which I
>> >>>>> obtained.
>> >>>>>
>> >>>>> So let's see if the bugfix 18 will help here (or at least on NPT
>> >>>>> tests)
>> >>>>> or not. As I wrote just before few minutes, it seems that it was
>> not
>> >>>>> still
>> >>>>> loaded
>> >>>>> to the given server, although it's description is already present
>> on
>> >>>>> the
>> >>>>> given
>> >>>>> web page ( see
>> >>>>> http://ambermd.org/bugfixes12.**html<
>> http://ambermd.org/bugfixes12.html>).
>> >>>>>
>> >>>>> As you can see, this bugfix contains also changes in CPU code
>> >>>>> although
>> >>>>> the majority is devoted to GPU code, so perhaps the best will be
>> to
>> >>>>> recompile
>> >>>>> whole amber with this patch although this patch would be perhaps
>> >>>>> applied
>> >>>>> even after just
>> >>>>> GPU configure command ( i.e. ./configure -cuda -noX11 gnu ) but
>> after
>> >>>>> consequent
>> >>>>> building, just the GPU binaries will be updated. Anyway I would
>> >>>>> rather
>> >>>>> recompile
>> >>>>> whole Amber after this patch.
>> >>>>>
>> >>>>> Regarding to GPU test under linux you may try memtestG80
>> >>>>> (please use the updated/patched version from here
>> >>>>> https://github.com/ihaque/**memtestG80<
>> https://github.com/ihaque/memtestG80>
>> >>>>> )
>> >>>>>
>> >>>>> just use git command like:
>> >>>>>
>> >>>>> git clone
>> >>>>> https://github.com/ihaque/**memtestG80.git<
>> https://github.com/ihaque/memtestG80.git>PATCHED_MEMTEST-G80
>> >>>>>
>> >>>>> to download all the files and save them into directory named
>> >>>>> PATCHED_MEMTEST-G80.
>> >>>>>
>> >>>>> another possibility is to try perhaps similar (but maybe more up
>> to
>> >>>>> date)
>> >>>>> test
>> >>>>> cuda_memtest (
>> >>>>> http://sourceforge.net/**projects/cudagpumemtest/<
>> http://sourceforge.net/projects/cudagpumemtest/>).
>> >>>>>
>> >>>>> regarding ig value: If ig is not present in mdin, the default
>> value
>> >>>>> is
>> >>>>> used
>> >>>>> (e.g. 71277) if ig=-1 the random seed will be based on the current
>> >>>>> date
>> >>>>> and time, and hence will be different for every run (not a good
>> >>>>> variant
>> >>>>> for our testts). I simply deleted eventual ig records from all
>> mdins
>> >>>>> so
>> >>>>> I
>> >>>>> assume that in each run the default seed 71277 was automatically
>> >>>>> used.
>> >>>>>
>> >>>>> M.
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> Dne Sat, 01 Jun 2013 20:26:16 +0200 ET <sketchfoot.gmail.com>
>> >>>>> napsal/-a:
>> >>>>>
>> >>>>> Hi,
>> >>>>>>
>> >>>>>> I've put the graphics card into a machine with the working GTX
>> titan
>> >>>>>> that I
>> >>>>>> mentioned earlier.
>> >>>>>>
>> >>>>>> The Nvidia driver version is: 133.30
>> >>>>>>
>> >>>>>> Amber version is:
>> >>>>>> AmberTools version 13.03
>> >>>>>> Amber version 12.16
>> >>>>>>
>> >>>>>> I ran 50k steps with the amber benchmark using ig=43689 on both
>> >>>>>> cards.
>> >>>>>> For
>> >>>>>> the purpose of discriminating between them, the card I believe
>> >>>>>> (fingers
>> >>>>>> crossed) is working is called GPU-00_TeaNCake, whilst the other
>> one
>> >>>>>> is
>> >>>>>> called GPU-01_008.
>> >>>>>>
>> >>>>>> *When I run the tests on GPU-01_008:*
>> >>>>>>
>> >>>>>> 1) All the tests (across 2x repeats) finish apart from the
>> >>>>>> following
>> >>>>>> which
>> >>>>>> have the errors listed:
>> >>>>>>
>> >>>>>> ------------------------------**--------------
>> >>>>>> CELLULOSE_PRODUCTION_NVE - 408,609 atoms PME
>> >>>>>> Error: unspecified launch failure launching kernel kNLSkinTest
>> >>>>>> cudaFree GpuBuffer::Deallocate failed unspecified launch failure
>> >>>>>>
>> >>>>>> ------------------------------**--------------
>> >>>>>> CELLULOSE_PRODUCTION_NPT - 408,609 atoms PME
>> >>>>>> cudaMemcpy GpuBuffer::Download failed unspecified launch failure
>> >>>>>>
>> >>>>>> ------------------------------**--------------
>> >>>>>> CELLULOSE_PRODUCTION_NVE - 408,609 atoms PME
>> >>>>>> Error: unspecified launch failure launching kernel kNLSkinTest
>> >>>>>> cudaFree GpuBuffer::Deallocate failed unspecified launch failure
>> >>>>>>
>> >>>>>> ------------------------------**--------------
>> >>>>>> CELLULOSE_PRODUCTION_NPT - 408,609 atoms PME
>> >>>>>> cudaMemcpy GpuBuffer::Download failed unspecified launch failure
>> >>>>>> grep: mdinfo.1GTX680: No such file or directory
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> 2) The sdiff logs indicate that reproducibility across the two
>> >>>>>> repeats
>> >>>>>> is
>> >>>>>> as follows:
>> >>>>>>
>> >>>>>> *GB_myoglobin: *Reproducible across 50k steps
>> >>>>>> *GB_nucleosome:* Reproducible till step 7400
>> >>>>>> *GB_TRPCage:* Reproducible across 50k steps
>> >>>>>>
>> >>>>>> *PME_JAC_production_NVE: *No reproducibility shown from step
>> 1,000
>> >>>>>> onwards
>> >>>>>> *PME_JAC_production_NPT*: Reproducible till step 1,000. Also
>> >>>>>> outfile
>> >>>>>> is
>> >>>>>> not written properly - blank gaps appear where something should
>> have
>> >>>>>> been
>> >>>>>> written
>> >>>>>>
>> >>>>>> *PME_FactorIX_production_NVE:* Reproducible across 50k steps
>> >>>>>> *PME_FactorIX_production_NPT:* Reproducible across 50k steps
>> >>>>>>
>> >>>>>> *PME_Cellulose_production_NVE:*** Failure means that both runs do
>> >>>>>> not
>> >>>>>> finish
>> >>>>>> (see point1)
>> >>>>>> *PME_Cellulose_production_NPT: *Failure means that both runs do
>> not
>> >>>>>> finish
>> >>>>>> (see point1)
>> >>>>>>
>> >>>>>> ##############################**##############################**
>> >>>>>> ###########################
>> >>>>>>
>> >>>>>> *When I run the tests on * *GPU-00_TeaNCake:*
>> >>>>>> *
>> >>>>>> *
>> >>>>>> 1) All the tests (across 2x repeats) finish apart from the
>> >>>>>> following
>> >>>>>> which
>> >>>>>> have the errors listed:
>> >>>>>> ------------------------------**-------
>> >>>>>> JAC_PRODUCTION_NPT - 23,558 atoms PME
>> >>>>>> PMEMD Terminated Abnormally!
>> >>>>>> ------------------------------**-------
>> >>>>>>
>> >>>>>>
>> >>>>>> 2) The sdiff logs indicate that reproducibility across the two
>> >>>>>> repeats
>> >>>>>> is
>> >>>>>> as follows:
>> >>>>>>
>> >>>>>> *GB_myoglobin:* Reproducible across 50k steps
>> >>>>>> *GB_nucleosome:* Reproducible across 50k steps
>> >>>>>> *GB_TRPCage:* Reproducible across 50k steps
>> >>>>>>
>> >>>>>> *PME_JAC_production_NVE:* No reproducibility shown from step
>> 10,000
>> >>>>>> onwards
>> >>>>>> *PME_JAC_production_NPT: * No reproducibility shown from step
>> 10,000
>> >>>>>> onwards. Also outfile is not written properly - blank gaps appear
>> >>>>>> where
>> >>>>>> something should have been written. Repeat 2 Crashes with error
>> >>>>>> noted
>> >>>>>> in
>> >>>>>> 1.
>> >>>>>>
>> >>>>>> *PME_FactorIX_production_NVE:* No reproducibility shown from step
>> >>>>>> 9,000
>> >>>>>> onwards
>> >>>>>> *PME_FactorIX_production_NPT: *Reproducible across 50k steps
>> >>>>>>
>> >>>>>> *PME_Cellulose_production_NVE: *No reproducibility shown from
>> step
>> >>>>>> 5,000
>> >>>>>> onwards
>> >>>>>> *PME_Cellulose_production_NPT: ** *No reproducibility shown from
>> >>>>>> step
>> >>>>>> 29,000 onwards. Also outfile is not written properly - blank gaps
>> >>>>>> appear
>> >>>>>> where something should have been written.
>> >>>>>>
>> >>>>>>
>> >>>>>> Out files and sdiff files are included as attatchments
>> >>>>>>
>> >>>>>> ##############################**###################
>> >>>>>>
>> >>>>>> So I'm going to update my nvidia driver to the latest version and
>> >>>>>> patch
>> >>>>>> amber to the latest version and rerun the tests to see if there
>> is
>> >>>>>> any
>> >>>>>> improvement. Could someone let me know if it is necessary to
>> >>>>>> recompile
>> >>>>>> any
>> >>>>>> or all of AMBER after applying the bugfixes?
>> >>>>>>
>> >>>>>> Additionally, I'm going to run memory tests and heaven
>> benchmarks on
>> >>>>>> the
>> >>>>>> cards to check whether they are faulty or not.
>> >>>>>>
>> >>>>>> I'm thinking that there is a mix of hardware error/configuration
>> >>>>>> (esp
>> >>>>>> in
>> >>>>>> the case of GPU-01_008) and amber software error in this
>> situation.
>> >>>>>> What
>> >>>>>> do
>> >>>>>> you guys think?
>> >>>>>>
>> >>>>>> Also am I right in thinking (from what Scott was saying) that all
>> >>>>>> the
>> >>>>>> benchmarks should be reproducible across 50k steps but begin to
>> >>>>>> diverge
>> >>>>>> at
>> >>>>>> around 100K steps? Is there any difference from in setting *ig
>> *to
>> >>>>>> an
>> >>>>>> explicit number to removing it from the mdin file?
>> >>>>>>
>> >>>>>> br,
>> >>>>>> g
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> On 31 May 2013 23:45, ET <sketchfoot.gmail.com> wrote:
>> >>>>>>
>> >>>>>> I don't need sysadmins, but sysadmins need me as it gives
>> purpose
>> >>>>>> to
>> >>>>>>> their
>> >>>>>>> bureaucratic existence. A encountered evil if working in an
>> >>>>>>> institution
>> >>>>>>> or
>> >>>>>>> comapny IMO. Good science and indiviguality being sacrificed for
>> >>>>>>> standardisation and mediocrity in the intrerests of maintaing a
>> >>>>>>> system
>> >>>>>>> that
>> >>>>>>> focusses on maintaining the system and not the objective.
>> >>>>>>>
>> >>>>>>> You need root to move fwd on these things, unfortunately. and
>> ppl
>> >>>>>>> with
>> >>>>>>> root are kinda like your parents when you try to borrow money
>> from
>> >>>>>>> them
>> >>>>>>> .
>> >>>>>>> age 12 :D
>> >>>>>>> On May 31, 2013 9:34 PM, "Marek Maly" <marek.maly.ujep.cz>
>> wrote:
>> >>>>>>>
>> >>>>>>> Sorry why do you need sysadmins :)) ?
>> >>>>>>>>
>> >>>>>>>> BTW here is the most recent driver:
>> >>>>>>>>
>> >>>>>>>> http://www.nvidia.com/object/**linux-display-amd64-319.23-**
>> >>>>>>>> driver.html<
>> http://www.nvidia.com/object/linux-display-amd64-319.23-driver.html>
>> >>>>>>>>
>> >>>>>>>> I do not remember anything easier than is to install driver
>> >>>>>>>> (especially
>> >>>>>>>> in case of binary (*.run) installer) :))
>> >>>>>>>>
>> >>>>>>>> M.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> Dne Fri, 31 May 2013 22:02:34 +0200 ET <sketchfoot.gmail.com>
>> >>>>>>>> napsal/-a:
>> >>>>>>>>
>> >>>>>>>> > Yup. I know. I replaced a 680 and the everknowing sysadmins
>> are
>> >>>>>>>> reluctant
>> >>>>>>>> > to install drivers not in the repositoery as they are lame.
>> :(
>> >>>>>>>> > On May 31, 2013 7:14 PM, "Marek Maly" <marek.maly.ujep.cz>
>> >>>>>>>> wrote:
>> >>>>>>>> >>
>> >>>>>>>> >> As I already wrote you,
>> >>>>>>>> >>
>> >>>>>>>> >> the first driver which properly/officially supports Titans,
>> >>>>>>>> should
>> >>>>>>>> be
>> >>>>>>>> >> 313.26 .
>> >>>>>>>> >>
>> >>>>>>>> >> Anyway I am curious mainly about your 100K repetitive tests
>> >>>>>>>> with
>> >>>>>>>> >> your Titan SC card. Especially in case of these tests (
>> >>>>>>>> JAC_NVE,
>> >>>>>>>> JAC_NPT
>> >>>>>>>> >> and CELLULOSE_NVE ) where
>> >>>>>>>> >> my Titans SC randomly failed or succeeded. In FACTOR_IX_NVE,
>> >>>>>>>> >> FACTOR_IX_NPT
>> >>>>>>>> >> tests both
>> >>>>>>>> >> my cards are perfectly stable (independently from drv.
>> version)
>> >>>>>>>> and
>> >>>>>>>> also
>> >>>>>>>> >> the runs
>> >>>>>>>> >> are perfectly or almost perfectly reproducible.
>> >>>>>>>> >>
>> >>>>>>>> >> Also if your test will crash please report the eventual
>> errs.
>> >>>>>>>> >>
>> >>>>>>>> >> To this moment I have this actual library of errs on my
>> Titans
>> >>>>>>>> SC
>> >>>>>>>> GPUs.
>> >>>>>>>> >>
>> >>>>>>>> >> #1 ERR writtent in mdout:
>> >>>>>>>> >> ------
>> >>>>>>>> >> | ERROR: max pairlist cutoff must be less than unit cell
>> max
>> >>>>>>>> sphere
>> >>>>>>>> >> radius!
>> >>>>>>>> >> ------
>> >>>>>>>> >>
>> >>>>>>>> >>
>> >>>>>>>> >> #2 no ERR writtent in mdout, ERR written in standard output
>> >>>>>>>> (nohup.out)
>> >>>>>>>> >>
>> >>>>>>>> >> ----
>> >>>>>>>> >> Error: unspecified launch failure launching kernel
>> kNLSkinTest
>> >>>>>>>> >> cudaFree GpuBuffer::Deallocate failed unspecified launch
>> >>>>>>>> failure
>> >>>>>>>> >> ----
>> >>>>>>>> >>
>> >>>>>>>> >>
>> >>>>>>>> >> #3 no ERR writtent in mdout, ERR written in standard output
>> >>>>>>>> (nohup.out)
>> >>>>>>>> >> ----
>> >>>>>>>> >> cudaMemcpy GpuBuffer::Download failed unspecified launch
>> >>>>>>>> failure
>> >>>>>>>> >> ----
>> >>>>>>>> >>
>> >>>>>>>> >> Another question, regarding your Titan SC, it is also EVGA
>> as
>> >>>>>>>> in
>> >>>>>>>> my
>> >>>>>>>> case
>> >>>>>>>> >> or it is another producer ?
>> >>>>>>>> >>
>> >>>>>>>> >> Thanks,
>> >>>>>>>> >>
>> >>>>>>>> >> M.
>> >>>>>>>> >>
>> >>>>>>>> >>
>> >>>>>>>> >>
>> >>>>>>>> >> Dne Fri, 31 May 2013 19:17:03 +0200 ET
>> <sketchfoot.gmail.com>
>> >>>>>>>> napsal/-a:
>> >>>>>>>> >>
>> >>>>>>>> >> > Well, this is interesting...
>> >>>>>>>> >> >
>> >>>>>>>> >> > I ran 50k steps on the Titan on the other machine with
>> driver
>> >>>>>>>> 310.44
>> >>>>>>>> >> and
>> >>>>>>>> >> > it
>> >>>>>>>> >> > passed all the GB steps. i.e totally identical results
>> over
>> >>>>>>>> two
>> >>>>>>>> >> repeats.
>> >>>>>>>> >> > However, it failed all the PME tests after step 1000. I'm
>> >>>>>>>> going
>> >>>>>>>> to
>> >>>>>>>> > update
>> >>>>>>>> >> > the driver and test it again.
>> >>>>>>>> >> >
>> >>>>>>>> >> > Files included as attachments.
>> >>>>>>>> >> >
>> >>>>>>>> >> > br,
>> >>>>>>>> >> > g
>> >>>>>>>> >> >
>> >>>>>>>> >> >
>> >>>>>>>> >> > On 31 May 2013 16:40, Marek Maly <marek.maly.ujep.cz>
>> wrote:
>> >>>>>>>> >> >
>> >>>>>>>> >> >> One more thing,
>> >>>>>>>> >> >>
>> >>>>>>>> >> >> can you please check under which frequency is running
>> that
>> >>>>>>>> your
>> >>>>>>>> >> titan ?
>> >>>>>>>> >> >>
>> >>>>>>>> >> >> As the base frequency of normal Titans is 837MHz and the
>> >>>>>>>> Boost
>> >>>>>>>> one
>> >>>>>>>> is
>> >>>>>>>> >> >> 876MHz I
>> >>>>>>>> >> >> assume that yor GPU is running automatically also under
>> it's
>> >>>>>>>> boot
>> >>>>>>>> >> >> frequency (876MHz).
>> >>>>>>>> >> >> You can find this information e.g. in Amber mdout file.
>> >>>>>>>> >> >>
>> >>>>>>>> >> >> You also mentioned some crashes in your previous email.
>> Your
>> >>>>>>>> ERRs
>> >>>>>>>> >> were
>> >>>>>>>> >> >> something like those here:
>> >>>>>>>> >> >>
>> >>>>>>>> >> >> #1 ERR writtent in mdout:
>> >>>>>>>> >> >> ------
>> >>>>>>>> >> >> | ERROR: max pairlist cutoff must be less than unit
>> cell
>> >>>>>>>> max
>> >>>>>>>> sphere
>> >>>>>>>> >> >> radius!
>> >>>>>>>> >> >> ------
>> >>>>>>>> >> >>
>> >>>>>>>> >> >>
>> >>>>>>>> >> >> #2 no ERR writtent in mdout, ERR written in standard
>> output
>> >>>>>>>> >> (nohup.out)
>> >>>>>>>> >> >>
>> >>>>>>>> >> >> ----
>> >>>>>>>> >> >> Error: unspecified launch failure launching kernel
>> >>>>>>>> kNLSkinTest
>> >>>>>>>> >> >> cudaFree GpuBuffer::Deallocate failed unspecified launch
>> >>>>>>>> failure
>> >>>>>>>> >> >> ----
>> >>>>>>>> >> >>
>> >>>>>>>> >> >>
>> >>>>>>>> >> >> #3 no ERR writtent in mdout, ERR written in standard
>> output
>> >>>>>>>> >> (nohup.out)
>> >>>>>>>> >> >> ----
>> >>>>>>>> >> >> cudaMemcpy GpuBuffer::Download failed unspecified launch
>> >>>>>>>> failure
>> >>>>>>>> >> >> ----
>> >>>>>>>> >> >>
>> >>>>>>>> >> >> or you obtained some new/additional errs ?
>> >>>>>>>> >> >>
>> >>>>>>>> >> >>
>> >>>>>>>> >> >>
>> >>>>>>>> >> >> M.
>> >>>>>>>> >> >>
>> >>>>>>>> >> >>
>> >>>>>>>> >> >>
>> >>>>>>>> >> >> Dne Fri, 31 May 2013 17:30:57 +0200 filip fratev
>> >>>>>>>> >> <filipfratev.yahoo.com
>> >>>>>>>> >>
>> >>>>>>>> >> >> napsal/-a:
>> >>>>>>>> >> >>
>> >>>>>>>> >> >> > Hi,
>> >>>>>>>> >> >> > This is what I obtained for 50K tests and "normal"
>> >>>>>>>> GTXTitan:
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> > run1:
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >>
>> >>>>>>>> >
>> >>>>>>>>
>> ------------------------------**------------------------------**
>> >>>>>>>> ------------------
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> > A V E R A G E S O V E R 50 S T E P S
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> > NSTEP = 50000 TIME(PS) = 120.020 TEMP(K) =
>> >>>>>>>> 299.87
>> >>>>>>>> >> PRESS
>> >>>>>>>> >> >> > = 0.0
>> >>>>>>>> >> >> > Etot = -443237.1079 EKtot = 257679.9750
>> EPtot
>> >>>>>>>> =
>> >>>>>>>> >> >> > -700917.0829
>> >>>>>>>> >> >> > BOND = 20193.1856 ANGLE = 53517.5432
>> DIHED
>> >>>>>>>> =
>> >>>>>>>> >> >> > 23575.4648
>> >>>>>>>> >> >> > 1-4 NB = 21759.5524 1-4 EEL = 742552.5939
>> >>>>>>>> VDWAALS
>> >>>>>>>> =
>> >>>>>>>> >> >> > 96286.7714
>> >>>>>>>> >> >> > EELEC = -1658802.1941 EHBOND = 0.0000
>> >>>>>>>> RESTRAINT
>> >>>>>>>> =
>> >>>>>>>> >> >> > 0.0000
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >>
>> >>>>>>>> >
>> >>>>>>>>
>> ------------------------------**------------------------------**
>> >>>>>>>> ------------------
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> > R M S F L U C T U A T I O N S
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> > NSTEP = 50000 TIME(PS) = 120.020 TEMP(K) =
>> >>>>>>>> 0.33
>> >>>>>>>> >> PRESS
>> >>>>>>>> >> >> > = 0.0
>> >>>>>>>> >> >> > Etot = 11.2784 EKtot = 284.8999
>> EPtot
>> >>>>>>>> =
>> >>>>>>>> >> >> > 289.0773
>> >>>>>>>> >> >> > BOND = 136.3417 ANGLE = 214.0054
>> DIHED
>> >>>>>>>> =
>> >>>>>>>> >> >> > 59.4893
>> >>>>>>>> >> >> > 1-4 NB = 58.5891 1-4 EEL = 330.5400
>> >>>>>>>> VDWAALS
>> >>>>>>>> =
>> >>>>>>>> >> >> > 559.2079
>> >>>>>>>> >> >> > EELEC = 743.8771 EHBOND = 0.0000
>> >>>>>>>> RESTRAINT
>> >>>>>>>> =
>> >>>>>>>> >> >> > 0.0000
>> >>>>>>>> >> >> > |E(PBS) = 21.8119
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >>
>> >>>>>>>> >
>> >>>>>>>>
>> ------------------------------**------------------------------**
>> >>>>>>>> ------------------
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> > run2:
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >>
>> >>>>>>>> >
>> >>>>>>>>
>> ------------------------------**------------------------------**
>> >>>>>>>> ------------------
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> > A V E R A G E S O V E R 50 S T E P S
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> > NSTEP = 50000 TIME(PS) = 120.020 TEMP(K) =
>> >>>>>>>> 299.89
>> >>>>>>>> >> PRESS
>> >>>>>>>> >> >> > = 0.0
>> >>>>>>>> >> >> > Etot = -443240.0999 EKtot = 257700.0950
>> EPtot
>> >>>>>>>> =
>> >>>>>>>> >> >> > -700940.1949
>> >>>>>>>> >> >> > BOND = 20241.9174 ANGLE = 53644.6694
>> DIHED
>> >>>>>>>> =
>> >>>>>>>> >> >> > 23541.3737
>> >>>>>>>> >> >> > 1-4 NB = 21803.1898 1-4 EEL = 742754.2254
>> >>>>>>>> VDWAALS
>> >>>>>>>> =
>> >>>>>>>> >> >> > 96298.8308
>> >>>>>>>> >> >> > EELEC = -1659224.4013 EHBOND = 0.0000
>> >>>>>>>> RESTRAINT
>> >>>>>>>> =
>> >>>>>>>> >> >> > 0.0000
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >>
>> >>>>>>>> >
>> >>>>>>>>
>> ------------------------------**------------------------------**
>> >>>>>>>> ------------------
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> > R M S F L U C T U A T I O N S
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> > NSTEP = 50000 TIME(PS) = 120.020 TEMP(K) =
>> >>>>>>>> 0.41
>> >>>>>>>> >> PRESS
>> >>>>>>>> >> >> > = 0.0
>> >>>>>>>> >> >> > Etot = 10.7633 EKtot = 348.2819
>> EPtot
>> >>>>>>>> =
>> >>>>>>>> >> >> > 353.9918
>> >>>>>>>> >> >> > BOND = 106.5314 ANGLE = 196.7052
>> DIHED
>> >>>>>>>> =
>> >>>>>>>> >> >> > 69.7476
>> >>>>>>>> >> >> > 1-4 NB = 60.3435 1-4 EEL = 400.7466
>> >>>>>>>> VDWAALS
>> >>>>>>>> =
>> >>>>>>>> >> >> > 462.7763
>> >>>>>>>> >> >> > EELEC = 651.9857 EHBOND = 0.0000
>> >>>>>>>> RESTRAINT
>> >>>>>>>> =
>> >>>>>>>> >> >> > 0.0000
>> >>>>>>>> >> >> > |E(PBS) = 17.0642
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >>
>> >>>>>>>> >
>> >>>>>>>>
>> ------------------------------**------------------------------**
>> >>>>>>>> ------------------
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >>
>> >>>>>>>> >
>> >>>>>>>>
>> ------------------------------**------------------------------**
>> >>>>>>>> --------------------
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> > ______________________________**__
>> >>>>>>>> >> >> > From: Marek Maly <marek.maly.ujep.cz>
>> >>>>>>>> >> >> > To: AMBER Mailing List <amber.ambermd.org>
>> >>>>>>>> >> >> > Sent: Friday, May 31, 2013 3:34 PM
>> >>>>>>>> >> >> > Subject: Re: [AMBER] experiences with EVGA GTX TITAN
>> >>>>>>>> Superclocked
>> >>>>>>>> -
>> >>>>>>>> >> >> > memtestG80 - UNDERclocking in Linux ?
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> > Hi here are my 100K results for driver 313.30 (and
>> still
>> >>>>>>>> Cuda
>> >>>>>>>> 5.0).
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> > The results are rather similar to those obtained
>> >>>>>>>> >> >> > under my original driver 319.17 (see the first table
>> >>>>>>>> >> >> > which I sent in this thread).
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> > M.
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> > Dne Fri, 31 May 2013 12:29:59 +0200 Marek Maly <
>> >>>>>>>> marek.maly.ujep.cz>
>> >>>>>>>> >> >> > napsal/-a:
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> >> Hi,
>> >>>>>>>> >> >> >>
>> >>>>>>>> >> >> >> please try to run at lest 100K tests twice to verify
>> >>>>>>>> exact
>> >>>>>>>> >> >> >> reproducibility
>> >>>>>>>> >> >> >> of the results on the given card. If you find in any
>> mdin
>> >>>>>>>> file
>> >>>>>>>> >> ig=-1
>> >>>>>>>> >> >> >> just
>> >>>>>>>> >> >> >> delete it to ensure that you are using the identical
>> >>>>>>>> random
>> >>>>>>>> seed
>> >>>>>>>> >> for
>> >>>>>>>> >> >> >> both
>> >>>>>>>> >> >> >> runs. You can eventually omit NUCLEOSOME test
>> >>>>>>>> >> >> >> as it is too much time consuming.
>> >>>>>>>> >> >> >>
>> >>>>>>>> >> >> >> Driver 310.44 ?????
>> >>>>>>>> >> >> >>
>> >>>>>>>> >> >> >> As far as I know the proper support for titans is from
>> >>>>>>>> version
>> >>>>>>>> > 313.26
>> >>>>>>>> >> >> >>
>> >>>>>>>> >> >> >> see e.g. here :
>> >>>>>>>> >> >> >>
>> >>>>>>>> >> >>
>> >>>>>>>> >
>> >>>>>>>> http://www.geeks3d.com/**20130306/nvidia-releases-r313-**
>> >>>>>>>> 26-for-linux-with-gtx-titan-**support/<
>> http://www.geeks3d.com/20130306/nvidia-releases-r313-26-for-linux-with-gtx-titan-support/
>> >
>> >>>>>>>> >> >> >>
>> >>>>>>>> >> >> >> BTW: On my site downgrade to drv. 313.30 did not
>> solved
>> >>>>>>>> the
>> >>>>>>>> >> >> situation, I
>> >>>>>>>> >> >> >> will post
>> >>>>>>>> >> >> >> my results soon here.
>> >>>>>>>> >> >> >>
>> >>>>>>>> >> >> >> M.
>> >>>>>>>> >> >> >>
>> >>>>>>>> >> >> >>
>> >>>>>>>> >> >> >>
>> >>>>>>>> >> >> >>
>> >>>>>>>> >> >> >>
>> >>>>>>>> >> >> >>
>> >>>>>>>> >> >> >>
>> >>>>>>>> >> >> >>
>> >>>>>>>> >> >> >> Dne Fri, 31 May 2013 12:21:21 +0200 ET <
>> >>>>>>>> sketchfoot.gmail.com>
>> >>>>>>>> >> >> napsal/-a:
>> >>>>>>>> >> >> >>
>> >>>>>>>> >> >> >>> ps. I have another install of amber on another
>> computer
>> >>>>>>>> with
>> >>>>>>>> a
>> >>>>>>>> >> >> >>> different
>> >>>>>>>> >> >> >>> Titan and different Driver Version: 310.44.
>> >>>>>>>> >> >> >>>
>> >>>>>>>> >> >> >>> In the interests of thrashing the proverbial horse,
>> I'll
>> >>>>>>>> run
>> >>>>>>>> the
>> >>>>>>>> >> >> >>> benchmark
>> >>>>>>>> >> >> >>> for 50k steps. :P
>> >>>>>>>> >> >> >>>
>> >>>>>>>> >> >> >>> br,
>> >>>>>>>> >> >> >>> g
>> >>>>>>>> >> >> >>>
>> >>>>>>>> >> >> >>>
>> >>>>>>>> >> >> >>> On 31 May 2013 11:17, ET <sketchfoot.gmail.com>
>> wrote:
>> >>>>>>>> >> >> >>>
>> >>>>>>>> >> >> >>>> Hi, I just ran the Amber benchmark for the default
>> >>>>>>>> (10000
>> >>>>>>>> steps)
>> >>>>>>>> >> >> on my
>> >>>>>>>> >> >> >>>> Titan.
>> >>>>>>>> >> >> >>>>
>> >>>>>>>> >> >> >>>> Using sdiff -sB showed that the two runs were
>> >>>>>>>> completely
>> >>>>>>>> > identical.
>> >>>>>>>> >> >> >>>> I've
>> >>>>>>>> >> >> >>>> attached compressed files of the mdout & diff files.
>> >>>>>>>> >> >> >>>>
>> >>>>>>>> >> >> >>>> br,
>> >>>>>>>> >> >> >>>> g
>> >>>>>>>> >> >> >>>>
>> >>>>>>>> >> >> >>>>
>> >>>>>>>> >> >> >>>> On 30 May 2013 23:41, Marek Maly
>> <marek.maly.ujep.cz>
>> >>>>>>>> wrote:
>> >>>>>>>> >> >> >>>>
>> >>>>>>>> >> >> >>>>> OK, let's see. The eventual downclocking I see as
>> the
>> >>>>>>>> very
>> >>>>>>>> last
>> >>>>>>>> >> >> >>>>> possibility
>> >>>>>>>> >> >> >>>>> (if I don't decide for RMAing). But now still some
>> >>>>>>>> other
>> >>>>>>>> >> >> experiments
>> >>>>>>>> >> >> >>>>> are
>> >>>>>>>> >> >> >>>>> available :))
>> >>>>>>>> >> >> >>>>> I just started 100K tests under 313.30 driver. For
>> >>>>>>>> today
>> >>>>>>>> good
>> >>>>>>>> >> >> night
>> >>>>>>>> >> >> >>>>> ...
>> >>>>>>>> >> >> >>>>>
>> >>>>>>>> >> >> >>>>> M.
>> >>>>>>>> >> >> >>>>>
>> >>>>>>>> >> >> >>>>> Dne Fri, 31 May 2013 00:45:49 +0200 Scott Le Grand
>> >>>>>>>> >> >> >>>>> <varelse2005.gmail.com
>> >>>>>>>> >> >> >>>>> >
>> >>>>>>>> >> >> >>>>> napsal/-a:
>> >>>>>>>> >> >> >>>>>
>> >>>>>>>> >> >> >>>>> > It will be very interesting if this behavior
>> >>>>>>>> persists
>> >>>>>>>> after
>> >>>>>>>> >> >> >>>>> downclocking.
>> >>>>>>>> >> >> >>>>> >
>> >>>>>>>> >> >> >>>>> > But right now, Titan 0 *looks* hosed and Titan 1
>> >>>>>>>> *looks*
>> >>>>>>>> like
>> >>>>>>>> > it
>> >>>>>>>> >> >> >>>>> needs
>> >>>>>>>> >> >> >>>>> > downclocking...
>> >>>>>>>> >> >> >>>>> > On May 30, 2013 3:20 PM, "Marek Maly"
>> >>>>>>>> <marek.maly.ujep.cz>
>> >>>>>>>> >> >> wrote:
>> >>>>>>>> >> >> >>>>> >
>> >>>>>>>> >> >> >>>>> >> Hi all,
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >> here are my results from the 500K steps 2 x
>> >>>>>>>> repeated
>> >>>>>>>> > benchmarks
>> >>>>>>>> >> >> >>>>> >> under 319.23 driver and still Cuda 5.0 (see the
>> >>>>>>>> attached
>> >>>>>>>> >> table
>> >>>>>>>> >> >> ).
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >> It is hard to say if the results are better or
>> >>>>>>>> worse
>> >>>>>>>> than
>> >>>>>>>> in
>> >>>>>>>> > my
>> >>>>>>>> >> >> >>>>> >> previous 100K test under driver 319.17.
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >> While results from Cellulose test were improved
>> and
>> >>>>>>>> the
>> >>>>>>>> > TITAN_1
>> >>>>>>>> >> >> >>>>> card
>> >>>>>>>> >> >> >>>>> >> even
>> >>>>>>>> >> >> >>>>> >> successfully finished all 500K steps moreover
>> with
>> >>>>>>>> exactly
>> >>>>>>>> >> the
>> >>>>>>>> >> >> >>>>> same
>> >>>>>>>> >> >> >>>>> >> final
>> >>>>>>>> >> >> >>>>> >> energy !
>> >>>>>>>> >> >> >>>>> >> (TITAN_0 at least finished more than 100K steps
>> >>>>>>>> and in
>> >>>>>>>> >> RUN_01
>> >>>>>>>> >> >> even
>> >>>>>>>> >> >> >>>>> more
>> >>>>>>>> >> >> >>>>> >> than 400K steps)
>> >>>>>>>> >> >> >>>>> >> In JAC_NPT test no GPU was able to finish at
>> least
>> >>>>>>>> 100K
>> >>>>>>>> >> steps
>> >>>>>>>> >> >> and
>> >>>>>>>> >> >> >>>>> the
>> >>>>>>>> >> >> >>>>> >> results from JAC_NVE
>> >>>>>>>> >> >> >>>>> >> test are also not too much convincing.
>> >>>>>>>> FACTOR_IX_NVE
>> >>>>>>>> and
>> >>>>>>>> >> >> >>>>> FACTOR_IX_NPT
>> >>>>>>>> >> >> >>>>> >> were successfully
>> >>>>>>>> >> >> >>>>> >> finished with 100% reproducibility in
>> >>>>>>>> FACTOR_IX_NPT
>> >>>>>>>> case
>> >>>>>>>> >> (on
>> >>>>>>>> >> >> both
>> >>>>>>>> >> >> >>>>> >> cards)
>> >>>>>>>> >> >> >>>>> >> and almost
>> >>>>>>>> >> >> >>>>> >> 100% reproducibility in case of FACTOR_IX_NVE
>> >>>>>>>> (again
>> >>>>>>>> 100%
>> >>>>>>>> in
>> >>>>>>>> >> >> case
>> >>>>>>>> >> >> >>>>> of
>> >>>>>>>> >> >> >>>>> >> TITAN_1). TRPCAGE, MYOGLOBIN
>> >>>>>>>> >> >> >>>>> >> again finished without any problem with 100%
>> >>>>>>>> >> reproducibility.
>> >>>>>>>> >> >> >>>>> NUCLEOSOME
>> >>>>>>>> >> >> >>>>> >> test was not done
>> >>>>>>>> >> >> >>>>> >> this time due to high time requirements. If you
>> >>>>>>>> find
>> >>>>>>>> in
>> >>>>>>>> the
>> >>>>>>>> >> >> table
>> >>>>>>>> >> >> >>>>> >> positive
>> >>>>>>>> >> >> >>>>> >> number finishing with
>> >>>>>>>> >> >> >>>>> >> K (which means "thousands") it means the last
>> >>>>>>>> number
>> >>>>>>>> of
>> >>>>>>>> step
>> >>>>>>>> >> >> >>>>> written in
>> >>>>>>>> >> >> >>>>> >> mdout before crash.
>> >>>>>>>> >> >> >>>>> >> Below are all the 3 types of detected errs with
>> >>>>>>>> relevant
>> >>>>>>>> >> >> >>>>> systems/rounds
>> >>>>>>>> >> >> >>>>> >> where the given err
>> >>>>>>>> >> >> >>>>> >> appeared.
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >> Now I will try just 100K tests under ETs
>> favourite
>> >>>>>>>> driver
>> >>>>>>>> >> >> version
>> >>>>>>>> >> >> >>>>> 313.30
>> >>>>>>>> >> >> >>>>> >> :)) and then
>> >>>>>>>> >> >> >>>>> >> I will eventually try to experiment with cuda
>> 5.5
>> >>>>>>>> which
>> >>>>>>>> I
>> >>>>>>>> >> >> already
>> >>>>>>>> >> >> >>>>> >> downloaded from the
>> >>>>>>>> >> >> >>>>> >> cuda zone ( I had to become cuda developer for
>> this
>> >>>>>>>> :))
>> >>>>>>>> )
>> >>>>>>>> >> BTW
>> >>>>>>>> >> >> ET
>> >>>>>>>> >> >> >>>>> thanks
>> >>>>>>>> >> >> >>>>> >> for the frequency info !
>> >>>>>>>> >> >> >>>>> >> and I am still ( perhaps not alone :)) ) very
>> >>>>>>>> curious
>> >>>>>>>> about
>> >>>>>>>> >> >> your 2
>> >>>>>>>> >> >> >>>>> x
>> >>>>>>>> >> >> >>>>> >> repeated Amber benchmark tests with superclocked
>> >>>>>>>> Titan.
>> >>>>>>>> >> Indeed
>> >>>>>>>> >> >> >>>>> that
>> >>>>>>>> >> >> >>>>> I
>> >>>>>>>> >> >> >>>>> am
>> >>>>>>>> >> >> >>>>> >> very curious also about that Ross "hot" patch.
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >> M.
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >> ERRORS DETECTED DURING THE 500K steps tests with
>> >>>>>>>> driver
>> >>>>>>>> >> 319.23
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >> #1 ERR writtent in mdout:
>> >>>>>>>> >> >> >>>>> >> ------
>> >>>>>>>> >> >> >>>>> >> | ERROR: max pairlist cutoff must be less than
>> >>>>>>>> unit
>> >>>>>>>> cell
>> >>>>>>>> >> max
>> >>>>>>>> >> >> >>>>> sphere
>> >>>>>>>> >> >> >>>>> >> radius!
>> >>>>>>>> >> >> >>>>> >> ------
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >> TITAN_0 ROUND_1 JAC_NPT (at least 5000 steps
>> >>>>>>>> successfully
>> >>>>>>>> > done
>> >>>>>>>> >> >> >>>>> before
>> >>>>>>>> >> >> >>>>> >> crash)
>> >>>>>>>> >> >> >>>>> >> TITAN_0 ROUND_2 JAC_NPT (at least 8000 steps
>> >>>>>>>> successfully
>> >>>>>>>> > done
>> >>>>>>>> >> >> >>>>> before
>> >>>>>>>> >> >> >>>>> >> crash)
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >> #2 no ERR writtent in mdout, ERR written in
>> >>>>>>>> standard
>> >>>>>>>> output
>> >>>>>>>> >> >> >>>>> (nohup.out)
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >> ----
>> >>>>>>>> >> >> >>>>> >> Error: unspecified launch failure launching
>> kernel
>> >>>>>>>> >> kNLSkinTest
>> >>>>>>>> >> >> >>>>> >> cudaFree GpuBuffer::Deallocate failed
>> unspecified
>> >>>>>>>> launch
>> >>>>>>>> >> >> failure
>> >>>>>>>> >> >> >>>>> >> ----
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >> TITAN_0 ROUND_1 CELLULOSE_NVE (at least 437 000
>> >>>>>>>> steps
>> >>>>>>>> >> >> successfully
>> >>>>>>>> >> >> >>>>> done
>> >>>>>>>> >> >> >>>>> >> before crash)
>> >>>>>>>> >> >> >>>>> >> TITAN_0 ROUND_2 JAC_NVE (at least 162 000 steps
>> >>>>>>>> >> successfully
>> >>>>>>>> >> >> done
>> >>>>>>>> >> >> >>>>> >> before
>> >>>>>>>> >> >> >>>>> >> crash)
>> >>>>>>>> >> >> >>>>> >> TITAN_0 ROUND_2 CELLULOSE_NVE (at least 117 000
>> >>>>>>>> steps
>> >>>>>>>> >> >> successfully
>> >>>>>>>> >> >> >>>>> done
>> >>>>>>>> >> >> >>>>> >> before crash)
>> >>>>>>>> >> >> >>>>> >> TITAN_1 ROUND_1 JAC_NVE (at least 119 000 steps
>> >>>>>>>> >> successfully
>> >>>>>>>> >> >> done
>> >>>>>>>> >> >> >>>>> >> before
>> >>>>>>>> >> >> >>>>> >> crash)
>> >>>>>>>> >> >> >>>>> >> TITAN_1 ROUND_2 JAC_NVE (at least 43 000 steps
>> >>>>>>>> successfully
>> >>>>>>>> >> >> done
>> >>>>>>>> >> >> >>>>> before
>> >>>>>>>> >> >> >>>>> >> crash)
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >> #3 no ERR writtent in mdout, ERR written in
>> >>>>>>>> standard
>> >>>>>>>> output
>> >>>>>>>> >> >> >>>>> (nohup.out)
>> >>>>>>>> >> >> >>>>> >> ----
>> >>>>>>>> >> >> >>>>> >> cudaMemcpy GpuBuffer::Download failed
>> unspecified
>> >>>>>>>> launch
>> >>>>>>>> >> >> failure
>> >>>>>>>> >> >> >>>>> >> ----
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >> TITAN_1 ROUND_1 JAC_NPT (at least 77 000 steps
>> >>>>>>>> successfully
>> >>>>>>>> >> >> done
>> >>>>>>>> >> >> >>>>> before
>> >>>>>>>> >> >> >>>>> >> crash)
>> >>>>>>>> >> >> >>>>> >> TITAN_1 ROUND_2 JAC_NPT (at least 58 000 steps
>> >>>>>>>> successfully
>> >>>>>>>> >> >> done
>> >>>>>>>> >> >> >>>>> before
>> >>>>>>>> >> >> >>>>> >> crash)
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >> Dne Thu, 30 May 2013 21:27:17 +0200 Scott Le
>> Grand
>> >>>>>>>> >> >> >>>>> >> <varelse2005.gmail.com>
>> >>>>>>>> >> >> >>>>> >> napsal/-a:
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >> Oops meant to send that to Jason...
>> >>>>>>>> >> >> >>>>> >>>
>> >>>>>>>> >> >> >>>>> >>> Anyway, before we all panic, we need to get
>> K20's
>> >>>>>>>> behavior
>> >>>>>>>> >> >> >>>>> analyzed
>> >>>>>>>> >> >> >>>>> >>> here.
>> >>>>>>>> >> >> >>>>> >>> If it's deterministic, this truly is a hardware
>> >>>>>>>> issue. If
>> >>>>>>>> >> >> not,
>> >>>>>>>> >> >> >>>>> then
>> >>>>>>>> >> >> >>>>> it
>> >>>>>>>> >> >> >>>>> >>> gets interesting because 680 is deterministic
>> as
>> >>>>>>>> far
>> >>>>>>>> as I
>> >>>>>>>> >> can
>> >>>>>>>> >> >> >>>>> tell...
>> >>>>>>>> >> >> >>>>> >>> On May 30, 2013 12:24 PM, "Scott Le Grand"
>> >>>>>>>> >> >> >>>>> <varelse2005.gmail.com>
>> >>>>>>>> >> >> >>>>> >>> wrote:
>> >>>>>>>> >> >> >>>>> >>>
>> >>>>>>>> >> >> >>>>> >>> If the errors are not deterministically
>> >>>>>>>> triggered,
>> >>>>>>>> they
>> >>>>>>>> >> >> probably
>> >>>>>>>> >> >> >>>>> >>> won't be
>> >>>>>>>> >> >> >>>>> >>>> fixed by the patch, alas...
>> >>>>>>>> >> >> >>>>> >>>> On May 30, 2013 12:15 PM, "Jason Swails"
>> >>>>>>>> >> >> >>>>> <jason.swails.gmail.com>
>> >>>>>>>> >> >> >>>>> >>>> wrote:
>> >>>>>>>> >> >> >>>>> >>>>
>> >>>>>>>> >> >> >>>>> >>>> Just a reminder to everyone based on what
>> Ross
>> >>>>>>>> said:
>> >>>>>>>> >> there
>> >>>>>>>> >> >> is a
>> >>>>>>>> >> >> >>>>> >>>> pending
>> >>>>>>>> >> >> >>>>> >>>>> patch to pmemd.cuda that will be coming out
>> >>>>>>>> shortly
>> >>>>>>>> >> (maybe
>> >>>>>>>> >> >> even
>> >>>>>>>> >> >> >>>>> >>>>> within
>> >>>>>>>> >> >> >>>>> >>>>> hours). It's entirely possible that several
>> of
>> >>>>>>>> these
>> >>>>>>>> > errors
>> >>>>>>>> >> >> >>>>> are
>> >>>>>>>> >> >> >>>>> >>>>> fixed
>> >>>>>>>> >> >> >>>>> >>>>> by
>> >>>>>>>> >> >> >>>>> >>>>> this patch.
>> >>>>>>>> >> >> >>>>> >>>>>
>> >>>>>>>> >> >> >>>>> >>>>> All the best,
>> >>>>>>>> >> >> >>>>> >>>>> Jason
>> >>>>>>>> >> >> >>>>> >>>>>
>> >>>>>>>> >> >> >>>>> >>>>>
>> >>>>>>>> >> >> >>>>> >>>>> On Thu, May 30, 2013 at 2:46 PM, filip
>> fratev <
>> >>>>>>>> >> >> >>>>> filipfratev.yahoo.com>
>> >>>>>>>> >> >> >>>>> >>>>> wrote:
>> >>>>>>>> >> >> >>>>> >>>>>
>> >>>>>>>> >> >> >>>>> >>>>> > I have observed the same crashes from time
>> to
>> >>>>>>>> time. I
>> >>>>>>>> > will
>> >>>>>>>> >> >> >>>>> run
>> >>>>>>>> >> >> >>>>> >>>>> cellulose
>> >>>>>>>> >> >> >>>>> >>>>> > nve for 100k and will past results here.
>> >>>>>>>> >> >> >>>>> >>>>> >
>> >>>>>>>> >> >> >>>>> >>>>> > All the best,
>> >>>>>>>> >> >> >>>>> >>>>> > Filip
>> >>>>>>>> >> >> >>>>> >>>>> >
>> >>>>>>>> >> >> >>>>> >>>>> >
>> >>>>>>>> >> >> >>>>> >>>>> >
>> >>>>>>>> >> >> >>>>> >>>>> >
>> >>>>>>>> >> >> >>>>> >>>>> > ______________________________****__
>> >>>>>>>> >> >> >>>>> >>>>> > From: Scott Le Grand
>> <varelse2005.gmail.com>
>> >>>>>>>> >> >> >>>>> >>>>> > To: AMBER Mailing List <amber.ambermd.org>
>> >>>>>>>> >> >> >>>>> >>>>> > Sent: Thursday, May 30, 2013 9:01 PM
>> >>>>>>>> >> >> >>>>> >>>>> > Subject: Re: [AMBER] experiences with EVGA
>> GTX
>> >>>>>>>> TITAN
>> >>>>>>>> >> >> >>>>> Superclocked
>> >>>>>>>> >> >> >>>>> -
>> >>>>>>>> >> >> >>>>> >>>>> > memtestG80 - UNDERclocking in Linux ?
>> >>>>>>>> >> >> >>>>> >>>>> >
>> >>>>>>>> >> >> >>>>> >>>>> >
>> >>>>>>>> >> >> >>>>> >>>>> > Run cellulose nve for 100k iterations twice
>> >>>>>>>> . If
>> >>>>>>>> the
>> >>>>>>>> >> >> final
>> >>>>>>>> >> >> >>>>> >>>>> energies
>> >>>>>>>> >> >> >>>>> >>>>> don't
>> >>>>>>>> >> >> >>>>> >>>>> > match, you have a hardware issue. No need
>> to
>> >>>>>>>> play
>> >>>>>>>> with
>> >>>>>>>> >> >> ntpr
>> >>>>>>>> >> >> >>>>> or
>> >>>>>>>> >> >> >>>>> any
>> >>>>>>>> >> >> >>>>> >>>>> other
>> >>>>>>>> >> >> >>>>> >>>>> > variable.
>> >>>>>>>> >> >> >>>>> >>>>> > On May 30, 2013 10:58 AM,
>> >>>>>>>> <pavel.banas.upol.cz>
>> >>>>>>>> wrote:
>> >>>>>>>> >> >> >>>>> >>>>> >
>> >>>>>>>> >> >> >>>>> >>>>> > >
>> >>>>>>>> >> >> >>>>> >>>>> > > Dear all,
>> >>>>>>>> >> >> >>>>> >>>>> > >
>> >>>>>>>> >> >> >>>>> >>>>> > > I would also like to share one of my
>> >>>>>>>> experience
>> >>>>>>>> with
>> >>>>>>>> >> >> titan
>> >>>>>>>> >> >> >>>>> >>>>> cards. We
>> >>>>>>>> >> >> >>>>> >>>>> have
>> >>>>>>>> >> >> >>>>> >>>>> > > one gtx titan card and with one system
>> (~55k
>> >>>>>>>> atoms,
>> >>>>>>>> > NVT,
>> >>>>>>>> >> >> >>>>> >>>>> RNA+waters)
>> >>>>>>>> >> >> >>>>> >>>>> we
>> >>>>>>>> >> >> >>>>> >>>>> > run
>> >>>>>>>> >> >> >>>>> >>>>> > > into same troubles you are describing. I
>> was
>> >>>>>>>> also
>> >>>>>>>> >> >> playing
>> >>>>>>>> >> >> >>>>> with
>> >>>>>>>> >> >> >>>>> >>>>> ntpr
>> >>>>>>>> >> >> >>>>> >>>>> to
>> >>>>>>>> >> >> >>>>> >>>>> > > figure out what is going on, step by
>> step. I
>> >>>>>>>> >> understand
>> >>>>>>>> >> >> >>>>> that
>> >>>>>>>> >> >> >>>>> the
>> >>>>>>>> >> >> >>>>> >>>>> code
>> >>>>>>>> >> >> >>>>> >>>>> is
>> >>>>>>>> >> >> >>>>> >>>>> > > using different routines for calculation
>> >>>>>>>> >> >> energies+forces or
>> >>>>>>>> >> >> >>>>> only
>> >>>>>>>> >> >> >>>>> >>>>> forces.
>> >>>>>>>> >> >> >>>>> >>>>> > > The
>> >>>>>>>> >> >> >>>>> >>>>> > > simulations of other systems are
>> perfectly
>> >>>>>>>> stable,
>> >>>>>>>> >> >> running
>> >>>>>>>> >> >> >>>>> for
>> >>>>>>>> >> >> >>>>> >>>>> days
>> >>>>>>>> >> >> >>>>> >>>>> and
>> >>>>>>>> >> >> >>>>> >>>>> > > weeks. Only that particular system
>> >>>>>>>> systematically
>> >>>>>>>> >> ends
>> >>>>>>>> >> >> up
>> >>>>>>>> >> >> >>>>> with
>> >>>>>>>> >> >> >>>>> >>>>> this
>> >>>>>>>> >> >> >>>>> >>>>> > error.
>> >>>>>>>> >> >> >>>>> >>>>> > >
>> >>>>>>>> >> >> >>>>> >>>>> > > However, there was one interesting issue.
>> >>>>>>>> When
>> >>>>>>>> I
>> >>>>>>>> set
>> >>>>>>>> >> >> >>>>> ntpr=1,
>> >>>>>>>> >> >> >>>>> the
>> >>>>>>>> >> >> >>>>> >>>>> error
>> >>>>>>>> >> >> >>>>> >>>>> > > vanished (systematically in multiple
>> runs)
>> >>>>>>>> and
>> >>>>>>>> the
>> >>>>>>>> >> >> >>>>> simulation
>> >>>>>>>> >> >> >>>>> was
>> >>>>>>>> >> >> >>>>> >>>>> able to
>> >>>>>>>> >> >> >>>>> >>>>> > > run for more than millions of steps (I
>> was
>> >>>>>>>> not
>> >>>>>>>> let
>> >>>>>>>> it
>> >>>>>>>> >> >> >>>>> running
>> >>>>>>>> >> >> >>>>> for
>> >>>>>>>> >> >> >>>>> >>>>> weeks
>> >>>>>>>> >> >> >>>>> >>>>> > as
>> >>>>>>>> >> >> >>>>> >>>>> > > in the meantime I shifted that
>> simulation to
>> >>>>>>>> other
>> >>>>>>>> >> card
>> >>>>>>>> >> >> -
>> >>>>>>>> >> >> >>>>> need
>> >>>>>>>> >> >> >>>>> >>>>> data,
>> >>>>>>>> >> >> >>>>> >>>>> not
>> >>>>>>>> >> >> >>>>> >>>>> > > testing). All other setting of ntpr
>> failed.
>> >>>>>>>> As
>> >>>>>>>> I
>> >>>>>>>> read
>> >>>>>>>> >> >> this
>> >>>>>>>> >> >> >>>>> >>>>> discussion, I
>> >>>>>>>> >> >> >>>>> >>>>> > > tried to set ene_avg_sampling=1 with some
>> >>>>>>>> high
>> >>>>>>>> value
>> >>>>>>>> >> of
>> >>>>>>>> >> >> >>>>> ntpr
>> >>>>>>>> >> >> >>>>> (I
>> >>>>>>>> >> >> >>>>> >>>>> expected
>> >>>>>>>> >> >> >>>>> >>>>> > > that this will shift the code to
>> permanently
>> >>>>>>>> use
>> >>>>>>>> the
>> >>>>>>>> >> >> >>>>> >>>>> force+energies
>> >>>>>>>> >> >> >>>>> >>>>> part
>> >>>>>>>> >> >> >>>>> >>>>> > of
>> >>>>>>>> >> >> >>>>> >>>>> > > the code, similarly to ntpr=1), but the
>> >>>>>>>> error
>> >>>>>>>> >> occurred
>> >>>>>>>> >> >> >>>>> again.
>> >>>>>>>> >> >> >>>>> >>>>> > >
>> >>>>>>>> >> >> >>>>> >>>>> > > I know it is not very conclusive for
>> finding
>> >>>>>>>> out
>> >>>>>>>> what
>> >>>>>>>> > is
>> >>>>>>>> >> >> >>>>> >>>>> happening,
>> >>>>>>>> >> >> >>>>> >>>>> at
>> >>>>>>>> >> >> >>>>> >>>>> > > least
>> >>>>>>>> >> >> >>>>> >>>>> > > not for me. Do you have any idea, why
>> ntpr=1
>> >>>>>>>> might
>> >>>>>>>> > help?
>> >>>>>>>> >> >> >>>>> >>>>> > >
>> >>>>>>>> >> >> >>>>> >>>>> > > best regards,
>> >>>>>>>> >> >> >>>>> >>>>> > >
>> >>>>>>>> >> >> >>>>> >>>>> > > Pavel
>> >>>>>>>> >> >> >>>>> >>>>> > >
>> >>>>>>>> >> >> >>>>> >>>>> > >
>> >>>>>>>> >> >> >>>>> >>>>> > >
>> >>>>>>>> >> >> >>>>> >>>>> > >
>> >>>>>>>> >> >> >>>>> >>>>> > >
>> >>>>>>>> >> >> >>>>> >>>>> > > --
>> >>>>>>>> >> >> >>>>> >>>>> > > Pavel Banáš
>> >>>>>>>> >> >> >>>>> >>>>> > > pavel.banas.upol.cz
>> >>>>>>>> >> >> >>>>> >>>>> > > Department of Physical Chemistry,
>> >>>>>>>> >> >> >>>>> >>>>> > > Palacky University Olomouc
>> >>>>>>>> >> >> >>>>> >>>>> > > Czech Republic
>> >>>>>>>> >> >> >>>>> >>>>> > >
>> >>>>>>>> >> >> >>>>> >>>>> > >
>> >>>>>>>> >> >> >>>>> >>>>> > >
>> >>>>>>>> >> >> >>>>> >>>>> > > ---------- Původní zpráva ----------
>> >>>>>>>> >> >> >>>>> >>>>> > > Od: Jason Swails <jason.swails.gmail.com>
>> >>>>>>>> >> >> >>>>> >>>>> > > Datum: 29. 5. 2013
>> >>>>>>>> >> >> >>>>> >>>>> > > Předmět: Re: [AMBER] experiences with
>> EVGA
>> >>>>>>>> GTX
>> >>>>>>>> TITAN
>> >>>>>>>> >> >> >>>>> >>>>> Superclocked -
>> >>>>>>>> >> >> >>>>> >>>>> > > memtestG
>> >>>>>>>> >> >> >>>>> >>>>> > > 80 - UNDERclocking in Linux ?
>> >>>>>>>> >> >> >>>>> >>>>> > >
>> >>>>>>>> >> >> >>>>> >>>>> > > "I'll answer a little bit:
>> >>>>>>>> >> >> >>>>> >>>>> > >
>> >>>>>>>> >> >> >>>>> >>>>> > > NTPR=10 Etot after 2000 steps
>> >>>>>>>> >> >> >>>>> >>>>> > > >
>> >>>>>>>> >> >> >>>>> >>>>> > > > -443256.6711
>> >>>>>>>> >> >> >>>>> >>>>> > > > -443256.6711
>> >>>>>>>> >> >> >>>>> >>>>> > > >
>> >>>>>>>> >> >> >>>>> >>>>> > > > NTPR=200 Etot after 2000 steps
>> >>>>>>>> >> >> >>>>> >>>>> > > >
>> >>>>>>>> >> >> >>>>> >>>>> > > > -443261.0705
>> >>>>>>>> >> >> >>>>> >>>>> > > > -443261.0705
>> >>>>>>>> >> >> >>>>> >>>>> > > >
>> >>>>>>>> >> >> >>>>> >>>>> > > > Any idea why energies should depend on
>> >>>>>>>> frequency
>> >>>>>>>> of
>> >>>>>>>> >> >> >>>>> energy
>> >>>>>>>> >> >> >>>>> >>>>> records
>> >>>>>>>> >> >> >>>>> >>>>> > (NTPR)
>> >>>>>>>> >> >> >>>>> >>>>> > > ?
>> >>>>>>>> >> >> >>>>> >>>>> > > >
>> >>>>>>>> >> >> >>>>> >>>>> > >
>> >>>>>>>> >> >> >>>>> >>>>> > > It is a subtle point, but the answer is
>> >>>>>>>> 'different
>> >>>>>>>> >> code
>> >>>>>>>> >> >> >>>>> paths.'
>> >>>>>>>> >> >> >>>>> >>>>> In
>> >>>>>>>> >> >> >>>>> >>>>> > > general, it is NEVER necessary to compute
>> >>>>>>>> the
>> >>>>>>>> actual
>> >>>>>>>> >> >> energy
>> >>>>>>>> >> >> >>>>> of a
>> >>>>>>>> >> >> >>>>> >>>>> molecule
>> >>>>>>>> >> >> >>>>> >>>>> > > during the course of standard molecular
>> >>>>>>>> dynamics
>> >>>>>>>> (by
>> >>>>>>>> >> >> >>>>> analogy, it
>> >>>>>>>> >> >> >>>>> >>>>> is
>> >>>>>>>> >> >> >>>>> >>>>> NEVER
>> >>>>>>>> >> >> >>>>> >>>>> > > necessary to compute atomic forces during
>> >>>>>>>> the
>> >>>>>>>> course
>> >>>>>>>> >> of
>> >>>>>>>> >> >> >>>>> random
>> >>>>>>>> >> >> >>>>> >>>>> Monte
>> >>>>>>>> >> >> >>>>> >>>>> > Carlo
>> >>>>>>>> >> >> >>>>> >>>>> > > sampling).
>> >>>>>>>> >> >> >>>>> >>>>> > >
>> >>>>>>>> >> >> >>>>> >>>>> > > For performance's sake, then, pmemd.cuda
>> >>>>>>>> computes
>> >>>>>>>> >> only
>> >>>>>>>> >> >> the
>> >>>>>>>> >> >> >>>>> force
>> >>>>>>>> >> >> >>>>> >>>>> when
>> >>>>>>>> >> >> >>>>> >>>>> > > energies are not requested, leading to a
>> >>>>>>>> different
>> >>>>>>>> >> >> order of
>> >>>>>>>> >> >> >>>>> >>>>> operations
>> >>>>>>>> >> >> >>>>> >>>>> > for
>> >>>>>>>> >> >> >>>>> >>>>> > > those runs. This difference ultimately
>> >>>>>>>> causes
>> >>>>>>>> >> >> divergence.
>> >>>>>>>> >> >> >>>>> >>>>> > >
>> >>>>>>>> >> >> >>>>> >>>>> > > To test this, try setting the variable
>> >>>>>>>> >> >> ene_avg_sampling=10
>> >>>>>>>> >> >> >>>>> in
>> >>>>>>>> >> >> >>>>> the
>> >>>>>>>> >> >> >>>>> >>>>> &cntrl
>> >>>>>>>> >> >> >>>>> >>>>> > > section. This will force pmemd.cuda to
>> >>>>>>>> compute
>> >>>>>>>> >> energies
>> >>>>>>>> >> >> >>>>> every 10
>> >>>>>>>> >> >> >>>>> >>>>> steps
>> >>>>>>>> >> >> >>>>> >>>>> > > (for energy averaging), which will in
>> turn
>> >>>>>>>> make
>> >>>>>>>> the
>> >>>>>>>> >> >> >>>>> followed
>> >>>>>>>> >> >> >>>>> code
>> >>>>>>>> >> >> >>>>> >>>>> path
>> >>>>>>>> >> >> >>>>> >>>>> > > identical for any multiple-of-10 value of
>> >>>>>>>> ntpr.
>> >>>>>>>> >> >> >>>>> >>>>> > >
>> >>>>>>>> >> >> >>>>> >>>>> > > --
>> >>>>>>>> >> >> >>>>> >>>>> > > Jason M. Swails
>> >>>>>>>> >> >> >>>>> >>>>> > > Quantum Theory Project,
>> >>>>>>>> >> >> >>>>> >>>>> > > University of Florida
>> >>>>>>>> >> >> >>>>> >>>>> > > Ph.D. Candidate
>> >>>>>>>> >> >> >>>>> >>>>> > > 352-392-4032
>> >>>>>>>> >> >> >>>>> >>>>> > > ______________________________**
>> >>>>>>>> **_________________
>> >>>>>>>> >> >> >>>>> >>>>> > > AMBER mailing list
>> >>>>>>>> >> >> >>>>> >>>>> > > AMBER.ambermd.org
>> >>>>>>>> >> >> >>>>> >>>>> > >
>> >>>>>>>> >> >> >>>>> >>>>> http://lists.ambermd.org/****
>> >>>>>>>> mailman/listinfo/amber<
>> http://lists.ambermd.org/**mailman/listinfo/amber>
>> >>>>>>>> <
>> >>>>>>>> >> >> >>>>>
>> >>>>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>>>> >
>> >>>>>>>> >> >> >>>>> >>>>> "
>> >>>>>>>> >> >> >>>>> >>>>> > > ______________________________**
>> >>>>>>>> **_________________
>> >>>>>>>> >> >> >>>>> >>>>> > > AMBER mailing list
>> >>>>>>>> >> >> >>>>> >>>>> > > AMBER.ambermd.org
>> >>>>>>>> >> >> >>>>> >>>>> > >
>> >>>>>>>> >> >> >>>>> >>>>> http://lists.ambermd.org/****
>> >>>>>>>> mailman/listinfo/amber<
>> http://lists.ambermd.org/**mailman/listinfo/amber>
>> >>>>>>>> <
>> >>>>>>>> >> >> >>>>>
>> >>>>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>>>> >
>> >>>>>>>> >> >> >>>>> >>>>> > >
>> >>>>>>>> >> >> >>>>> >>>>> > ______________________________**
>> >>>>>>>> **_________________
>> >>>>>>>> >> >> >>>>> >>>>> > AMBER mailing list
>> >>>>>>>> >> >> >>>>> >>>>> > AMBER.ambermd.org
>> >>>>>>>> >> >> >>>>> >>>>> >
>> >>>>>>>> >> >> >>>>> >>>>> http://lists.ambermd.org/****
>> >>>>>>>> mailman/listinfo/amber<
>> http://lists.ambermd.org/**mailman/listinfo/amber>
>> >>>>>>>> <
>> >>>>>>>> >> >> >>>>>
>> >>>>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>>>> >
>> >>>>>>>> >> >> >>>>> >>>>> > ______________________________**
>> >>>>>>>> **_________________
>> >>>>>>>> >> >> >>>>> >>>>> > AMBER mailing list
>> >>>>>>>> >> >> >>>>> >>>>> > AMBER.ambermd.org
>> >>>>>>>> >> >> >>>>> >>>>> >
>> >>>>>>>> >> >> >>>>> >>>>> http://lists.ambermd.org/****
>> >>>>>>>> mailman/listinfo/amber<
>> http://lists.ambermd.org/**mailman/listinfo/amber>
>> >>>>>>>> <
>> >>>>>>>> >> >> >>>>>
>> >>>>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>>>> >
>> >>>>>>>> >> >> >>>>> >>>>> >
>> >>>>>>>> >> >> >>>>> >>>>>
>> >>>>>>>> >> >> >>>>> >>>>>
>> >>>>>>>> >> >> >>>>> >>>>>
>> >>>>>>>> >> >> >>>>> >>>>> --
>> >>>>>>>> >> >> >>>>> >>>>> Jason M. Swails
>> >>>>>>>> >> >> >>>>> >>>>> Quantum Theory Project,
>> >>>>>>>> >> >> >>>>> >>>>> University of Florida
>> >>>>>>>> >> >> >>>>> >>>>> Ph.D. Candidate
>> >>>>>>>> >> >> >>>>> >>>>> 352-392-4032
>> >>>>>>>> >> >> >>>>> >>>>> ______________________________**
>> >>>>>>>> **_________________
>> >>>>>>>> >> >> >>>>> >>>>> AMBER mailing list
>> >>>>>>>> >> >> >>>>> >>>>> AMBER.ambermd.org
>> >>>>>>>> >> >> >>>>> >>>>> http://lists.ambermd.org/****
>> >>>>>>>> mailman/listinfo/amber<
>> http://lists.ambermd.org/**mailman/listinfo/amber>
>> >>>>>>>> <
>> >>>>>>>> >> >> >>>>>
>> >>>>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>>>> >
>> >>>>>>>> >> >> >>>>> >>>>>
>> >>>>>>>> >> >> >>>>> >>>>>
>> >>>>>>>> >> >> >>>>> >>>> ______________________________**
>> >>>>>>>> **_________________
>> >>>>>>>> >> >> >>>>> >>> AMBER mailing list
>> >>>>>>>> >> >> >>>>> >>> AMBER.ambermd.org
>> >>>>>>>> >> >> >>>>> >>>
>> >>>>>>>> http://lists.ambermd.org/****mailman/listinfo/amber<
>> http://lists.ambermd.org/**mailman/listinfo/amber>
>> >>>>>>>> <
>> >>>>>>>> >> >> >>>>>
>> >>>>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>>>> >
>> >>>>>>>> >> >> >>>>> >>>
>> >>>>>>>> >> >> >>>>> >>> __________ Informace od ESET NOD32 Antivirus,
>> >>>>>>>> verze
>> >>>>>>>> >> databaze
>> >>>>>>>> >> >> 8394
>> >>>>>>>> >> >> >>>>> >>> (20130530) __________
>> >>>>>>>> >> >> >>>>> >>>
>> >>>>>>>> >> >> >>>>> >>> Tuto zpravu proveril ESET NOD32 Antivirus.
>> >>>>>>>> >> >> >>>>> >>>
>> >>>>>>>> >> >> >>>>> >>> http://www.eset.cz
>> >>>>>>>> >> >> >>>>> >>>
>> >>>>>>>> >> >> >>>>> >>>
>> >>>>>>>> >> >> >>>>> >>>
>> >>>>>>>> >> >> >>>>> >>>
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >> --
>> >>>>>>>> >> >> >>>>> >> Tato zpráva byla vytvořena převratným poštovním
>> >>>>>>>> klientem
>> >>>>>>>> > Opery:
>> >>>>>>>> >> >> >>>>> >> http://www.opera.com/mail/
>> >>>>>>>> >> >> >>>>> >>
>> ______________________________**_________________
>> >>>>>>>> >> >> >>>>> >> AMBER mailing list
>> >>>>>>>> >> >> >>>>> >> AMBER.ambermd.org
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> >>
>> >>>>>>>> >> >> >>>>> > ______________________________**_________________
>> >>>>>>>> >> >> >>>>> > AMBER mailing list
>> >>>>>>>> >> >> >>>>> > AMBER.ambermd.org
>> >>>>>>>> >> >> >>>>> >
>> >>>>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>>>> >> >> >>>>> >
>> >>>>>>>> >> >> >>>>> > __________ Informace od ESET NOD32 Antivirus,
>> verze
>> >>>>>>>> databaze
>> >>>>>>>> >> >> 8394
>> >>>>>>>> >> >> >>>>> > (20130530) __________
>> >>>>>>>> >> >> >>>>> >
>> >>>>>>>> >> >> >>>>> > Tuto zpravu proveril ESET NOD32 Antivirus.
>> >>>>>>>> >> >> >>>>> >
>> >>>>>>>> >> >> >>>>> > http://www.eset.cz
>> >>>>>>>> >> >> >>>>> >
>> >>>>>>>> >> >> >>>>> >
>> >>>>>>>> >> >> >>>>> >
>> >>>>>>>> >> >> >>>>>
>> >>>>>>>> >> >> >>>>>
>> >>>>>>>> >> >> >>>>> --
>> >>>>>>>> >> >> >>>>> Tato zpráva byla vytvořena převratným poštovním
>> >>>>>>>> klientem
>> >>>>>>>> Opery:
>> >>>>>>>> >> >> >>>>> http://www.opera.com/mail/
>> >>>>>>>> >> >> >>>>>
>> >>>>>>>> >> >> >>>>> ______________________________**_________________
>> >>>>>>>> >> >> >>>>> AMBER mailing list
>> >>>>>>>> >> >> >>>>> AMBER.ambermd.org
>> >>>>>>>> >> >> >>>>>
>> >>>>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>>>> >> >> >>>>>
>> >>>>>>>> >> >> >>>>
>> >>>>>>>> >> >> >>>>
>> >>>>>>>> >> >> >>> ______________________________**_________________
>> >>>>>>>> >> >> >>> AMBER mailing list
>> >>>>>>>> >> >> >>> AMBER.ambermd.org
>> >>>>>>>> >> >> >>>
>> >>>>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>>>> >> >> >>>
>> >>>>>>>> >> >> >>> __________ Informace od ESET NOD32 Antivirus, verze
>> >>>>>>>> databaze
>> >>>>>>>> 8395
>> >>>>>>>> >> >> >>> (20130531) __________
>> >>>>>>>> >> >> >>>
>> >>>>>>>> >> >> >>> Tuto zpravu proveril ESET NOD32 Antivirus.
>> >>>>>>>> >> >> >>>
>> >>>>>>>> >> >> >>> http://www.eset.cz
>> >>>>>>>> >> >> >>>
>> >>>>>>>> >> >> >>>
>> >>>>>>>> >> >> >>>
>> >>>>>>>> >> >> >>
>> >>>>>>>> >> >> >>
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >> >
>> >>>>>>>> >> >>
>> >>>>>>>> >> >>
>> >>>>>>>> >> >> --
>> >>>>>>>> >> >> Tato zpráva byla vytvořena převratným poštovním klientem
>> >>>>>>>> Opery:
>> >>>>>>>> >> >> http://www.opera.com/mail/
>> >>>>>>>> >> >>
>> >>>>>>>> >> >> ______________________________**_________________
>> >>>>>>>> >> >> AMBER mailing list
>> >>>>>>>> >> >> AMBER.ambermd.org
>> >>>>>>>> >> >>
>> >>>>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>>>> >> >>
>> >>>>>>>> >> >
>> >>>>>>>> >> >
>> >>>>>>>> >> >
>> >>>>>>>> >> > __________ Informace od ESET NOD32 Antivirus, verze
>> databaze
>> >>>>>>>> 8397
>> >>>>>>>> >> > (20130531) __________
>> >>>>>>>> >> >
>> >>>>>>>> >> > Tuto zpravu proveril ESET NOD32 Antivirus.
>> >>>>>>>> >> >
>> >>>>>>>> >> > GB_out_plus_diff_Files.tar.gz - poskozeny archiv
>> >>>>>>>> >> > GB_out_plus_diff_Files.tar.gz > GZIP >
>> >>>>>>>> >> GB_out_plus_diff_Files.tar
>> >>>>>>>> >> > - poskozeny archiv
>> >>>>>>>> >> > GB_out_plus_diff_Files.tar.gz > GZIP >
>> >>>>>>>> >> > GB_out_plus_diff_Files.tar > TAR >
>> >>>>>>>> GB_out_plus_diff_Files.tar.gz -
>> >>>>>>>> >> > poskozeny archiv
>> >>>>>>>> >> > GB_out_plus_diff_Files.tar.gz > GZIP >
>> >>>>>>>> >> > GB_out_plus_diff_Files.tar > TAR >
>> >>>>>>>> GB_out_plus_diff_Files.tar.gz >
>> >>>>>>>> >> GZIP
>> >>>>>>>> >> > > GB_out_plus_diff_Files.tar - poskozeny archiv
>> >>>>>>>> >> > GB_out_plus_diff_Files.tar.gz > GZIP >
>> >>>>>>>> >> > GB_out_plus_diff_Files.tar > TAR >
>> >>>>>>>> GB_out_plus_diff_Files.tar.gz >
>> >>>>>>>> >> GZIP
>> >>>>>>>> >> > > GB_out_plus_diff_Files.tar > TAR >
>> >>>>>>>> GB_nucleosome-sim3.mdout-full -
>> >>>>>>>> >> > vyskytl se problem pri cteni archivu
>> >>>>>>>> >> > PME_out_plus_diff_Files.tar.gz - poskozeny archiv
>> >>>>>>>> >> > PME_out_plus_diff_Files.tar.gz > GZIP >
>> >>>>>>>> >> > PME_out_plus_diff_Files.tar - poskozeny archiv
>> >>>>>>>> >> > PME_out_plus_diff_Files.tar.gz > GZIP >
>> >>>>>>>> >> > PME_out_plus_diff_Files.tar > TAR >
>> >>>>>>>> PME_out_plus_diff_Files.tar.gz -
>> >>>>>>>> >> > poskozeny archiv
>> >>>>>>>> >> > PME_out_plus_diff_Files.tar.gz > GZIP >
>> >>>>>>>> >> > PME_out_plus_diff_Files.tar > TAR >
>> >>>>>>>> PME_out_plus_diff_Files.tar.gz >
>> >>>>>>>> >> > GZIP > PME_out_plus_diff_Files.tar - poskozeny archiv
>> >>>>>>>> >> > PME_out_plus_diff_Files.tar.gz > GZIP >
>> >>>>>>>> >> > PME_out_plus_diff_Filestar > TAR >
>> >>>>>>>> PME_out_plus_diff_Files.tar.gz
>> >>>>>>>> >
>> >>>>>>>> >> GZIP
>> >>>>>>>> >> > > PME_out_plus_diff_Files.tar > TAR >
>> >>>>>>>> >> > PME_JAC_production_NPT-sim3.**mdout-full - vyskytl se
>> problem
>> >>>>>>>> pri
>> >>>>>>>> cteni
>> >>>>>>>> >> > archivu
>> >>>>>>>> >> >
>> >>>>>>>> >> > http://www.eset.cz
>> >>>>>>>> >> >
>> >>>>>>>> >>
>> >>>>>>>> >>
>> >>>>>>>> >> --
>> >>>>>>>> >> Tato zpráva byla vytvořena převratným poštovním klientem
>> Opery:
>> >>>>>>>> >> http://www.opera.com/mail/
>> >>>>>>>> >>
>> >>>>>>>> >> ______________________________**_________________
>> >>>>>>>> >> AMBER mailing list
>> >>>>>>>> >> AMBER.ambermd.org
>> >>>>>>>> >>
>> >>>>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>>>> > ______________________________**_________________
>> >>>>>>>> > AMBER mailing list
>> >>>>>>>> > AMBER.ambermd.org
>> >>>>>>>> >
>> >>>>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>>>> >
>> >>>>>>>> > __________ Informace od ESET NOD32 Antivirus, verze databaze
>> >>>>>>>> 8398
>> >>>>>>>> > (20130531) __________
>> >>>>>>>> >
>> >>>>>>>> > Tuto zpravu proveril ESET NOD32 Antivirus.
>> >>>>>>>> >
>> >>>>>>>> > http://www.eset.cz
>> >>>>>>>> >
>> >>>>>>>> >
>> >>>>>>>> >
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> --
>> >>>>>>>> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
>> >>>>>>>> http://www.opera.com/mail/
>> >>>>>>>>
>> >>>>>>>> ______________________________**_________________
>> >>>>>>>> AMBER mailing list
>> >>>>>>>> AMBER.ambermd.org
>> >>>>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
>> http://lists.ambermd.org/mailman/listinfo/amber>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> __________ Informace od ESET NOD32 Antivirus, verze databaze 8401
>> >>>>>> (20130601) __________
>> >>>>>>
>> >>>>>> Tuto zpravu proveril ESET NOD32 Antivirus.
>> >>>>>>
>> >>>>>> http://www.eset.cz
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>> --
>> >>>> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
>> >>>> http://www.opera.com/mail/
>> >>>>
>> >>>> _______________________________________________
>> >>>> AMBER mailing list
>> >>>> AMBER.ambermd.org
>> >>>> http://lists.ambermd.org/mailman/listinfo/amber
>> >>>>
>> >>>>
>> >>>
>> >>
>> > _______________________________________________
>> > AMBER mailing list
>> > AMBER.ambermd.org
>> > http://lists.ambermd.org/mailman/listinfo/amber
>> >
>> > __________ Informace od ESET NOD32 Antivirus, verze databaze 8403
>> > (20130602) __________
>> >
>> > Tuto zpravu proveril ESET NOD32 Antivirus.
>> >
>> > http://www.eset.cz
>> >
>> >
>> >
>>
>>
>> --
>> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
>> http://www.opera.com/mail/
>>
>> _______________________________________________
>> AMBER mailing list
>> AMBER.ambermd.org
>> http://lists.ambermd.org/mailman/listinfo/amber
>>
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
--
Tato zpráva byla vytvořena převratným poštovním klientem Opery:
http://www.opera.com/mail/
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Sun Jun 02 2013 - 11:00:04 PDT
