Sorry, I missed the attachment because I'm on a cellphone and on a business
trip. Anyway, neither Titan showed fully deterministic behavior and that
is worrisome. Notice that the 680 runs were indeed so. The latter is the
expected behavior and exactly what I see with one of my Titans and my K20.
Which means we need to figure this out. For now, could you take it on
faith that changing ntpr changes the trajectory by changing the code
executed and that it doing so is not a bug? Playing around with it is just
confusing the issue right now.
What would help clarify is if someone could try these tests on K20 or
K20X. I would love for someone to demonstrate this is a coding error on
my part because I can fix that. The evidence just isn't leading me that
way right now.
Scott
On May 29, 2013 2:41 PM, "Marek Maly" <marek.maly.ujep.cz> wrote:
> Hi Scott,
>
> what do you mean by "try running for 100k steps before comparing
> energies". In all
> tests I have actually done I did exactly 100k steps before comparing
> energies
> E_tot(at step 100 000). So you mean to extend tests to 200k steps now ?
>
> M.
>
>
> Dne Wed, 29 May 2013 22:46:58 +0200 Scott Le Grand <varelse2005.gmail.com>
> napsal/-a:
>
> > Ps try running for 100k steps before comparing energies and I suspect no
> > two simulations will match.
> > On May 29, 2013 1:41 PM, "Scott Le Grand" <varelse2005.gmail.com> wrote:
> >
> >> Your Titan setup is hosed. Your results were not 100% deterministic for
> >> the same inputs.
> >>
> >> Energies + Forces use a different subroutine than just Forces hence the
> >> ntpr dependence. Hence changing ntpr effectively is changing the input.
> >>
> >> It's 100% ironclad reproducibility that matters and you demonstrated
> >> it's
> >> not happening.
> >> On May 29, 2013 1:30 PM, "Marek Maly" <marek.maly.ujep.cz> wrote:
> >>
> >>> Hi all,
> >>>
> >>> First of all thanks to Ross for his update ! although it is question
> >>> if it helps to solve all the reported Amber issues with Titan/OC Titan
> >>> GPUs .
> >>> So let's see and hope :))
> >>>
> >>> Here are my results - see the attached TXT file with tables where
> >>> the results from the tests are summarised. I did twice the same
> >>> Amber benchmark tests on each GPU (both titans, GTX 680 and GTX 580)
> >>> to see reproducibility of the results after 100K steps at ig=default
> >>> (so ig not present in mdin file).
> >>>
> >>> The first table contains ns/day estimates obtained for each molecular
> >>> system
> >>> for each TITAN GPU. Interestingly estimates obtained for the same
> >>> system
> >>> in different
> >>> round slightly differ, but maybe that's OK.
> >>>
> >>> In the second table there are values of the total energy after 100k
> >>> steps
> >>> to check
> >>> reproducibility of the results.
> >>>
> >>> Here is summarisation :
> >>>
> >>> #1 - simulation crashes on TITANs
> >>>
> >>> Interestingly there was just one simulation crash in JAC_NPT (TITAN_0,
> >>> ROUND_1) the remaining
> >>> 3 TITAN JAC_NPT simulations were finished. There were also 3 times
> >>> crashes in CELLULOSE_NVE
> >>> test but the last simulation (TITAN_1,ROUND_2) was finished without any
> >>> problem. All the remaining
> >>> simulations were always finished without any problem. So the simulation
> >>> crashes seem to be
> >>> not-reproducible/unpredictible on some moleacular systems/(mdin
> >>> setups).
> >>>
> >>> CRASH ERRORS:
> >>>
> >>> a) JAC_NPT (TITAN_0, ROUND_1)
> >>> Here 11k steps were successfully done before crash, I found this error
> >>> in mdout file:
> >>>
> >>> | ERROR: max pairlist cutoff must be less than unit cell max sphere
> >>> radius!
> >>>
> >>> b) CELLULOSE_NVE (TITAN_0, ROUND_1, ROUND_2; TITAN_1, ROUND_1 )
> >>> Here I did not find any error in mdout file. Just this error was
> >>> written
> >>> on standard output
> >>> (screen/nohup.out file):
> >>>
> >>> ------
> >>> Error: unspecified launch failure launching kernel kNLSkinTest
> >>> cudaFree GpuBuffer::Deallocate failed unspecified launch failure
> >>> grep: mdinfo.1GTX_TITAN: No such file or directory
> >>> -----
> >>>
> >>> in all three cases.
> >>>
> >>> Here on CELLULOSE_NVE case I started to play with NTPR parameter
> >>> (originally just
> >>> on TITAN-0 GPU), to see how many steps were successfully done here
> >>> before
> >>> crash, then this my
> >>> small research started to be more interesting than I ever thought :))
> >>> see
> >>> here
> >>> chronologically my results for E_tot after 2000 steps for different
> >>> GPUs
> >>> (machines) - I repeated calculation several times for the given NTPR
> >>> just
> >>> to be sure.
> >>>
> >>> TITAN-0, Etot after 2000 steps
> >>>
> >>> NTPR=10
> >>>
> >>> -443256.6867
> >>> -443256.6867
> >>> -443256.6867
> >>>
> >>> NTPR=100
> >>>
> >>> -443250.1350
> >>> -443250.1350
> >>> -443250.1350
> >>>
> >>> NTPR=200
> >>>
> >>> -443261.0705
> >>> -443261.0705
> >>> -443072.3097
> >>> -443261.0705
> >>> -443261.0705
> >>> -443261.0705
> >>> -443261.0705
> >>>
> >>> NTPR=10 (again just to verify)
> >>>
> >>> -443256.6867
> >>> -443256.6867
> >>>
> >>>
> >>> Then I tried with TITAN-1
> >>>
> >>> NTPR=10
> >>>
> >>> -443256.6867
> >>> -443256.6867
> >>>
> >>> NTPR=100
> >>>
> >>> -443250.1350
> >>> -443250.1350
> >>>
> >>> NTPR=200
> >>>
> >>> -443261.0705
> >>> -443261.0705
> >>>
> >>>
> >>> Then I tried with GTX-580
> >>>
> >>> NTPR=10
> >>>
> >>> -443256.6867
> >>> -443256.6867
> >>>
> >>> NTPR=200
> >>>
> >>> -443261.0705
> >>> -443261.0705
> >>>
> >>> then I tried with GTX-680
> >>>
> >>> NTPR=10 Etot after 2000 steps
> >>>
> >>> -443256.6711
> >>> -443256.6711
> >>>
> >>> NTPR=200 Etot after 2000 steps
> >>>
> >>> -443261.0705
> >>> -443261.0705
> >>>
> >>> Any idea why energies should depend on frequency of energy records
> >>> (NTPR)
> >>> ?
> >>>
> >>>
> >>>
> >>> #2 - reproducibility on TITANs (see attached table.txt)
> >>>
> >>> Also here are differences depending on concrete systems/setups.
> >>> While in case of FACTOR_IX_NVE, FACTOR_IX_NPT, TRPCAGE, MYOGLOBIN
> >>> systems
> >>> I have obtained
> >>> 100% reproducibility (the results for the given system were identical
> >>> for
> >>> both cards/all ROUNDs)
> >>> on systems JAC_NVE, JAC_NPT, NUCLEOSOME I obtained small differences
> >>> in
> >>> general however in case
> >>> of TITAN_1 GPU also NUCLEOSOME results were 100% reproducible.
> >>> Moreover
> >>> for the TITAN_1 card which succeeded to finish CELLULOSE test at least
> >>> in
> >>> ROUND_2 I did 3rd additional round and I got the identical result as
> >>> from
> >>> the ROUND_2 (i.e. -443246.3206 ) so regarding TITAN_1 GPU I can say
> >>> that
> >>> it is able to 100% reproduce 100k steps CELLULOSE_NVE test result
> >>> perhaps
> >>> on all eventually successfully finished runs :))
> >>>
> >>>
> >>> #3 - GTX-580, GTX-680 controls
> >>>
> >>> Here the simulations were done without any problems and were 100%
> >>> reproducible on each card however
> >>> the results for the given system slightly differ between those two
> >>> cards
> >>> with exception of the
> >>> CELLULOSE system where both cards GTX-580, GTX-680 provided identical
> >>> result which is moreover
> >>> nearly identical with result obtained with TITAN_1 during ROUND_2
> >>> (relative difference 2e-6).
> >>>
> >>>
> >>> TO ET:
> >>> a)
> >>> I had no problems with minimization stages in my own simul. bigger than
> >>> 100k which crashed
> >>> during heat NVT phase.
> >>>
> >>> b)
> >>> 313.30 driver ??? OK so after 319.23 I will try experiment with this a
> >>> bit "outdated" version :))
> >>> Actually I am working under 319.17. (and CUDA 5.0)
> >>>
> >>> c)
> >>> Can you please do at least JAC_NPT, JAC_NVE, NUCLEOSOME and
> >>> CELLULOSE_NVE
> >>> tests using 100 000 steps
> >>> (same random seed e.g. default = ig deleted from mdin if is there)
> >>> twice
> >>> to confirm 100% reproducibility on your TITAN GPU ?
> >>>
> >>> TO Divi:
> >>>
> >>> This is also my usual approach to divide whole simulation into many
> >>> subtrajectories (in my case 0.5 ns = 250k 2fs steps) but it does not
> >>> seem
> >>> to help here it self. Can you please also do the same tests which I
> >>> asked
> >>> ET (point c) )
> >>>
> >>>
> >>> BTW CUDA release candidate 5.5 was just released (
> >>> https://developer.nvidia.com/**cuda-toolkit<
> https://developer.nvidia.com/cuda-toolkit>)
> >>> would it be reasonable idea to try compile/run pmemd.cuda with this
> >>> brand
> >>> new cuda version ?
> >>>
> >>> Thanks !
> >>>
> >>> Best wishes,
> >>>
> >>> Marek
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Dne Wed, 29 May 2013 03:44:33 +0200 Ross Walker <ross.rosswalker.co.uk
> >
> >>> napsal/-a:
> >>>
> >>> Hi All,
> >>>>
> >>>> Just an update that we will have some fixes out soon that address some
> >>>> errors we have been noticing with simulations crashing during NPT
> >>>> runs.
> >>>> It
> >>>> is possible that this is confusing the issue here as to whether the
> >>>> problem is related to the GTX Titan or to a possible bug in the code.
> >>>> I
> >>>> hope to have the patch released within a few days at which point it
> >>>> would
> >>>> be good to repeat these tests and then hopefully we can try to track
> >>>> down
> >>>> what is going on. I find it hard to believe that so many cards are
> >>>> faulty
> >>>> so I suspect that there may be something funky in the code with
> >>>> regards
> >>>> to
> >>>> GTX Titans. We'll try and get it fixed as soon as possible but for now
> >>>> please just wait until we get the update released for AMBER 12 in a
> >>>> few
> >>>> days and see if that helps at all.
> >>>>
> >>>> All the best
> >>>> Ross
> >>>>
> >>>>
> >>>> On 5/28/13 5:12 PM, "Divi/GMAIL" <dvenkatlu.gmail.com> wrote:
> >>>>
> >>>> I have two TITANs in my Gigabyte workstation. I have had similar
> >>>>> issues
> >>>>> of NANs for some of the simulation setups. Never could figure out why
> >>>>> the
> >>>>> simulations failed for no reason. I tried 10, 12 ang. box sizes.
> >>>>> same
> >>>>> random breakdowns. Thought of returning them suspecting memory
> >>>>> errors.
> >>>>> But
> >>>>> some simulations ran perfectly fine. Currently running two
> >>>>> calculations
> >>>>> without any problems. Both are running pretty stable for over 100ns.
> >>>>> I
> >>>>> suspect AMBER CUDA code may have some issues under some simulation
> >>>>> conditions such as NPT. In general, NVT setup is more successful than
> >>>>> NPT,
> >>>>> in my case.
> >>>>>
> >>>>> These are 287426 atoms simulation on one card (9 ns/day)
> >>>>> On other card: 129049 atom setup (20 ns/day)
> >>>>>
> >>>>> Both using same NVT setup. (AMBER12/INTEL-12.x
> >>>>> compilers/CentOS-6.3/Drivers 319.17/CUDA5.0)
> >>>>>
> >>>>> Input is below:
> >>>>> &cntrl
> >>>>> nstlim=500000, dt=0.002,
> >>>>> ntx=5, irest=1, ig=-1,
> >>>>> ntpr=1000, ntwr=10000, ntwx=10000,
> >>>>> ntt=1, tautp=2, ntb=1, ntp=0, ntc=2, ntf=2,
> >>>>> iwrap=1, ioutfm=1, ntxo=2,
> >>>>> &end
> >>>>>
> >>>>> One suggestion If I may add: If you could run short simulations for
> >>>>> no
> >>>>> more
> >>>>> than 500,000 steps (or 1ns with 2 fs), you might find some stability.
> >>>>> Again,
> >>>>> not scientific rationale from my side. But it worked in some cases
> >>>>> for
> >>>>> me.
> >>>>>
> >>>>> This is self-assembled system with GIGABYTE GA-Z77X-UP7 (with core
> >>>>> i5
> >>>>> processor) and 1200W PS/16GB memory.
> >>>>>
> >>>>>
> >>>>> Best regards
> >>>>> Divi
> >>>>>
> >>>>>
> >>>>>
> >>>>> -----Original Message-----
> >>>>> From: Scott Le Grand
> >>>>> Sent: Tuesday, May 28, 2013 4:46 PM
> >>>>> To: AMBER Mailing List
> >>>>> Subject: Re: [AMBER] experiences with EVGA GTX TITAN Superclocked -
> >>>>> memtestG80 - UNDERclocking in Linux ?
> >>>>>
> >>>>> You can play Russian Roulette a whole bunch of rounds without blowing
> >>>>> your
> >>>>> head off.
> >>>>>
> >>>>> Similarly, when you have a GPU that occasionally flips a bit the
> >>>>> wrong
> >>>>> way,
> >>>>> most of the time it will be some low order perturbation to the
> >>>>> coordinates
> >>>>> that does little more than make the trajectory nondeterministic...
> >>>>> Except
> >>>>> when it doesn't...
> >>>>>
> >>>>> You can't even detect this kind of misbehavior in GROMACS, ACEMD, or
> >>>>> NAMD
> >>>>> because *none* of them (to my knowledge) are capable of producing
> >>>>> deterministic output at production-level performance.
> >>>>>
> >>>>> Titans and 680s are consumer cards. I love them to death, but if
> >>>>> you're
> >>>>> going to do production work with them, you need to qual them
> >>>>> thoroughly
> >>>>> before proceeding or you need to pay up and use Teslas instead. I'd
> >>>>> still
> >>>>> build a cluster with Titans myself, but I'd ruthlessly RMA them
> >>>>> until I
> >>>>> got
> >>>>> satisfaction if they couldn't pass a test consisting of running an
> >>>>> AMBER
> >>>>> simulation for 100K iterations without either crashing or producing a
> >>>>> nondeterministic result. The customer is always right.
> >>>>>
> >>>>>
> >>>>> On Tue, May 28, 2013 at 1:20 PM, Marek Maly <marek.maly.ujep.cz>
> >>>>> wrote:
> >>>>>
> >>>>> I would wait for the results of my GOPU0, GPU1 double tests before
> >>>>>> any serious conclusions.
> >>>>>>
> >>>>>> BTW what exactly means "GPU is hosed" ? Something like GPU is
> >>>>>> damaged
> >>>>>> or
> >>>>>> so ?
> >>>>>>
> >>>>>> Also would be strange (not probable) to buy 2 somehow damaged GPUs
> >>>>>> (even
> >>>>>> in the same way).
> >>>>>>
> >>>>>> As I wrote, memtestG80 tests were negative on both cards, if
> >>>>>> moreover
> >>>>>> both cards will perfectly reproduce both repetitions of the Amber
> >>>>>> benchmarks
> >>>>>> and eventually pass some another GPU tests (can you recommend any
> >>>>>> except
> >>>>>> memtestG80 ?)
> >>>>>> I still believe that the GPU cards are OK (also thank to particular
> >>>>>> successes in my Amb. simulations and actual A. benchmarks). So
> >>>>>> maybe I
> >>>>>> will eventually try downclock, but there might be some another
> >>>>>> variables,
> >>>>>> e.g. driver, OS, motherboard (I will perhaps test one card in
> >>>>>> another
> >>>>>> MB
> >>>>>> just to be sure, that problem is not MB based) etc. that's why I
> >>>>>> asked
> >>>>>> before that guy "ET" for the info about driver version, would be
> >>>>>> also
> >>>>>> interesting OS info or MB.
> >>>>>>
> >>>>>> M.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Dne Tue, 28 May 2013 22:13:36 +0200 Scott Le Grand
> >>>>>> <varelse2005.gmail.com>
> >>>>>> napsal/-a:
> >>>>>>
> >>>>>> > Marek,
> >>>>>> > Your GPU is hosed. I don't have anything else to add. I'm not
> >>>>>> going
> >>>>>> to
> >>>>>> > go
> >>>>>> > snark hunting for a bug that doesn't exist.
> >>>>>> >
> >>>>>> >
> >>>>>> >
> >>>>>> > On Tue, May 28, 2013 at 12:24 PM, Marek Maly <marek.maly.ujep.cz>
> >>>>>> wrote:
> >>>>>> >
> >>>>>> >> Hi, just for the curiosity which driver are you using
> >>>>>> >> on that machine with perfectly working with OC TITAN,
> >>>>>> >> 319.17 or some more actual e.g. 319.23 ?
> >>>>>> >>
> >>>>>> >> RMA is a good idea but it could be also long time story and
> >>>>>> >> also to succeed here you need to have strong arguments
> >>>>>> >> especially if you are going to RMA two OC TITANs.
> >>>>>> >>
> >>>>>> >> I am not sure if my arguments "The cards have problems with some
> >>>>>> Amber
> >>>>>> >> calculations"
> >>>>>> >> would be strong enough here. Would be much better to have clear
> >>>>>> results
> >>>>>> >> from
> >>>>>> >> respected GPU tests and as it seems you may do extensive GPU
> >>>>>> tests
> >>>>>> also
> >>>>>> >> with
> >>>>>> >> multiple routines without any errors but still have problems with
> >>>>>> >> particular
> >>>>>> >> Amber simulations...
> >>>>>> >>
> >>>>>> >> BTW I am now doing Amber benchmarks with nstlim=100K and
> >>>>>> ig=default
> >>>>>> for
> >>>>>> >> each card
> >>>>>> >> twice. The tests will be done in cca 3 hours (due to slow
> >>>>>> nucleosome
> >>>>>> GB
> >>>>>> >> test).
> >>>>>> >>
> >>>>>> >> But even now I have interesting results from the first test on
> >>>>>> GPU0
> >>>>>> >> (nucleosome is still running) see below.
> >>>>>> >>
> >>>>>> >> As you can see JAC_NPT crashed around 11000 step, here is the
> >>>>>> last
> >>>>>> >> md.out
> >>>>>> >> record:
> >>>>>> >>
> >>>>>> >> *********
> >>>>>> >>
> >>>>>> >>
> >>>>>>
> >>>>>> ------------------------------**------------------------------**
> >>>>>> -------------
> >>>>>> -----
> >>>>>> >>
> >>>>>> >> check COM velocity, temp: 0.000021 0.00(Removed)
> >>>>>> >>
> >>>>>> >> NSTEP = 11000 TIME(PS) = 28.000 TEMP(K) = 300.39
> >>>>>> PRESS
> >>>>>> >> =
> >>>>>> >> -9.4
> >>>>>> >> Etot = -58092.8958 EKtot = 14440.2520 EPtot =
> >>>>>> >> -72533.1478
> >>>>>> >> BOND = 443.3912 ANGLE = 1253.5177 DIHED =
> >>>>>> >> 970.1275
> >>>>>> >> 1-4 NB = 567.2497 1-4 EEL = 6586.9007 VDWAALS =
> >>>>>> >> 8664.9960
> >>>>>> >> EELEC = -91019.3306 EHBOND = 0.0000 RESTRAINT =
> >>>>>> >> 0.0000
> >>>>>> >> EKCMT = 6274.0354 VIRIAL = 6321.9969 VOLUME =
> >>>>>> >> 236141.9494
> >>>>>> >> Density =
> >>>>>> >> 1.0162
> >>>>>> >>
> >>>>>> >>
> >>>>>>
> >>>>>> ------------------------------**------------------------------**
> >>>>>> -------------
> >>>>>> -----
> >>>>>> >>
> >>>>>> >> | ERROR: max pairlist cutoff must be less than unit cell max
> >>>>>> sphere
> >>>>>> >> radius!
> >>>>>> >>
> >>>>>> >> ********
> >>>>>> >>
> >>>>>> >> Any idea about that ERROR ?
> >>>>>> >>
> >>>>>> >> On the other hand FACTOR_IX_NPT which has much more atoms passed
> >>>>>> >> without
> >>>>>> >> any issue.
> >>>>>> >>
> >>>>>> >> Cellulose crashed on the beginning without any ERROR message in
> >>>>>> md.out
> >>>>>> >> file.
> >>>>>> >>
> >>>>>> >>
> >>>>>> >> I am very curious regarding exact reproducibility of the results
> >>>>>> at
> >>>>>> >> least
> >>>>>> >> in the
> >>>>>> >> framework of both tests on individual cards.
> >>>>>> >>
> >>>>>> >> BTW regarding eventual downclocking, has anyone idea about some
> >>>>>> NVclock
> >>>>>> >> alternative or
> >>>>>> >> I will be really eventually forced to edit frequency value in GPU
> >>>>>> BIOS
> >>>>>> >> ?
> >>>>>> >>
> >>>>>> >> Best,
> >>>>>> >>
> >>>>>> >> Marek
> >>>>>> >>
> >>>>>> >> HERE ARE THE FIRST DATA FROM MY 2x2 Bench tests
> >>>>>> >>
> >>>>>> >> JAC_PRODUCTION_NVE - 23,558 atoms PME
> >>>>>> >> ------------------------------**-------
> >>>>>> >>
> >>>>>> >> 1 x GTX_TITAN: | ns/day = 115.91
> >>>>>> seconds/ns =
> >>>>>> >> 745.39
> >>>>>> >>
> >>>>>> >> JAC_PRODUCTION_NPT - 23,558 atoms PME
> >>>>>> >> ------------------------------**-------
> >>>>>> >>
> >>>>>> >> 1 x GTX_TITAN: STOP PMEMD Terminated Abnormally!
> >>>>>> >> | ns/day = 90.72 seconds/ns = 952.42
> >>>>>> >>
> >>>>>> >> FACTOR_IX_PRODUCTION_NVE - 90,906 atoms PME
> >>>>>> >> ------------------------------**-------------
> >>>>>> >>
> >>>>>> >> 1 x GTX_TITAN: | ns/day = 30.56
> >>>>>> seconds/ns =
> >>>>>> >> 2827.33
> >>>>>> >>
> >>>>>> >> FACTOR_IX_PRODUCTION_NPT - 90,906 atoms PME
> >>>>>> >> ------------------------------**-------------
> >>>>>> >>
> >>>>>> >> 1 x GTX_TITAN: | ns/day = 25.01
> >>>>>> seconds/ns =
> >>>>>> >> 3454.56
> >>>>>> >>
> >>>>>> >> CELLULOSE_PRODUCTION_NVE - 408,609 atoms PME
> >>>>>> >> ------------------------------**--------------
> >>>>>> >>
> >>>>>> >> 1 x GTX_TITAN: Error: unspecified launch failure
> >>>>>> launching
> >>>>>> >> kernel
> >>>>>> >> kNLSkinTest
> >>>>>> >> cudaFree GpuBuffer::Deallocate failed unspecified launch failure
> >>>>>> >> grep: mdinfo.1GTX_TITAN: No such file or directory
> >>>>>> >>
> >>>>>> >> TRPCAGE_PRODUCTION - 304 atoms GB
> >>>>>> >> ------------------------------**---
> >>>>>> >> 1 x GTX_TITAN: | ns/day = 595.09
> >>>>>> seconds/ns =
> >>>>>> >> 145.19
> >>>>>> >>
> >>>>>> >> MYOGLOBIN_PRODUCTION - 2,492 atoms GB
> >>>>>> >> ------------------------------**-------
> >>>>>> >>
> >>>>>> >> 1 x GTX_TITAN: | ns/day = 202.56
> >>>>>> seconds/ns =
> >>>>>> >> 426.53
> >>>>>> >>
> >>>>>> >> NUCLEOSOME_PRODUCTION - 25,095 atoms GB
> >>>>>> >> ------------------------------**---------
> >>>>>> >>
> >>>>>> >> 1 x GTX_TITAN:
> >>>>>> >>
> >>>>>> >>
> >>>>>> >>
> >>>>>> >>
> >>>>>> >>
> >>>>>> >>
> >>>>>> >>
> >>>>>> >> Dne Tue, 28 May 2013 20:42:32 +0200 ET <sketchfoot.gmail.com>
> >>>>>> napsal/-a:
> >>>>>> >>
> >>>>>> >> > Hi,
> >>>>>> >> >
> >>>>>> >> > I just got a superclocked Titan and one at normal freq. The
> >>>>>> first
> >>>>>> one
> >>>>>> >> ran
> >>>>>> >> > like a charm with no issues so far. The other standard clocked
> >>>>>> one
> >>>>>> >> could
> >>>>>> >> > never get past the constant pressure stage in an NPT
> >>>>>> simulation.
> >>>>>> It
> >>>>>> >> kept
> >>>>>> >> > writing NAN or ********* in the outfile. I swapped them about
> >>>>>> in
> >>>>>> the
> >>>>>> >> pcie
> >>>>>> >> > lanes then ran it solo in each one of the lanes. Despite all
> >>>>>> this
> >>>>>> it
> >>>>>> >> was
> >>>>>> >> > still failing the benchmark that the other one had no problems
> >>>>>> with.
> >>>>>> >> >
> >>>>>> >> > I couldn't find any memory errors with GPU-burn either, but as
> >>>>>> they
> >>>>>> >> cost
> >>>>>> >> > near a grand a piece, I RMA'd it today. I recommend you to do
> >>>>>> the
> >>>>>> >> same if
> >>>>>> >> > its not giving you any joy. Life's too short. :)
> >>>>>> >> >
> >>>>>> >> > br,
> >>>>>> >> > g
> >>>>>> >> >
> >>>>>> >> >
> >>>>>> >> > On 28 May 2013 16:57, Scott Le Grand <varelse2005.gmail.com>
> >>>>>> wrote:
> >>>>>> >> >
> >>>>>> >> >> AMBER != NAMD...
> >>>>>> >> >>
> >>>>>> >> >> GTX 680 != GTX Titan...
> >>>>>> >> >>
> >>>>>> >> >> Ian's suggestion is a good one. But even then, you need to
> >>>>>> test
> >>>>>> >> >> your
> >>>>>> >> >> GPUs
> >>>>>> >> >> as the Titans are running right on the edge of stability.
> >>>>>> Like I
> >>>>>> >> told
> >>>>>> >> >> Marek, try running 100K iterations of Cellulose NVE twice with
> >>>>>> the
> >>>>>> >> same
> >>>>>> >> >> random seed. if you don't get identically bit accurate
> >>>>>> output,
> >>>>>> your
> >>>>>> >> >> GPU is
> >>>>>> >> >> not working. Memtest programs do not catch this because (I am
> >>>>>> >> guessing)
> >>>>>> >> >> they are designed for a uniform memory hierarchy and only one
> >>>>>> path
> >>>>>> >> >> to
> >>>>>> >> >> read
> >>>>>> >> >> and write data. I have a stock GTX Titan that cannot pass the
> >>>>>> >> Cellulose
> >>>>>> >> >> NVE test and another one that does. I spent a couple days on
> >>>>>> the
> >>>>>> >> former
> >>>>>> >> >> GPU looking for the imaginary bug that went away like magic
> >>>>>> the
> >>>>>> >> second I
> >>>>>> >> >> switched out the GPU.
> >>>>>> >> >>
> >>>>>> >> >> Scott
> >>>>>> >> >>
> >>>>>> >> >>
> >>>>>> >> >>
> >>>>>> >> >>
> >>>>>> >> >>
> >>>>>> >> >> On Tue, May 28, 2013 at 8:11 AM, Robert Konecny <rok.ucsd.edu
> >
> >>>>>> wrote:
> >>>>>> >> >>
> >>>>>> >> >> > Hi Scott,
> >>>>>> >> >> >
> >>>>>> >> >> > unfortunately we are seeing similar Amber instability on GTX
> >>>>>> >> Titans as
> >>>>>> >> >> > Marek is. We have a box with four GTX Titans (not
> >>>>>> oveclocked)
> >>>>>> >> running
> >>>>>> >> >> > CentOS 6.3 with NVidia 319.17 driver and Amber 12.2. Any
> >>>>>> Amber
> >>>>>> >> >> simulation
> >>>>>> >> >> > longer than 10-15 min eventually crashes on these cards,
> >>>>>> including
> >>>>>> >> >> both
> >>>>>> >> >> JAC
> >>>>>> >> >> > benchmarks (with extended run time). This is reproducible on
> >>>>>> all
> >>>>>> >> four
> >>>>>> >> >> > cards.
> >>>>>> >> >> >
> >>>>>> >> >> > To eliminate the possible hardware error we ran extended GPU
> >>>>>> >> >> > memory
> >>>>>> >> >> tests
> >>>>>> >> >> > on all four Titans with memtestG80, cuda_memtest and also
> >>>>>> gpu_burn
> >>>>>> >> -
> >>>>>> >> >> all
> >>>>>> >> >> > finished without errors. Since I agree that these programs
> >>>>>> may
> >>>>>> not
> >>>>>> >> >> test
> >>>>>> >> >> the
> >>>>>> >> >> > GPU completely we also set up simulations with NAMD. We can
> >>>>>> run
> >>>>>> >> four
> >>>>>> >> >> NAMD
> >>>>>> >> >> > simulations simultaneously for many days without any errors
> >>>>>> on
> >>>>>> >> >> > this
> >>>>>> >> >> > hardware. For reference - we also have exactly the same
> >>>>>> server
> >>>>>> >> >> > with
> >>>>>> >> >> the
> >>>>>> >> >> > same hardware components but with four GTX680s and this
> >>>>>> setup
> >>>>>> >> >> > works
> >>>>>> >> >> just
> >>>>>> >> >> > fine for Amber. So all this leads me to believe that a
> >>>>>> hardware
> >>>>>> >> error
> >>>>>> >> >> is
> >>>>>> >> >> > not very likely.
> >>>>>> >> >> >
> >>>>>> >> >> > I would appreciate your comments on this, perhaps there is
> >>>>>> >> something
> >>>>>> >> >> else
> >>>>>> >> >> > causing these errors which we are not seeing.
> >>>>>> >> >> >
> >>>>>> >> >> > Thanks,
> >>>>>> >> >> >
> >>>>>> >> >> > Robert
> >>>>>> >> >> >
> >>>>>> >> >> >
> >>>>>> >> >> > On Mon, May 27, 2013 at 04:25:24PM -0700, Scott Le Grand
> >>>>>> wrote:
> >>>>>> >> >> > > I have two GTX Titans. One is defective, the other is
> >>>>>> not.
> >>>>>> >> >> > Unfortunately,
> >>>>>> >> >> > > they both pass all standard GPU memory tests.
> >>>>>> >> >> > >
> >>>>>> >> >> > > What the defective one doesn't do is generate reproducibly
> >>>>>> >> >> bit-accurate
> >>>>>> >> >> > > outputs for simulations of Factor IX (90,986 atoms) or
> >>>>>> larger,
> >>>>>> >> >> > > of
> >>>>>> >> >> 100K
> >>>>>> >> >> or
> >>>>>> >> >> > > so iterations.
> >>>>>> >> >> > >
> >>>>>> >> >> > > Which is yet another reason why I insist on MD algorithms
> >>>>>> >> >> (especially
> >>>>>> >> >> on
> >>>>>> >> >> > > GPUS) being deterministic. Besides its ability to find
> >>>>>> software
> >>>>>> >> >> bugs,
> >>>>>> >> >> > and
> >>>>>> >> >> > > fulfilling one of the most important tenets of science,
> >>>>>> it's
> >>>>>> a
> >>>>>> >> great
> >>>>>> >> >> way
> >>>>>> >> >> > to
> >>>>>> >> >> > > diagnose defective hardware with very little effort.
> >>>>>> >> >> > >
> >>>>>> >> >> > > 928 MHz? That's 6% above the boost clock of a stock
> >>>>>> Titan.
> >>>>>> >> Titan
> >>>>>> >> >> is
> >>>>>> >> >> > > pushing the performance envelope as is. If you're going
> >>>>>> to
> >>>>>> pay
> >>>>>> >> the
> >>>>>> >> >> > premium
> >>>>>> >> >> > > for such chips, I'd send them back until you get one that
> >>>>>> runs
> >>>>>> >> >> correctly.
> >>>>>> >> >> > > I'm very curious how fast you can push one of these things
> >>>>>> >> >> > > before
> >>>>>> >> >> they
> >>>>>> >> >> > give
> >>>>>> >> >> > > out.
> >>>>>> >> >> > >
> >>>>>> >> >> > >
> >>>>>> >> >> > >
> >>>>>> >> >> > >
> >>>>>> >> >> > >
> >>>>>> >> >> > >
> >>>>>> >> >> > >
> >>>>>> >> >> > > On Mon, May 27, 2013 at 10:01 AM, Marek Maly
> >>>>>> <marek.maly.ujep.cz
> >>>>>> >
> >>>>>> >> >> wrote:
> >>>>>> >> >> > >
> >>>>>> >> >> > > > Dear all,
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > I have recently bought two "EVGA GTX TITAN Superclocked"
> >>>>>> GPUs.
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > I did the first calculations (pmemd.cuda in Amber12)
> >>>>>> with
> >>>>>> >> systems
> >>>>>> >> >> > around
> >>>>>> >> >> > > > 60K atoms without any problems (NPT, Langevin), but
> >>>>>> when I
> >>>>>> >> later
> >>>>>> >> >> tried
> >>>>>> >> >> > > > with bigger systems (around 100K atoms) I obtained
> >>>>>> "classical"
> >>>>>> >> >> > irritating
> >>>>>> >> >> > > > errors
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > cudaMemcpy GpuBuffer::Download failed unspecified launch
> >>>>>> >> failure
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > just after few thousands of MD steps.
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > So this was obviously the reason for memtestG80 tests.
> >>>>>> >> >> > > > ( https://simtk.org/home/memtest ).
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > So I compiled memtestG80 from sources (
> >>>>>> >> memtestG80-1.1-src.tar.gz
> >>>>>> >> >> )
> >>>>>> >> >> and
> >>>>>> >> >> > > > then tested
> >>>>>> >> >> > > > just small part of memory GPU (200 MB) using 100
> >>>>>> iterations.
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > On both cards I have obtained huge amount of errors but
> >>>>>> "just"
> >>>>>> >> on
> >>>>>> >> >> > > > "Random blocks:". 0 errors in all remaining tests in all
> >>>>>> >> >> iterations.
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > ------THE LAST ITERATION AND FINAL RESULTS-------
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > Test iteration 100 (GPU 0, 200 MiB): 169736847 errors so
> >>>>>> far
> >>>>>> >> >> > > > Moving Inversions (ones and zeros): 0 errors (6
> >>>>>> ms)
> >>>>>> >> >> > > > Memtest86 Walking 8-bit: 0 errors (53 ms)
> >>>>>> >> >> > > > True Walking zeros (8-bit): 0 errors (26 ms)
> >>>>>> >> >> > > > True Walking ones (8-bit): 0 errors (26 ms)
> >>>>>> >> >> > > > Moving Inversions (random): 0 errors (6 ms)
> >>>>>> >> >> > > > Memtest86 Walking zeros (32-bit): 0 errors (105
> >>>>>> ms)
> >>>>>> >> >> > > > Memtest86 Walking ones (32-bit): 0 errors (104
> >>>>>> ms)
> >>>>>> >> >> > > > Random blocks: 1369863 errors (27 ms)
> >>>>>> >> >> > > > Memtest86 Modulo-20: 0 errors (215 ms)
> >>>>>> >> >> > > > Logic (one iteration): 0 errors (4 ms)
> >>>>>> >> >> > > > Logic (4 iterations): 0 errors (8 ms)
> >>>>>> >> >> > > > Logic (shared memory, one iteration): 0 errors
> >>>>>> (8
> >>>>>> ms)
> >>>>>> >> >> > > > Logic (shared-memory, 4 iterations): 0 errors
> >>>>>> (25
> >>>>>> ms)
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > Final error count after 100 iterations over 200 MiB of
> >>>>>> GPU
> >>>>>> >> memory:
> >>>>>> >> >> > > > 171106710 errors
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > ------------------------------**------------
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > I have some questions and would be really grateful for
> >>>>>> any
> >>>>>> >> >> comments.
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > Regarding overclocking, using the deviceQuery I found
> >>>>>> out
> >>>>>> that
> >>>>>> >> >> under
> >>>>>> >> >> > linux
> >>>>>> >> >> > > > both cards run
> >>>>>> >> >> > > > automatically using boost shader/GPU frequency which is
> >>>>>> here
> >>>>>> >> 928
> >>>>>> >> >> MHz
> >>>>>> >> >> > (the
> >>>>>> >> >> > > > basic value for these factory OC cards is 876 MHz).
> >>>>>> >> >> > > > deviceQuery
> >>>>>> >> >> > reported
> >>>>>> >> >> > > > Memory Clock rate is 3004 MHz although "it" should be
> >>>>>> 6008
> >>>>>> MHz
> >>>>>> >> but
> >>>>>> >> >> > maybe
> >>>>>> >> >> > > > the quantity which is reported by deviceQuery "Memory
> >>>>>> Clock
> >>>>>> >> rate"
> >>>>>> >> >> is
> >>>>>> >> >> > > > different from the product specification "Memory Clock"
> >>>>>> .
> >>>>>> It
> >>>>>> >> seems
> >>>>>> >> >> that
> >>>>>> >> >> > > > "Memory Clock rate" = "Memory Clock"/2. Am I right ? Or
> >>>>>> just
> >>>>>> >> >> > deviceQuery
> >>>>>> >> >> > > > is not able to read this spec. properly
> >>>>>> >> >> > > > in Titan GPU ?
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > Anyway for the moment I assume that the problem might be
> >>>>>> due
> >>>>>> >> >> > > > to
> >>>>>> >> >> the
> >>>>>> >> >> > high
> >>>>>> >> >> > > > shader/GPU frequency.
> >>>>>> >> >> > > > (see here :
> >>>>>> http://folding.stanford.edu/**English/DownloadUtils<
> http://folding.stanford.edu/English/DownloadUtils>
> >>>>>> )
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > To verify this hypothesis one should perhaps UNDERclock
> >>>>>> to
> >>>>>> >> basic
> >>>>>> >> >> > frequency
> >>>>>> >> >> > > > which is in this
> >>>>>> >> >> > > > model 876 MHz or even to the TITAN REFERENCE frequency
> >>>>>> which
> >>>>>> >> >> > > > is
> >>>>>> >> >> 837
> >>>>>> >> >> > MHz.
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > Obviously I am working with these cards under linux
> >>>>>> (CentOS
> >>>>>> >> >> > > > 2.6.32-358.6.1.el6.x86_64) and as I found, the OC tools
> >>>>>> under
> >>>>>> >> >> linux
> >>>>>> >> >> > are in
> >>>>>> >> >> > > > fact limited just to NVclock utility, which is
> >>>>>> unfortunately
> >>>>>> >> >> > > > out of date (at least speaking about the GTX Titan ). I
> >>>>>> have
> >>>>>> >> >> obtained
> >>>>>> >> >> > this
> >>>>>> >> >> > > > message when I wanted
> >>>>>> >> >> > > > just to let NVclock utility to read and print shader and
> >>>>>> >> >> > > > memory
> >>>>>> >> >> > > > frequencies of my Titan's:
> >>>>>> >> >> > > >
> >>>>>> >> >> > > >
> >>>>>> >> >>
> >>>>>> ------------------------------**------------------------------**
> >>>>>> -------
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > [root.dyn-138-272 NVCLOCK]# nvclock -s --speeds
> >>>>>> >> >> > > > Card: Unknown Nvidia card
> >>>>>> >> >> > > > Card number: 1
> >>>>>> >> >> > > > Memory clock: -2147483.750 MHz
> >>>>>> >> >> > > > GPU clock: -2147483.750 MHz
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > Card: Unknown Nvidia card
> >>>>>> >> >> > > > Card number: 2
> >>>>>> >> >> > > > Memory clock: -2147483.750 MHz
> >>>>>> >> >> > > > GPU clock: -2147483.750 MHz
> >>>>>> >> >> > > >
> >>>>>> >> >> > > >
> >>>>>> >> >> > > >
> >>>>>> >> >>
> >>>>>> ------------------------------**------------------------------**
> >>>>>> -------
> >>>>>> >> >> > > >
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > I would be really grateful for some tips regarding
> >>>>>> "NVclock
> >>>>>> >> >> > alternatives",
> >>>>>> >> >> > > > but after wasting some hours with googling it seems that
> >>>>>> there
> >>>>>> >> is
> >>>>>> >> >> no
> >>>>>> >> >> > other
> >>>>>> >> >> > > > Linux
> >>>>>> >> >> > > > tool with NVclock functionality. So the only
> >>>>>> possibility is
> >>>>>> >> here
> >>>>>> >> >> > perhaps
> >>>>>> >> >> > > > to edit
> >>>>>> >> >> > > > GPU bios with some Lin/DOS/Win tools like (Kepler BIOS
> >>>>>> >> >> > > > Tweaker,
> >>>>>> >> >> > NVflash)
> >>>>>> >> >> > > > but obviously
> >>>>>> >> >> > > > I would like to rather avoid such approach as using it
> >>>>>> means
> >>>>>> >> >> perhaps
> >>>>>> >> >> > also
> >>>>>> >> >> > > > to void the warranty even if I am going to underclock
> >>>>>> the
> >>>>>> GPUs
> >>>>>> >> >> not to
> >>>>>> >> >> > > > overclock them.
> >>>>>> >> >> > > > So before this eventual step (GPU bios editing) I would
> >>>>>> like
> >>>>>> >> >> > > > to
> >>>>>> >> >> have
> >>>>>> >> >> > some
> >>>>>> >> >> > > > approximative estimate
> >>>>>> >> >> > > > of the probability, that the problems are here really
> >>>>>> because
> >>>>>> >> of
> >>>>>> >> >> the
> >>>>>> >> >> > > > overclocking
> >>>>>> >> >> > > > (too high (boost) default shader frequency).
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > This probability I hope to estimate from the eventual
> >>>>>> >> responses of
> >>>>>> >> >> > another
> >>>>>> >> >> > > > Amber/Titan SC users, if I am not the only crazy guy who
> >>>>>> >> >> > > > bought
> >>>>>> >> >> this
> >>>>>> >> >> > model
> >>>>>> >> >> > > > for Amber calculations :)) But of course any eventual
> >>>>>> >> experiences
> >>>>>> >> >> with
> >>>>>> >> >> > > > Titan cards related to their memtestG80 results and
> >>>>>> >> >> UNDER/OVERclocking
> >>>>>> >> >> > > > (if possible in Linux OS) are of course welcomed as
> >>>>>> well !
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > My HW/SW configuration
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > motherboard: ASUS P9X79 PRO
> >>>>>> >> >> > > > CPU: Intel Core i7-3930K
> >>>>>> >> >> > > > RAM: CRUCIAL Ballistix Sport 32GB (4x8GB) DDR3 1600 VLP
> >>>>>> >> >> > > > CASE: CoolerMaster Dominator CM-690 II Advanced,
> >>>>>> >> >> > > > Power:Enermax PLATIMAX EPM1200EWT 1200W, 80+, Platinum
> >>>>>> >> >> > > > GPUs : 2 x EVGA GTX TITAN Superclocked 6GB
> >>>>>> >> >> > > > cooler: Cooler Master Hyper 412 SLIM
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > OS: CentOS (2.6.32-358.6.1.el6.x86_64)
> >>>>>> >> >> > > > driver version: 319.17
> >>>>>> >> >> > > > cudatoolkit_5.0.35_linux_64_**rhel6.x
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > The computer is in air-conditioned room with permanent
> >>>>>> >> >> > > > external
> >>>>>> >> >> > > > temperature around 18°C
> >>>>>> >> >> > > >
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > Thanks a lot in advance for any comment/experience !
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > Best wishes,
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > Marek
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > --
> >>>>>> >> >> > > > Tato zpráva byla vytvořena převratným poštovním klientem
> >>>>>> >> >> > > > Opery:
> >>>>>> >> >> > > > http://www.opera.com/mail/
> >>>>>> >> >> > > >
> >>>>>> >> >> > > > ______________________________**_________________
> >>>>>> >> >> > > > AMBER mailing list
> >>>>>> >> >> > > > AMBER.ambermd.org
> >>>>>> >> >> > > >
> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
> http://lists.ambermd.org/mailman/listinfo/amber>
> >>>>>> >> >> > > >
> >>>>>> >> >> > > ______________________________**_________________
> >>>>>> >> >> > > AMBER mailing list
> >>>>>> >> >> > > AMBER.ambermd.org
> >>>>>> >> >> > >
> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
> http://lists.ambermd.org/mailman/listinfo/amber>
> >>>>>> >> >> >
> >>>>>> >> >> > ______________________________**_________________
> >>>>>> >> >> > AMBER mailing list
> >>>>>> >> >> > AMBER.ambermd.org
> >>>>>> >> >> >
> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
> http://lists.ambermd.org/mailman/listinfo/amber>
> >>>>>> >> >> >
> >>>>>> >> >> ______________________________**_________________
> >>>>>> >> >> AMBER mailing list
> >>>>>> >> >> AMBER.ambermd.org
> >>>>>> >> >>
> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
> http://lists.ambermd.org/mailman/listinfo/amber>
> >>>>>> >> >>
> >>>>>> >> > ______________________________**_________________
> >>>>>> >> > AMBER mailing list
> >>>>>> >> > AMBER.ambermd.org
> >>>>>> >> >
> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
> http://lists.ambermd.org/mailman/listinfo/amber>
> >>>>>> >> >
> >>>>>> >> > __________ Informace od ESET NOD32 Antivirus, verze databaze
> >>>>>> 8385
> >>>>>> >> > (20130528) __________
> >>>>>> >> >
> >>>>>> >> > Tuto zpravu proveril ESET NOD32 Antivirus.
> >>>>>> >> >
> >>>>>> >> > http://www.eset.cz
> >>>>>> >> >
> >>>>>> >> >
> >>>>>> >> >
> >>>>>> >>
> >>>>>> >>
> >>>>>> >> --
> >>>>>> >> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
> >>>>>> >> http://www.opera.com/mail/
> >>>>>> >>
> >>>>>> >> ______________________________**_________________
> >>>>>> >> AMBER mailing list
> >>>>>> >> AMBER.ambermd.org
> >>>>>> >>
> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
> http://lists.ambermd.org/mailman/listinfo/amber>
> >>>>>> >>
> >>>>>> > ______________________________**_________________
> >>>>>> > AMBER mailing list
> >>>>>> > AMBER.ambermd.org
> >>>>>> >
> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
> http://lists.ambermd.org/mailman/listinfo/amber>
> >>>>>> >
> >>>>>> > __________ Informace od ESET NOD32 Antivirus, verze databaze 8386
> >>>>>> > (20130528) __________
> >>>>>> >
> >>>>>> > Tuto zpravu proveril ESET NOD32 Antivirus.
> >>>>>> >
> >>>>>> > http://www.eset.cz
> >>>>>> >
> >>>>>> >
> >>>>>> >
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
> >>>>>> http://www.opera.com/mail/
> >>>>>>
> >>>>>> ______________________________**_________________
> >>>>>> AMBER mailing list
> >>>>>> AMBER.ambermd.org
> >>>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
> http://lists.ambermd.org/mailman/listinfo/amber>
> >>>>>>
> >>>>>> ______________________________**_________________
> >>>>> AMBER mailing list
> >>>>> AMBER.ambermd.org
> >>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
> http://lists.ambermd.org/mailman/listinfo/amber>
> >>>>>
> >>>>>
> >>>>> ______________________________**_________________
> >>>>> AMBER mailing list
> >>>>> AMBER.ambermd.org
> >>>>> http://lists.ambermd.org/**mailman/listinfo/amber<
> http://lists.ambermd.org/mailman/listinfo/amber>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> ______________________________**_________________
> >>>> AMBER mailing list
> >>>> AMBER.ambermd.org
> >>>> http://lists.ambermd.org/**mailman/listinfo/amber<
> http://lists.ambermd.org/mailman/listinfo/amber>
> >>>>
> >>>> __________ Informace od ESET NOD32 Antivirus, verze databaze 8386
> >>>> (20130528) __________
> >>>>
> >>>> Tuto zpravu proveril ESET NOD32 Antivirus.
> >>>>
> >>>> http://www.eset.cz
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>> --
> >>> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
> >>> http://www.opera.com/mail/
> >>> _______________________________________________
> >>> AMBER mailing list
> >>> AMBER.ambermd.org
> >>> http://lists.ambermd.org/mailman/listinfo/amber
> >>>
> >>>
> > _______________________________________________
> > AMBER mailing list
> > AMBER.ambermd.org
> > http://lists.ambermd.org/mailman/listinfo/amber
> >
> > __________ Informace od ESET NOD32 Antivirus, verze databaze 8390
> > (20130529) __________
> >
> > Tuto zpravu proveril ESET NOD32 Antivirus.
> >
> > http://www.eset.cz
> >
> >
> >
>
>
> --
> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
> http://www.opera.com/mail/
>
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
>
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Wed May 29 2013 - 16:30:03 PDT
