100k nucleosome test = identical results:
A V E R A G E S O V E R 100000 S T E P S A V E
R A G E S O V E R 100000 S T E P S
NSTEP = 100000 TIME(PS) = 300.000 TEMP(K) = 310.0 NSTEP =
100000 TIME(PS) = 300.000 TEMP(K) = 310.0
Etot = -66600.0926 EKtot = 19654.9595 EPtot Etot
= -66600.0926 EKtot = 19654.9595 EPtot
BOND = 5795.1298 ANGLE = 13672.2739 DIHED BOND
= 5795.1298 ANGLE = 13672.2739 DIHED
1-4 NB = 5612.4805 1-4 EEL = 1436.2790 VDWAALS 1-4 NB
= 5612.4805 1-4 EEL = 1436.2790 VDWAALS
EELEC = -11449.2413 EGB = -105134.8815 RESTRAINT EELEC
= -11449.2413 EGB = -105134.8815 RESTRAINT
EAMBER (non-restraint) = -86607.8501 EAMBER
(non-restraint) = -86607.8501
------------------------------------------------------------
------------------------------------------------------------
On 4 June 2013 12:39, Marek Maly <marek.maly.ujep.cz> wrote:
> Hi,
> here are my results from the "NTPR" experiment:
>
>
> Total energy at step 100 000 reported for ROUND_1 and ROUND_2
> (driver 319.23, Amber12 bugfix 18 applied, cuda 5.0) (In all cases)
>
> GTX580 (NTPR=1000)
> -66801.3274
> -66801.3274
>
> TITAN_0 (NTPR=1)
> -66854.0492
> -66802.4419
>
> TITAN_1 (NTPR=1)
> -66858.7444
> -66858.7444
>
>
> M.
>
>
>
>
> Dne Tue, 04 Jun 2013 06:14:28 +0200 Marek Maly <marek.maly.ujep.cz>
> napsal/-a:
>
> > Hi Scott,
> >
> > I am sending again my very first tests/table (see attached) where
> > I did also GTX 580/GTX 680 tests as a control and as you can see
> > here I have obtained perfect reproducibility on those GTX but also
> > on my second TITAN card (TITAN_1) for NUCLEOSOME ! But that was with
> > driver 319.17
> > (and also before bugfix 18).
> >
> > Now I will try on my titans again with ntpr=1 as you wish
> > (driver 319.23, Amber12 bugfix 18 applied, cuda 5.0).
> >
> > Simultaneously I will repeat this test on GTX 580 with ntpr=1000
> > (driver 319.23, Amber12 bugfix 18 applied, cuda 5.0).
> >
> > BTW I also experimented a bit, first try to use some settings from
> > NUCLEOSOME (e.g. igb=5, ntt=1/3, saltcon=0.1, tautp=1.0 + restrains) and
> > use it
> > for TRP cage and Myoglob. assuming these params which are different
> > between NUCLE and TRP + MYO will affect the TRP + MYO reproducibility.
> >
> > This was not confirmed i.e. TRP + MYO still perfectly reproducible.
> >
> > So then (to be sure) I did opposite exper. and used TRP mdin file for
> > NUCLEOSOME to see
> > if it influence NUCL reproducibility, but in agreement with "TRP-MYO"
> > tests NUCL
> > was again irreproducible ...
> >
> > So let's see the ntpr tests.
> >
> > M.
> >
> >
> >
> >
> > Dne Tue, 04 Jun 2013 04:51:08 +0200 Scott Le Grand
> > <varelse2005.gmail.com>
> > napsal/-a:
> >
> >> Update: The nucleosome GB irreproducibility is weird. it goes away on
> >> my
> >> Titan if I set ntpr to 1 (was trying to find the offending energy
> >> component
> >> that diverges first). Can you guys try this on your machines? I think
> >> this might be SW...
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Mon, Jun 3, 2013 at 1:18 PM, ET <sketchfoot.gmail.com> wrote:
> >>
> >>> Hi Scott & Ross,
> >>>
> >>> I take it you will post to this thread once a fix has been found? :)
> >>>
> >>> br,
> >>> g
> >>>
> >>>
> >>> On 3 June 2013 20:31, Marek Maly <marek.maly.ujep.cz> wrote:
> >>>
> >>> > OK,
> >>> > I just took deep breath and started to pray :))
> >>> >
> >>> > BTW, the difference between GB results TRPcage/myoglobin (perfectly
> >>> > reproducible)
> >>> > versus Nucleosome (irreproducible res.) might be connected with some
> >>> > differences
> >>> > in mdin parameters:
> >>> >
> >>> > TRPcage/myoglobin (igb=1, ntt=3) versus Nucleosome (igb=5, ntt=1).
> >>> > Nucleosome simul. is also
> >>> > with restraint:
> >>> >
> >>> > RESTRAIN DNA
> >>> > 0.1
> >>> > RES 1 294
> >>> > END
> >>> > END
> >>> >
> >>> > I will try to experiment here to learn which parameter is responsible
> >>> for
> >>> > the
> >>> > Nucleosome irreproducible results.
> >>> >
> >>> > M.
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > Dne Mon, 03 Jun 2013 21:17:23 +0200 Ross Walker
> >>> <ross.rosswalker.co.uk>
> >>> > napsal/-a:
> >>> >
> >>> > > Hi Marek,
> >>> > >
> >>> > > To be honest I would just take a deep breath and give us some time
> >>> to
> >>> > > figure out what is going on with the Titan and work around it.
> >>> Hopefully
> >>> > > this won't take too long and we can have a patch out shortly.
> >>> > >
> >>> > > All the best
> >>> > > Ross
> >>> > >
> >>> > >
> >>> > >
> >>> > > On 6/3/13 11:47 AM, "Marek Maly" <marek.maly.ujep.cz> wrote:
> >>> > >
> >>> > >> Thanks Scott !
> >>> > >>
> >>> > >> sounds me like "Of course you can win gold treasure if you survive
> >>> > >> Russian
> >>> > >> roulette before ..."
> >>> > >>
> >>> > >> It seems that the difference in reliability for sci. calc. between
> >>> > >> Teslas
> >>> > >>
> >>> > >> and "equivalent" stock GTXs
> >>> > >> is now (with chip GTK110) clearly bigger. I am curious how it will
> >>> be
> >>> > >> with
> >>> > >> GTX 780 comparing to Titans.
> >>> > >>
> >>> > >> So let's hope that in the worst case downclocking of Titans might
> >>> solve
> >>> > >> the problem.
> >>> > >>
> >>> > >> BTW what is the working temperature of your K20c ? My Titans works
> >>> under
> >>> > >> 80°C (cca
> >>> > >> 60% Fan utilization). For the older cards (GTX 680/580 ...) this
> >>> temp.
> >>> > >> should be OK but
> >>> > >> maybe for the GTK110 this temp is already too high to ensure zero
> >>> "bit
> >>> > >> fluctuations".
> >>> > >>
> >>> > >> cuFFT is maybe responsible for crashes and maybe also some
> >>> > >> irreproducibility but the irreproducibility of the results will
> >>> have
> >>> > >> also
> >>> > >>
> >>> > >> some another source as suggests
> >>> > >> NUCLEOSOME GB test where perhaps no FFT is involved ? (just the
> >>> real
> >>> > >> space calc.).
> >>> > >>
> >>> > >> So thanks for the moment and please let us know when you do some
> >>> > >> progress.
> >>> > >>
> >>> > >>
> >>> > >> M.
> >>> > >>
> >>> > >>
> >>> > >>
> >>> > >> Dne Mon, 03 Jun 2013 20:12:04 +0200 Scott Le Grand
> >>> > >> <varelse2005.gmail.com>
> >>> > >> napsal/-a:
> >>> > >>
> >>> > >>> Addressing Divi's two points:
> >>> > >>>
> >>> > >>> 1. We're trying to find a way to do this...
> >>> > >>>
> >>> > >>> 2. I am extremely paranoid and while I would still use the Titans
> >>> for
> >>> > >>> development and testing, I would also currently do my publishable
> >>> runs
> >>> > >>> on
> >>> > >>> GK104 GPUs or K20s. Given that, if you're comfortable with
> >>> > >>> nondeterministic execution ala GROMACS, ACEMD, and NAMD, what's
> >>> going
> >>> > >>> on
> >>> > >>> here is seemingly no worse. I'm *not* comfortable with that
> >>> myself
> >>> and
> >>> > >>> I
> >>> > >>> intend to find a fix or workaround like we did a couple years ago
> >>> with
> >>> > >>> GTX4xx and GTX5xx. So your best strategy might just be to wait a
> >>> week
> >>> > >>> or
> >>> > >>> two and see what comes of the bug hunt.
> >>> > >>>
> >>> > >>> Marek et al. if these GPU tests are failing on the Titans, then
> >>> by
> >>> all
> >>> > >>> means return them without hesitation, but I don't think consumer
> >>> level
> >>> > >>> GPUs
> >>> > >>> are tested with the same level of rigor as Teslas. The upside is
> >>> you
> >>> > >>> get
> >>> > >>> 30% better performance for 1/3 the price. The downside is that
> >>> IMO
> >>> you
> >>> > >>> should be carefully validate them before using them. What I'm
> >>> seeing
> >>> > >>> here
> >>> > >>> looks like single bit differences at the low-order bits that
> >>> cause a
> >>> > >>> tiny
> >>> > >>> fluctuation that ultimately mushrooms and diverges the whole
> >>> shebang
> >>> > >>> along
> >>> > >>> with occasional crashes. The crashes seem to occur in cuFFT
> >>> somewhere.
> >>> > >>>
> >>> > >>> I
> >>> > >>> have yet to see divergence there yet.
> >>> > >>>
> >>> > >>> Scott
> >>> > >>>
> >>> > >>>
> >>> > >>> On Mon, Jun 3, 2013 at 9:42 AM, Marek Maly <marek.maly.ujep.cz>
> >>> wrote:
> >>> > >>>
> >>> > >>>> Hi,
> >>> > >>>> so here are my NUCLEOSOME test results. All tests finished
> >>> (although
> >>> > >>>> the
> >>> > >>>> TITAN_0/ROUND_2) with "****" energy (*** records starts from the
> >>> 75K
> >>> > >>>> step
> >>> > >>>> so
> >>> > >>>> it is surprise for me that test was finished at the end). All
> >>> the
> >>> > >>>> results
> >>> > >>>> are irreproducible (driver 319.23, Amber12 bugfix 18 applied,
> >>> cuda
> >>> > >>>> 5.5)
> >>> > >>>> I
> >>> > >>>> will
> >>> > >>>> repeat it with CUDA 5.0.
> >>> > >>>>
> >>> > >>>> M.
> >>> > >>>>
> >>> > >>>> >>>>>> TITAN_0
> >>> > >>>>
> >>> > >>>>
> >>> > >>>> ROUND_1
> >>> > >>>>
> >>> > >>>>
> >>> > >>>>
> >>> >
> >>>
> ------------------------------------------------------------------------
> >>> > >>>> ------
> >>> > >>>>
> >>> > >>>>
> >>> > >>>> NSTEP = 100000 TIME(PS) = 300.000 TEMP(K) = 310.60
> >>> PRESS
> >>> > >>>> = 0.0
> >>> > >>>> Etot = -66843.8345 EKtot = 19690.5156 EPtot
> >>> =
> >>> > >>>> -86534.3502
> >>> > >>>> BOND = 5887.3611 ANGLE = 13673.5215 DIHED
> >>> =
> >>> > >>>> 16941.7678
> >>> > >>>> 1-4 NB = 5576.6911 1-4 EEL = 1371.5924 VDWAALS
> >>> =
> >>> > >>>> -13647.8461
> >>> > >>>> EELEC = -14410.1252 EGB = -102286.9459 RESTRAINT
> >>> =
> >>> > >>>> 359.6331
> >>> > >>>> EAMBER (non-restraint) = -86893.9832
> >>> > >>>>
> >>> > >>>>
> >>> > >>>>
> >>> >
> >>>
> ------------------------------------------------------------------------
> >>> > >>>> ------
> >>> > >>>>
> >>> > >>>> ROUND_2
> >>> > >>>>
> >>> > >>>>
> >>> > >>>>
> >>> >
> >>>
> ------------------------------------------------------------------------
> >>> > >>>> ------
> >>> > >>>>
> >>> > >>>>
> >>> > >>>> NSTEP = 100000 TIME(PS) = 300.000 TEMP(K) =*********
> >>> PRESS
> >>> > >>>> = 0.0
> >>> > >>>> Etot = ************** EKtot = ************** EPtot
> >>> =
> >>> > >>>> 4279668.7807
> >>> > >>>> BOND = -0.0000 ANGLE = 4681740.3488 DIHED
> >>> =
> >>> > >>>> 67661.6797
> >>> > >>>> 1-4 NB = -0.0000 1-4 EEL = -2.0373 VDWAALS
> >>> =
> >>> > >>>> 244.1012
> >>> > >>>> EELEC = 72548.4049 EGB = -542523.7166 RESTRAINT
> >>> =
> >>> > >>>> -0.0000
> >>> > >>>> EAMBER (non-restraint) = 4279668.7807
> >>> > >>>>
> >>> > >>>>
> >>> > >>>>
> >>> >
> >>>
> ------------------------------------------------------------------------
> >>> > >>>> ------
> >>> > >>>> STARS from the 75k step ...
> >>> > >>>>
> >>> > >>>>
> >>> > >>>> >>>>>> TITAN_1
> >>> > >>>>
> >>> > >>>>
> >>> > >>>> ROUND_1
> >>> > >>>>
> >>> > >>>>
> >>> > >>>>
> >>> >
> >>>
> ------------------------------------------------------------------------
> >>> > >>>> ------
> >>> > >>>>
> >>> > >>>>
> >>> > >>>> NSTEP = 100000 TIME(PS) = 300.000 TEMP(K) = 310.36
> >>> PRESS
> >>> > >>>> = 0.0
> >>> > >>>> Etot = -66846.8801 EKtot = 19675.0488 EPtot
> >>> =
> >>> > >>>> -86521.9289
> >>> > >>>> BOND = 5760.2422 ANGLE = 13619.8710 DIHED
> >>> =
> >>> > >>>> 16996.9045
> >>> > >>>> 1-4 NB = 5645.6416 1-4 EEL = 1774.6967 VDWAALS
> >>> =
> >>> > >>>> -13622.9343
> >>> > >>>> EELEC = -14168.1788 EGB = -102880.8089 RESTRAINT
> >>> =
> >>> > >>>> 352.6371
> >>> > >>>> EAMBER (non-restraint) = -86874.5660
> >>> > >>>>
> >>> > >>>>
> >>> > >>>>
> >>> >
> >>>
> ------------------------------------------------------------------------
> >>> > >>>> ------
> >>> > >>>>
> >>> > >>>> ROUND_2
> >>> > >>>>
> >>> > >>>>
> >>> > >>>>
> >>> >
> >>>
> ------------------------------------------------------------------------
> >>> > >>>> ------
> >>> > >>>>
> >>> > >>>>
> >>> > >>>> NSTEP = 100000 TIME(PS) = 300.000 TEMP(K) = 311.00
> >>> PRESS
> >>> > >>>> = 0.0
> >>> > >>>> Etot = -66874.9016 EKtot = 19715.3633 EPtot
> >>> =
> >>> > >>>> -86590.2649
> >>> > >>>> BOND = 5819.0667 ANGLE = 13683.6633 DIHED
> >>> =
> >>> > >>>> 16918.8596
> >>> > >>>> 1-4 NB = 5627.0932 1-4 EEL = 1576.9564 VDWAALS
> >>> =
> >>> > >>>> -13747.1032
> >>> > >>>> EELEC = -15232.3280 EGB = -101590.5078 RESTRAINT
> >>> =
> >>> > >>>> 354.0348
> >>> > >>>> EAMBER (non-restraint) = -86944.2997
> >>> > >>>>
> >>> > >>>>
> >>> > >>>>
> >>> >
> >>>
> ------------------------------------------------------------------------
> >>> > >>>> ------
> >>> > >>>>
> >>> > >>>>
> >>> > >>>>
> >>> > >>>>
> >>> > >>>>
> >>> > >>>>
> >>> > >>>>
> >>> > >>>>
> >>> > >>>> Dne Mon, 03 Jun 2013 12:34:15 +0200 Marek Maly
> >>> <marek.maly.ujep.cz>
> >>> > >>>> napsal/-a:
> >>> > >>>>
> >>> > >>>> > OK, I will try NUCLEOSOME case as well with my latest
> >>> > >>>> > settings : (driver 319.23, Amber12 bugfix 18 applied, cuda
> >>> 5.5)
> >>> > >>>> >
> >>> > >>>> > M.
> >>> > >>>> >
> >>> > >>>> >
> >>> > >>>> >
> >>> > >>>> >
> >>> > >>>> > Dne Mon, 03 Jun 2013 11:51:46 +0200 ET <sketchfoot.gmail.com>
> >>> > >>>> napsal/-a:
> >>> > >>>> >
> >>> > >>>> >> Hi all,
> >>> > >>>> >>
> >>> > >>>> >> I reran the benchmark with Amber recompiled and at the latest
> >>> > >>>> drivers
> >>> > >>>> >> with
> >>> > >>>> >> GPU in solo configuration yields the following results:
> >>> > >>>> >>
> >>> > >>>> >>
> >>> > >>>> >> When I run the tests on GPU-00_TeaNCake:
> >>> > >>>> >>
> >>> > >>>> >> 1) All the tests (across 2x repeats) finish successfully:
> >>> > >>>> >>
> >>> > >>>> >>
> >>> > >>>> >> 2) The sdiff logs indicate that reproducibility across the
> >>> two
> >>> > >>>> repeats
> >>> > >>>> >> is
> >>> > >>>> >> as follows:
> >>> > >>>> >>
> >>> > >>>> >> GB_myoglobin: Reproducible across 1,000,000 steps
> >>> > >>>> >> GB_nucleosome: No reproducibility shown from step 3,400
> >>> onwards.
> >>> > >>>> Also
> >>> > >>>> >> outfile is not written properly - blank gaps appear where
> >>> something
> >>> > >>>> >> should
> >>> > >>>> >> have been written.
> >>> > >>>> >> GB_TRPCage: Reproducible across 1,000,000 steps
> >>> > >>>> >>
> >>> > >>>> >> PME_JAC_production_NVE: No reproducibility shown from step
> >>> 35,000
> >>> > >>>> >> onwards.
> >>> > >>>> >> Also outfile is not written properly - blank gaps appear
> >>> where
> >>> > >>>> something
> >>> > >>>> >> should have been written.
> >>> > >>>> >> PME_JAC_production_NPT: No reproducibility shown from step
> >>> 69,000
> >>> > >>>> >> onwards.
> >>> > >>>> >> Also outfile is not written properly - blank gaps appear
> >>> where
> >>> > >>>> something
> >>> > >>>> >> should have been written.
> >>> > >>>> >> PME_FactorIX_production_NVE: Reproducible across 100k steps
> >>> > >>>> >> PME_FactorIX_production_NPT: Reproducible across 100k steps
> >>> > >>>> >> PME_Cellulose_production_NVE: Reproducible across 100k steps
> >>> > >>>> >> PME_Cellulose_production_NPT: No reproducibility shown from
> >>> step
> >>> > >>>> 17,000
> >>> > >>>> >> onwards. Also outfile is not written properly - blank gaps
> >>> appear
> >>> > >>>> where
> >>> > >>>> >> something should have been written.
> >>> > >>>> >>
> >>> > >>>> >> #################################################
> >>> > >>>> >>
> >>> > >>>> >>
> >>> > >>>> >> So it looks like the problem does occur in GB runs too.
> >>> Though I
> >>> > >>>> notice
> >>> > >>>> >> that running in single GPU mode seems to make the problem
> >>> appear
> >>> > >>>> much
> >>> > >>>> >> later
> >>> > >>>> >> than it occurs with dual GPUs, though obviously this is quite
> >>> > >>>> >> qualitative
> >>> > >>>> >> and based only of 1 repeat.
> >>> > >>>> >>
> >>> > >>>> >> br,
> >>> > >>>> >> g
> >>> > >>>> >>
> >>> > >>>> >>
> >>> > >>>> >>
> >>> > >>>> >>
> >>> > >>>> >> On 3 June 2013 10:28, ET <sketchfoot.gmail.com> wrote:
> >>> > >>>> >>
> >>> > >>>> >>> Hi Marek,
> >>> > >>>> >>>
> >>> > >>>> >>> I think what you say about Valley and Heaven are true to a
> >>> certain
> >>> > >>>> >>> extent,
> >>> > >>>> >>> but I think the links I posted to the EVGA overclock utility
> >>> &
> >>> MSI
> >>> > >>>> >>> Kombuster are very good ways of testing the card. I don't
> >>> know
> >>> the
> >>> > >>>> >>> details
> >>> > >>>> >>> of memtestG80 and cuda_memtest, but it seems to me that they
> >>> are
> >>> > >>>> >>> testing
> >>> > >>>> >>> one very specific component. i.e. The Memory. As the
> >>> graphics
> >>> card
> >>> > >>>> >>> consists
> >>> > >>>> >>> of more than this, it is better to have a test that checks
> >>> the
> >>> > >>>> card
> >>> > >>>> in
> >>> > >>>> >>> a
> >>> > >>>> >>> more holistic manner IMO. :)
> >>> > >>>> >>>
> >>> > >>>> >>> I think this argument is supported by the fact that tech
> >>> support
> >>> > >>>> at
> >>> > >>>> the
> >>> > >>>> >>> store used a program called FurMark to stress test the GPU.
> >>> As
> >>> the
> >>> > >>>>
> >>> > >>>> GPU
> >>> > >>>> >>> I
> >>> > >>>> >>> returned kept failing the benchmark, they realized in less
> >>> than
> >>> > >>>> half a
> >>> > >>>> >>> day
> >>> > >>>> >>> it was faulty, whilst I wasted a couple of days mucking
> >>> about
> >>> with
> >>> > >>>>
> >>> > >>>> GPU
> >>> > >>>> >>> memory tests using Gpuburn on linux.
> >>> > >>>> >>>
> >>> > >>>> >>> http://www.ozone3d.net/benchmarks/fur/
> >>> > >>>> >>>
> >>> > >>>> >>> I think if you are going to test on windows, you are better
> >>> of
> >>> > >>>> getting
> >>> > >>>> >>> MSI
> >>> > >>>> >>> Kombuster which I posted earlier. It contains the test
> >>> contained
> >>> > >>>> in
> >>> > >>>> >>> Furmark
> >>> > >>>> >>> and many additional tests that test the compute capability
> >>> of
> >>> the
> >>> > >>>> card.
> >>> > >>>> >>>
> >>> > >>>> >>> best regards,
> >>> > >>>> >>> g
> >>> > >>>> >>>
> >>> > >>>> >> _______________________________________________
> >>> > >>>> >> AMBER mailing list
> >>> > >>>> >> AMBER.ambermd.org
> >>> > >>>> >> http://lists.ambermd.org/mailman/listinfo/amber
> >>> > >>>> >>
> >>> > >>>> >> __________ Informace od ESET NOD32 Antivirus, verze databaze
> >>> 8405
> >>> > >>>> >> (20130603) __________
> >>> > >>>> >>
> >>> > >>>> >> Tuto zpravu proveril ESET NOD32 Antivirus.
> >>> > >>>> >>
> >>> > >>>> >> http://www.eset.cz
> >>> > >>>> >>
> >>> > >>>> >>
> >>> > >>>> >>
> >>> > >>>> >
> >>> > >>>> >
> >>> > >>>>
> >>> > >>>>
> >>> > >>>> --
> >>> > >>>> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
> >>> > >>>> http://www.opera.com/mail/
> >>> > >>>>
> >>> > >>>> _______________________________________________
> >>> > >>>> AMBER mailing list
> >>> > >>>> AMBER.ambermd.org
> >>> > >>>> http://lists.ambermd.org/mailman/listinfo/amber
> >>> > >>>>
> >>> > >>> _______________________________________________
> >>> > >>> AMBER mailing list
> >>> > >>> AMBER.ambermd.org
> >>> > >>> http://lists.ambermd.org/mailman/listinfo/amber
> >>> > >>>
> >>> > >>> __________ Informace od ESET NOD32 Antivirus, verze databaze 8407
> >>> > >>> (20130603) __________
> >>> > >>>
> >>> > >>> Tuto zpravu proveril ESET NOD32 Antivirus.
> >>> > >>>
> >>> > >>> http://www.eset.cz
> >>> > >>>
> >>> > >>>
> >>> > >>>
> >>> > >>
> >>> > >>
> >>> > >> --
> >>> > >> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
> >>> > >> http://www.opera.com/mail/
> >>> > >>
> >>> > >> _______________________________________________
> >>> > >> AMBER mailing list
> >>> > >> AMBER.ambermd.org
> >>> > >> http://lists.ambermd.org/mailman/listinfo/amber
> >>> > >
> >>> > >
> >>> > >
> >>> > > _______________________________________________
> >>> > > AMBER mailing list
> >>> > > AMBER.ambermd.org
> >>> > > http://lists.ambermd.org/mailman/listinfo/amber
> >>> > >
> >>> > > __________ Informace od ESET NOD32 Antivirus, verze databaze 8408
> >>> > > (20130603) __________
> >>> > >
> >>> > > Tuto zpravu proveril ESET NOD32 Antivirus.
> >>> > >
> >>> > > http://www.eset.cz
> >>> > >
> >>> > >
> >>> > >
> >>> >
> >>> >
> >>> > --
> >>> > Tato zpráva byla vytvořena převratným poštovním klientem Opery:
> >>> > http://www.opera.com/mail/
> >>> >
> >>> > _______________________________________________
> >>> > AMBER mailing list
> >>> > AMBER.ambermd.org
> >>> > http://lists.ambermd.org/mailman/listinfo/amber
> >>> >
> >>> _______________________________________________
> >>> AMBER mailing list
> >>> AMBER.ambermd.org
> >>> http://lists.ambermd.org/mailman/listinfo/amber
> >>>
> >> _______________________________________________
> >> AMBER mailing list
> >> AMBER.ambermd.org
> >> http://lists.ambermd.org/mailman/listinfo/amber
> >>
> >> __________ Informace od ESET NOD32 Antivirus, verze databaze 8408
> >> (20130603) __________
> >>
> >> Tuto zpravu proveril ESET NOD32 Antivirus.
> >>
> >> http://www.eset.cz
> >>
> >>
> >>
> >
> >
>
>
> --
> Tato zpráva byla vytvořena převratným poštovním klientem Opery:
> http://www.opera.com/mail/
>
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
>
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Tue Jun 04 2013 - 06:30:02 PDT
