Hi Scott,
We are in the process of getting you a temporary account on our
clusters. Details to follow off list shortly.
All the best,
Patrick
On 5 September 2012 13:50, Scott Le Grand <varelse2005.gmail.com> wrote:
> Yep, it runs all 25,000 steps. Looking at the output you guys sent,
> whatever's happening to Patrick is hosed from the get-go...
>
>
>
> On Wed, Sep 5, 2012 at 4:18 AM, Marc van der Kamp
> <marcvanderkamp.gmail.com>wrote:
>
>> Hi again,
>>
>> The old driver was used by mistake (not commented out). Now 295.41 is
>> installed, but Patrick's problem persists, i.e. NaN values appear in the
>> energy output after a few steps. (With Amber 11 pmemd.cuda the simulation
>> kept running even though it was producing NaN values.)
>> I assume that when you say "it runs at my end" that there aren't any NaN
>> values?
>>
>> I'll ask about getting you an account on the machine in question. It is our
>> institution-wide (University of Bristol) HPC cluster (BlueCrystal), so I'm
>> not sure if that is going to be possible.
>>
>> --Marc
>>
>> On 4 September 2012 22:34, Scott Le Grand <varelse2005.gmail.com> wrote:
>>
>> > Well, can your sysadmin give me account on the machine in question?
>> >
>> > Or please point him to http://developer.nvidia.com/cuda/cuda-downloads
>> >
>> > It indicates 295.41 as the latest driver...
>> >
>> > Your point about passing test.cuda is valid, but since it runs at my end,
>> > there's not much else I can suggest...
>> >
>> >
>> > On Tue, Sep 4, 2012 at 1:31 PM, Marc van der Kamp
>> > <marcvanderkamp.gmail.com>wrote:
>> >
>> > > PS As pmemd.cuda essentially passed make test.cuda completely, I'd
>> think
>> > > the driver shouldn't be the issue?
>> > > It is only Patrick's test cases with transition metals in co-factors
>> that
>> > > are failing. So you can run these same test cases without problems?
>> > > --Marc
>> > >
>> > > On 4 September 2012 21:21, Marc van der Kamp <marcvanderkamp.gmail.com
>> > > >wrote:
>> > >
>> > > > Thanks Scott,
>> > > > The driver was installed by the sysadmin - I don't have root access
>> on
>> > > > this cluster.
>> > > > He wrote to me that it was "the latest driver", but apparently not.
>> > I'll
>> > > > ask him to download a fresh driver from nvidia.
>> > > >
>> > > > Can pmemd.cuda post-bugfix.9 still be compiled with the 4.0 toolkit?
>> I
>> > > > initially tried this, but got an error saying I needed to use 4.2.
>> > > >
>> > > > Thanks,
>> > > > --Marc
>> > > >
>> > > > On 4 September 2012 21:15, Scott Le Grand <varelse2005.gmail.com>
>> > wrote:
>> > > >
>> > > >> Your driver is far too old for 4.2. Either install a newer driver
>> or
>> > > use
>> > > >> the 4.0 toolkit...
>> > > >> On Sep 4, 2012 1:08 PM, "Marc van der Kamp" <
>> marcvanderkamp.gmail.com
>> > >
>> > > >> wrote:
>> > > >>
>> > > >> > Hi Scott,
>> > > >> >
>> > > >> > I compiled pmemd.cuda_SPFP for Patrick and ran make test.cuda. All
>> > > tests
>> > > >> > passed, apart from a few (6 I think) that only had minor
>> differences
>> > > in
>> > > >> > values (different 4th digit).
>> > > >> >
>> > > >> > The CUDA Toolkit:
>> > > >> > $ nvcc -V
>> > > >> > nvcc: NVIDIA (R) Cuda compiler driver
>> > > >> > Copyright (c) 2005-2012 NVIDIA Corporation
>> > > >> > Built on Thu_Apr__5_00:24:31_PDT_2012
>> > > >> > Cuda compilation tools, release 4.2, V0.2.1221
>> > > >> >
>> > > >> > The driver was freshly installed today:
>> > > >> > devdriver_4.0_linux_64_270.41.19
>> > > >> >
>> > > >> > Cards on the node where both test.cuda and Patrick's jobs ran:
>> > > >> > $ nvidia-smi -L
>> > > >> > GPU 0: Tesla M2050 (S/N: 0322310084063)
>> > > >> > GPU 1: Tesla M2050 (S/N: 0322310082367)
>> > > >> >
>> > > >> > Hope this helps,
>> > > >> > Marc
>> > > >> >
>> > > >> >
>> > > >> > On 4 September 2012 19:44, Scott Le Grand <varelse2005.gmail.com>
>> > > >> wrote:
>> > > >> >
>> > > >> > > Does your build of pmemd.cuda pass a make test.cuda?
>> > > >> > >
>> > > >> > > Also what CUDA Toolkit/Display driver are you using?
>> > > >> > >
>> > > >> > > On Tue, Sep 4, 2012 at 10:27 AM, Patrick von Glehn <
>> > > >> > > patrickvonglehn.gmail.com> wrote:
>> > > >> > >
>> > > >> > > > Hi Jason and Scott,
>> > > >> > > >
>> > > >> > > > Unfortunately bugfix 9 has not solved the problem.
>> > > >> > > >
>> > > >> > > > To reiterate for anyone else who is interested, molecular
>> > dynamics
>> > > >> on
>> > > >> > > > my system of interest runs smoothly with pmemd but the system
>> > > blows
>> > > >> up
>> > > >> > > > when run with pmemd.cuda on GPUs (a few atoms in the region of
>> > the
>> > > >> > > > hexacoordinated cobalt fly off in different directions). This
>> > > >> happens
>> > > >> > > > with either a 0.002ps timestep or a 0.000002ps timestep.
>> > > >> > > >
>> > > >> > > > I initially ran the calculations on NVIDIA Tesla M2090 GPUs
>> with
>> > > >> > > > pmemd.cuda_SPDP and then I tried again on Nvidia Fermi M2050
>> > GPUs
>> > > >> with
>> > > >> > > > bufix.9 applied.
>> > > >> > > >
>> > > >> > > > Input files can be found attached to the first message in this
>> > > >> thread.
>> > > >> > > >
>> > > >> > > > Any help would be greatly appreciated,
>> > > >> > > >
>> > > >> > > > Patrick von Glehn
>> > > >> > > > PhD student in the Harvey and Mulholland groups
>> > > >> > > > Centre for Computational Chemistry
>> > > >> > > > University of Bristol
>> > > >> > > >
>> > > >> > > > On 22 August 2012 15:50, Jason Swails <jason.swails.gmail.com
>> >
>> > > >> wrote:
>> > > >> > > > > On Wed, Aug 22, 2012 at 10:28 AM, Patrick von Glehn <
>> > > >> > > > > patrickvonglehn.gmail.com> wrote:
>> > > >> > > > >
>> > > >> > > > >> Hi Scott,
>> > > >> > > > >>
>> > > >> > > > >> Thanks for your reply.
>> > > >> > > > >>
>> > > >> > > > >> Do you have reason to believe that the new patch will
>> resolve
>> > > >> this
>> > > >> > > > >> error? Were you able to reproduce the error with an
>> unpatched
>> > > >> > version
>> > > >> > > > >> of amber? Also, forgive my ignorance, but what does TOT
>> > mean?
>> > > >> > > > >>
>> > > >> > > > >
>> > > >> > > > > Top Of Tree, I think :). What this means is that he doesn't
>> > see
>> > > >> the
>> > > >> > > > error
>> > > >> > > > > with the soon-to-be-released pmemd.cuda upgrade (I don't
>> think
>> > > the
>> > > >> > > > current
>> > > >> > > > > version of amber was tested, but the upcoming patch is known
>> > to
>> > > >> have
>> > > >> > > > fixed
>> > > >> > > > > a handful of bugs).
>> > > >> > > > >
>> > > >> > > > >
>> > > >> > > > >> What sort of timescale are we talking about here for the
>> new
>> > > >> patch
>> > > >> > > > >> release? Days/weeks/months? I am very keen to get my GPU
>> > > >> simulations
>> > > >> > > > >> going!
>> > > >> > > > >>
>> > > >> > > > >
>> > > >> > > > > No promises here, but in conversations I've had with Ross, I
>> > > would
>> > > >> > say
>> > > >> > > > > we're aiming for 'days'. The patch is a large one, and has
>> to
>> > > be
>> > > >> > > handled
>> > > >> > > > > with care, but we're taking a crack at generating the patch
>> > > >> tonight.
>> > > >> > > If
>> > > >> > > > > the merge goes smoothly and everything tests out correctly
>> the
>> > > >> first
>> > > >> > > time
>> > > >> > > > > through, you probably will not have more than a few days to
>> > > wait.
>> > > >> > > > >
>> > > >> > > > > HTH,
>> > > >> > > > > Jason
>> > > >> > > > >
>> > > >> > > > > --
>> > > >> > > > > Jason M. Swails
>> > > >> > > > > Quantum Theory Project,
>> > > >> > > > > University of Florida
>> > > >> > > > > Ph.D. Candidate
>> > > >> > > > > 352-392-4032
>> > > >> > > > > _______________________________________________
>> > > >> > > > > AMBER mailing list
>> > > >> > > > > AMBER.ambermd.org
>> > > >> > > > > http://lists.ambermd.org/mailman/listinfo/amber
>> > > >> > > >
>> > > >> > > > _______________________________________________
>> > > >> > > > AMBER mailing list
>> > > >> > > > AMBER.ambermd.org
>> > > >> > > > http://lists.ambermd.org/mailman/listinfo/amber
>> > > >> > > >
>> > > >> > > _______________________________________________
>> > > >> > > AMBER mailing list
>> > > >> > > AMBER.ambermd.org
>> > > >> > > http://lists.ambermd.org/mailman/listinfo/amber
>> > > >> > >
>> > > >> > _______________________________________________
>> > > >> > AMBER mailing list
>> > > >> > AMBER.ambermd.org
>> > > >> > http://lists.ambermd.org/mailman/listinfo/amber
>> > > >> >
>> > > >> _______________________________________________
>> > > >> AMBER mailing list
>> > > >> AMBER.ambermd.org
>> > > >> http://lists.ambermd.org/mailman/listinfo/amber
>> > > >>
>> > > >
>> > > >
>> > > _______________________________________________
>> > > AMBER mailing list
>> > > AMBER.ambermd.org
>> > > http://lists.ambermd.org/mailman/listinfo/amber
>> > >
>> > _______________________________________________
>> > AMBER mailing list
>> > AMBER.ambermd.org
>> > http://lists.ambermd.org/mailman/listinfo/amber
>> >
>> _______________________________________________
>> AMBER mailing list
>> AMBER.ambermd.org
>> http://lists.ambermd.org/mailman/listinfo/amber
>>
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Mon Sep 10 2012 - 08:30:03 PDT
