Hi Ross,
This is very interesting... load balancing is not what I would have presumed the variance to come from. It sounds like the only way to control for this variance in the cpu version is to run it on a single thread, which is one step closer to leveling the playing field. Thank you for the very helpful tip because the more variance I can account for the better! The fact that cuda converges while the cpu doesn't seems like a benefit to using cuda as it leads to more reproducable results and more control of the problem at hand.
TG
________________________________________
From: Ross Walker [ross.rosswalker.co.uk]
Sent: Monday, September 13, 2010 4:55 PM
To: 'AMBER Mailing List'
Subject: Re: [AMBER] Exactly the same energy output over multiple runs of cuda_SPDP
Hi Trevor,
This is perfectly normal, and mildly refreshing!!! The single GPU runs are
actually totally deterministic. The load balancing is static on the GPU
hence there is no variation between runs etc. You will see the exact same
behavior with a single CPU run. It is just in parallel where the load
balancer is dynamic that you see round off differences.
In summary you should ALWAYS change the value of ig. Set it to -1 is
generally the best option. Also you should always start from slightly
different starting structures if you can. And or begin your runs from 10K or
so instead of 0K and then at the very least your initial velocity
distribution should be different and your runs should diverge over time.
All the best
Ross
> -----Original Message-----
> From: Trevor Gokey [mailto:tgokey.mail.sfsu.edu]
> Sent: Monday, September 13, 2010 4:33 PM
> To: amber.ambermd.org
> Subject: [AMBER] Exactly the same energy output over multiple runs of
> cuda_SPDP
>
> Hello,
>
> I found that all runs using cuda produce the exact same output (same
> energy, same velocities, same coordinates, etc) across 30 mutually
> exclusive runs starting from the same inpcrd and prmtop. This occurs
> for all of my stages, including minimization, heating, etc. Running the
> same job on the same computer using pmemd.MPI produces small
> fluctuations as one would expect. I know these are deterministic but
> something seems fishy when I get the exact same output for 30 tries. I
> should note that the numbers that cuda produces are comparable to cpu
> version, just no variance. I'm running cuda on a GTX260. I can provide
> inputs and more detailed specs if requested...
>
> -Trevor Gokey
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Mon Sep 13 2010 - 18:30:02 PDT
