pps:
http://www.netspi.com/blog/2013/04/07/gpu-cracking-building-the-box/
On 25 June 2013 06:19, ET <sketchfoot.gmail.com> wrote:
> ps. Anyone been tempted by a bitcoin mining rig type setup? :P
>
> http://bitcoinexaminer.org/20-insane-bitcoin-mining-rigs/
>
> The PCIe extender cable looks good in terms of allowing your card to
> breathe if you can custom rig a case that's not going to burn down your
> house.
>
> http://www.ebay.co.uk/bhp/pci-e-riser
>
> br,
> g
>
>
>
>
> On 25 June 2013 04:06, ET <sketchfoot.gmail.com> wrote:
>
>> Hi Kevin,
>>
>> Thanks for the info on the case mods & temps. I decided to go with the
>> HAF-X case, which despite it's ugliness has the the ducts that you
>> mentioned.. Have got my fingers crossed that the switch connectors won't be
>> covered by the final GPU. :)
>>
>> I think the price difference between my build & that of Ross is ~ £250.
>> However, as I'm short on time & did not really like the look of AM3+ for
>> various reasons, I don't mind the cash difference. Additionally, I'm prob
>> going to sell this machine year end, so kitted it so it would be desirable
>> for an overclocker.
>>
>>
>> br,
>> g
>>
>>
>> On 25 June 2013 00:58, Ross Walker <ross.rosswalker.co.uk> wrote:
>>
>>> Hi Kevin,
>>>
>>> Thanks for the detailed info. The GTX680s in our experience have been
>>> rock
>>> solid stable. 1 out of 20 for infant death sounds about right in my
>>> experience. You should be able to just RMA that card and you should be
>>> good. Note, you can also build such systems with AMD processors. Attached
>>> is a pdf with an Amazon shopping list for a 4 GTX680 system which we have
>>> built many of for $3200 - might even be cheaper now with the 680s coming
>>> down in price. - This motherboard takes all 4 GPUs without hitting
>>> anything - as long as you don't try to connect up a bunch of external USB
>>> connectors.
>>>
>>> This same system should work great for GTX780s as well - we just need to
>>> make sure they are giving the correct answers - looking more positive by
>>> the day.
>>>
>>> I've not seen any major issues with 4 GPU cooling in these systems - as
>>> long as you have plenty of back airflow as you have ones should be good.
>>> 90C is a normal temperature for GTX680 and I've run several flat out for
>>> months on end at this temperature.
>>>
>>> I second the choice of slurm. It's certainly far from "Simple" but it
>>> does
>>> seem to understand GPUs better than any of the other queuing systems out
>>> there. Indeed - the 'certified' clusters I designed with exxact use
>>> Slurm.
>>> Rocks with the slurm roll works great for GPU clusters.
>>>
>>> Again, thanks for this info, should be useful to lots of people here.
>>>
>>> All the best
>>> Ross
>>>
>>>
>>> On 6/24/13 4:40 PM, "Kevin Hauser" <84hauser.gmail.com> wrote:
>>>
>>> >A good bit late to the discussion, but we've been having success with a
>>> >relatively cheap setup ~ $3,200 per machine and five machines (about 3
>>> >weeks of production burning; acceptable temperatures; see below).
>>> >
>>> >Bad news first:
>>> >We lost one GPU (out of 20). Not bad, considering our expectations from
>>> >commodity kit.
>>> >
>>> >Briefly, we got five quad-GPU boxes running EVGA GTX680-FTW, Intel
>>> >i7-3820,
>>> >on the GigaByte GA-X79-UD3 LGA2011 mobo. # See slide one of attached
>>> *pdf
>>> >for overview.
>>> >
>>> >Discussion on our kit:
>>> >The Antec P280 cases are great b/c all metal is rubber coated for quiet
>>> >running, there's three fans included, a quick mount for a fourth 120mm
>>> >fan,
>>> >and space for half-dozen HDDs. It's pretty heavy, though (22 lbs, dry).
>>> >--$110
>>> >
>>> >The CPU is what it is; from our vendor, cpu cooler is not included. We
>>> got
>>> >the Cooler Master Hyper212 --$300 + $33
>>> >
>>> >The mobos appear to be well manufactured, especially given the price
>>> (half
>>> >of ET's Asus kit). BUT, I needed to take apart the power and reset
>>> >terminals connecting case to mobo so the last GPU fully seats into the
>>> >mobo. -- $230 # See slide two of attached *pdf.
>>> >
>>> >Good news last:
>>> >I took the DHFR test case (mdin below) and ran it for 100 ps. I left
>>> >ntpr=1
>>> >to see when and where things got funky... Every single GPU in all five
>>> >"nodes" produced identical mdouts. Afterwards, we had that one GPU die,
>>> >though. I'm getting double the speed we were getting on our center's
>>> mega
>>> >expensive server (Tesla M2070s).
>>> >custom short md
>>> > &cntrl
>>> > nstlim=100000, ig=11,ioutfm=1,ntxo=2,
>>> > ntx=5, irest=1,
>>> > ntc=2, ntf=2,
>>> > ntpr=1, ntwr=10000,
>>> > dt=0.002,
>>> > ntt=1, tautp=10.0,
>>> > temp0=300.0,
>>> > ntb=2,ntp=1,taup=10.0,
>>> > /
>>> >
>>> >
>>> >Burn test info:
>>> >Overall, our GPUs have not exceeded 90 C, yet. Max sustained we've seen
>>> is
>>> >87 C. They're in our old, cold server room.
>>> >
>>> >Only down time last three weeks or so was when we were sorting out our
>>> NFS
>>> >or PXE or Slurm queuing system (only a few hours, really). Slurm is
>>> >actually quite nice, simple, and very free.
>>> >
>>> >Of course, the intake fans for the GPUs (save bottom one) suck air right
>>> >off the heat sink of the GPU below it. The GPUs are cleverly tapered
>>> right
>>> >where the intake fans face the heat sink, leaving a whopping 2 or so
>>> >millimeters for air. On slide one, we installed a massive fan to ram
>>> fresh
>>> >air into the intakes (mod_1). Tests show that 3 GPUs heat up to 85 C
>>> +/- 2
>>> >(CVD=0,1,3) and 1 GPU to 76 C (CVD=2). Cooler Master R4-MFJR-07FK-R1
>>> 200mm
>>> >MegaFlow --$19 # cvd=cuda_visible_device
>>> >
>>> >On slide 3, you can see I ghetto-fabricated a cardboard box that ducts
>>> air
>>> >from a 120mm fan directly into the GPUs' intake-tapered section (mod_2).
>>> >Tests show that CVD=0 hits 84 C, CDV=1 hits 83 C, CVD=2 hits 74 C, and
>>> >CVD=3 hits 82 C. Benefit, the case has a very simple clip-on mount for
>>> >120mm. Downside, we needed that duct to realize benefit. We yanked a fan
>>> >from the top of the case that was needlessly serving the CPU.
>>> >
>>> >
>>> >HTH,
>>> >kevin
>>> >
>>> >
>>> >On Mon, Jun 24, 2013 at 2:48 PM, ET <sketchfoot.gmail.com> wrote:
>>> >
>>> >> Thanks for the further info Ross! :)
>>> >>
>>> >> Decided in the end to go for a Asus P8Z77 WS board with a Intel i7
>>> >>3770K.
>>> >> Slightly overkill, but needed to proof it in the event of resale or
>>> >>finding
>>> >> another use for it.
>>> >>
>>> >> br,
>>> >> g
>>> >>
>>> >>
>>> >> On 25 June 2013 03:44, <deeptinayar.gmail.com> wrote:
>>> >>
>>> >> >
>>> >> > Sent from my BlackBerry® smartphone from !DEA
>>> >> >
>>> >> > -----Original Message-----
>>> >> > From: ET <sketchfoot.gmail.com>
>>> >> > Date: Sat, 22 Jun 2013 20:45:20
>>> >> > To: AMBER Mailing List<amber.ambermd.org>
>>> >> > Reply-To: AMBER Mailing List <amber.ambermd.org>
>>> >> > Subject: Re: [AMBER] Anyone running machines with Quad GPU setups
>>> >> >
>>> >> > Looks like the Asus P9X79-E WS is for you then Scott! :) Haven't
>>> seen
>>> >> many
>>> >> > (if any!) boards with that anount of bandwidth so far.
>>> >> >
>>> >> > br,
>>> >> > g
>>> >> >
>>> >> >
>>> >> > On 22 June 2013 19:31, Scott Le Grand <varelse2005.gmail.com>
>>> wrote:
>>> >> >
>>> >> > > It may be overkill now but I'm planning to revisit the Multi GPU
>>> >>code
>>> >> in
>>> >> > > the near future and that's why I need a motherboard that can
>>> really
>>> >> take
>>> >> > > advantage of it.
>>> >> > > On Jun 22, 2013 10:12 AM, "ET" <sketchfoot.gmail.com> wrote:
>>> >> > >
>>> >> > > > Hi,
>>> >> > > >
>>> >> > > > Thanks v much for you specs Divi. :) I've been debating with
>>> >>myself
>>> >> as
>>> >> > my
>>> >> > > > board as it looks good and has a very nice spec. From what I've
>>> >>read,
>>> >> > the
>>> >> > > > only problems with it is the higher than average power draw.
>>> >> > > >
>>> >> > > > .Scott. I believe the board runs in x8/x8/x8/x8 for a 4 GPU
>>> >>config -
>>> >> so
>>> >> > > > effectively PCI2 2.0 x16 rate. Would this present any problems,
>>> >>if
>>> >> you
>>> >> > > > were running the serial GPU code, From what I read on the AMBER
>>> >>GPU
>>> >> > > > hardware page, this is more important for the parallel GPU code?
>>> >> > Though,
>>> >> > > I
>>> >> > > > imagine having 4x serial ruins going simultaneously would also
>>> tax
>>> >> the
>>> >> > > GPU
>>> >> > > > to CPU interface, though how much I'm not sure.
>>> >> > > >
>>> >> > > > Apparently, if you are going intel, you can only acheive PCIe
>>> 3.0
>>> >> using
>>> >> > > at
>>> >> > > > least a Sandy Bridge-E or ivy bridge CPU in a socket 155. Please
>>> >> > correct
>>> >> > > me
>>> >> > > > if I have understood this incorrectly though.
>>> >> > > >
>>> >> > > > http://www.enthusiastpc.net/articles/00003/3.aspx
>>> >> > > >
>>> >> > > >
>>> >> > > > A socket 2011 proposition would be the Asus P9X79-E WES which
>>> has
>>> >>2x
>>> >> > PLX
>>> >> > > > PEX 8747 chips so can run at x16/x16/x16/x16 with four GPUs
>>> >> > > >
>>> >> > > > https://www.asus.com/Motherboards/P9X79E_WS/#specifications
>>> >> > > >
>>> >> > > > However, I'm unsure whether this is overkill for running 4xGPUs
>>> >>doing
>>> >> > > AMBER
>>> >> > > > serial code.
>>> >> > > >
>>> >> > > > What do you guys think?
>>> >> > > >
>>> >> > > > br,
>>> >> > > > g
>>> >> > > >
>>> >> > > >
>>> >> > > >
>>> >> > > > On 22 June 2013 16:15, Scott Le Grand <varelse2005.gmail.com>
>>> >>wrote:
>>> >> > > >
>>> >> > > > > Does this MB support full p2p at 16x PCIE Gen 3 speeds between
>>> >>all
>>> >> 4
>>> >> > > > GPUs?
>>> >> > > > > On Jun 21, 2013 4:09 PM, "Divi/GMAIL" <dvenkatlu.gmail.com>
>>> >>wrote:
>>> >> > > > >
>>> >> > > > > >
>>> >> > > > > > ET:
>>> >> > > > > > I am using GA-Z77X-UP7 that has PLX chipset and supports
>>> >>3rd
>>> >> Gen
>>> >> > > > > LGA1155
>>> >> > > > > > socket. Bought together with 2 TITANS sometime in March.
>>> >> > > > > > It has been running pretty stable 24/7 since then. I
>>> >>thought
>>> >> of
>>> >> > > > buying
>>> >> > > > > > two more titans later to fill all four slots. With so much
>>> >>mess
>>> >> > > going
>>> >> > > > on
>>> >> > > > > > with TITANS, I put off that plan until the dust settles.
>>> You
>>> >> might
>>> >> > > > want
>>> >> > > > > to
>>> >> > > > > > check new 4th GEN cpus and supporting motherboards as the
>>> >> Hardware
>>> >> > > keep
>>> >> > > > > > changing pretty rapidly these days.
>>> >> > > > > >
>>> >> > > > > > I have i5-processor with 16 GB ram and 256 GB SSD. All
>>> >>four
>>> >> > PCI-E
>>> >> > > > > lanes
>>> >> > > > > > are X-16. It also has native X-16 link directly "hardwired"
>>> to
>>> >> > > > CPU-lanes
>>> >> > > > > > that will bypass PLX chipset, in case if you run single GPU.
>>> >>This
>>> >> > > might
>>> >> > > > > > reduce a bit of latency but not much. I get 35ns/day on
>>> >>FIX/NVE
>>> >> > > > benchmark
>>> >> > > > > > bypassing PLX chipset, but get about 34ns/day using PLX
>>> >>chipset
>>> >> (on
>>> >> > > > TITAN
>>> >> > > > > > of
>>> >> > > > > > course!!). Not a deal breaker..
>>> >> > > > > >
>>> >> > > > > > Link below:
>>> >> > > > > >
>>> >> > > > > >
>>> >>http://www.gigabyte.com/products/product-page.aspx?pid=4334#ov
>>> >> > > > > >
>>> >> > > > > > HTH
>>> >> > > > > > Divi
>>> >> > > > > >
>>> >> > > > > > -----Original Message-----
>>> >> > > > > > From: ET
>>> >> > > > > > Sent: Thursday, June 20, 2013 8:18 PM
>>> >> > > > > > To: AMBER Mailing List
>>> >> > > > > > Subject: [AMBER] Anyone running machines with Quad GPU
>>> setups
>>> >> > > > > >
>>> >> > > > > > Hi all,
>>> >> > > > > >
>>> >> > > > > > I was looking at getting a new mobo to run a quad GPU
>>> system.
>>> >>I
>>> >> was
>>> >> > > > > > wondering if anyone has done this. If you could post the
>>> >>model &
>>> >> > make
>>> >> > > > of:
>>> >> > > > > >
>>> >> > > > > > 1) motherboard
>>> >> > > > > > 2) CPU
>>> >> > > > > > 3) RAM
>>> >> > > > > > 4) Case
>>> >> > > > > > 5) The aggregate estimate of ns in simulation you have run
>>> on
>>> >> your
>>> >> > > > setup
>>> >> > > > > > without issue,
>>> >> > > > > >
>>> >> > > > > > I would be much obliged! :)
>>> >> > > > > >
>>> >> > > > > > br,
>>> >> > > > > > g
>>> >> > > > > > _______________________________________________
>>> >> > > > > > AMBER mailing list
>>> >> > > > > > AMBER.ambermd.org
>>> >> > > > > > http://lists.ambermd.org/mailman/listinfo/amber
>>> >> > > > > >
>>> >> > > > > >
>>> >> > > > > > _______________________________________________
>>> >> > > > > > AMBER mailing list
>>> >> > > > > > AMBER.ambermd.org
>>> >> > > > > > http://lists.ambermd.org/mailman/listinfo/amber
>>> >> > > > > >
>>> >> > > > > _______________________________________________
>>> >> > > > > AMBER mailing list
>>> >> > > > > AMBER.ambermd.org
>>> >> > > > > http://lists.ambermd.org/mailman/listinfo/amber
>>> >> > > > >
>>> >> > > > _______________________________________________
>>> >> > > > AMBER mailing list
>>> >> > > > AMBER.ambermd.org
>>> >> > > > http://lists.ambermd.org/mailman/listinfo/amber
>>> >> > > >
>>> >> > > _______________________________________________
>>> >> > > AMBER mailing list
>>> >> > > AMBER.ambermd.org
>>> >> > > http://lists.ambermd.org/mailman/listinfo/amber
>>> >> > >
>>> >> > _______________________________________________
>>> >> > AMBER mailing list
>>> >> > AMBER.ambermd.org
>>> >> > http://lists.ambermd.org/mailman/listinfo/amber
>>> >> > _______________________________________________
>>> >> > AMBER mailing list
>>> >> > AMBER.ambermd.org
>>> >> > http://lists.ambermd.org/mailman/listinfo/amber
>>> >> >
>>> >> _______________________________________________
>>> >> AMBER mailing list
>>> >> AMBER.ambermd.org
>>> >> http://lists.ambermd.org/mailman/listinfo/amber
>>> >>
>>> >
>>> >
>>> >
>>> >--
>>> >-- - -
>>> >HK
>>> >
>>> >
>>> >════════════════════════════════════════════
>>> >Kevin E. Hauser, Ph.D. Candidate
>>> >NRSA Fellow, National Institutes of Health
>>> >Carlos Simmerling Laboratory
>>> >Miguel Garcia-Diaz Laboratory
>>> >100 Laufer Center for Physical and Quantitative Biology
>>> >Stony Brook, New York 11794-5252
>>> >Phone: (631) 632.5394 Email: 84hauser.gmail.com
>>> >════════════════════════════════════════════
>>> >
>>>
>>> >**************************************************************************
>>> >****
>>> >This e- mail message, including any attachments,
>>> >is for the sole use of the intended recipient(s) and may
>>> >contain confidential and privileged information.
>>> >Any unauthorized review, use, disclosure or distribution is prohibited.
>>> >If you are not the intended recipient, please contact the sender
>>> >by e-mail and destroy all copies of the original.
>>>
>>> >**************************************************************************
>>> >****
>>> >_______________________________________________
>>> >AMBER mailing list
>>> >AMBER.ambermd.org
>>> >http://lists.ambermd.org/mailman/listinfo/amber
>>>
>>>
>>> _______________________________________________
>>> AMBER mailing list
>>> AMBER.ambermd.org
>>> http://lists.ambermd.org/mailman/listinfo/amber
>>>
>>>
>>
>
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Mon Jun 24 2013 - 23:00:03 PDT
