HPC build...

[Airbozo]

Not a true mod, but my latest creation. I will get pics up soon.

Specs:
32x Xeon 5520's
40x Xeon 5570's

1728 GB DDR3 memory

216TB disk space

4x 4gb Tesla GPU

1x melanox Infiniband switch

1x GigE switch

1x 10/100 switch (for IPMI only)

I have all the systems built and racked. I am waiting on the intelligent power units to show up before I can start cabling. The cabling will be a nightmare since each node will have: one power cord, one Infiniband cable, one IPMI cord and one network cord. All but 4 of the nodes are 2U quad twin nodes meaning in 2U of rack space there are 4 complete systems. Those nodes are serviceable from the back and my cabling has to allow them to remove one node at a time for service (and they are hot swappable).

More info and pictures soon.

[artoodeeto]

whoa! can you send me one so I can play Pong on it? :D

[Airbozo]

whoa! can you send me one so I can play Pong on it? :D

Pong would play so fast on this thing, the game would be over before you clicked start...

I am checking now if I can post who this rack is for.

Robotics much?

[Zephik]

That's ridiculous. How big is that setup and how much did it cost?

[Airbozo]

All the systems including spaces for air intake between the 1U Tesla systems, fit into 28U of cabinet space. That also includes 4U for some UPS's.

The cost is somewhere over $200k... Not sure of the exact cost since I never see the numbers and usually don't want to. This is going to be running the ROCKS cluster software and the systems will be split into 2 different HPC's.

Hey I may finally finish my ROCKS Cluster tutorial...

Once I finish my rack diagrams I will post those up with pictures. It kind of looks boring since they all look similar and everything is packed tight. Once I start cabling it will really look cool.

BTW: The PDU units are IP addressable and you can shut individual ports off, schedule times when they come on, measure the current across each module and keep track of usage to bill each department... Really cool units. They even have temperature probes...

[x88x]

O_O Holy flying monkeys Batman! I read the thread title and thought you misspelled HTPC. :P

That is really awesome. Are you allowed to take pics?

[Airbozo]

O_O Holy flying monkeys Batman! I read the thread title and thought you misspelled HTPC. :P

That is really awesome. Are you allowed to take pics?

Hehe, that would be cool. This rack has enough power to deliver hundreds of movie streams...

I can take pics and am documenting the build now so my team mates can build the next few. I will post them up soon. I have not started the cabling yet and that is the tough part...

[Trace]

<3 mind sending me 2 5570s? :)

[x88x]

so my team mates can build the next few.

...sorry, what was that? I think my brain just exploded. :D

[p0Pe]

text awsomeness! now gief picture awsomnezzzzneezz!!!

[billygoat333]

I want your job. haha

[Airbozo]

I want your job. haha

We are hiring...

[billygoat333]

what qualifications do you have to have? lol I am about 1/2 way 3/4 through my CIS degree :P

[Kayin]

If you'll move me out there, I'll come work for you. No lie.

But, I have to think of this as just trying to top the Bat.

I'm rolling out a new mainboard since I lost my MSI, this comes with IPMI 2.0 and a host of other goodies...

[Airbozo]

Small update and some teaser shots...

One of the nodes in the 2U systems (there are 4 in each 2U chassis).

Yes those are 4gb dimms...
http://img262.imageshack.us/img262/7397/willowgarage1u013.jpg (http://img262.imageshack.us/i/willowgarage1u013.jpg/)

Starting the cabling...
http://img402.imageshack.us/img402/6485/willowgarage1u014.jpg (http://img402.imageshack.us/i/willowgarage1u014.jpg/)

[Trace]

I hate you!
When can I come see that in person?

[artoodeeto]

actually, for me, while tempting to have a machine like that, it'd be cool, but there's no way I'd ever need or even use the whole thing...it's kinda like having the full library of Alexandria.

[x88x]

Heheh, veery nice. I love how some manufacturers have started moving over to units like that; cut out all the unnecessary cruft, and you're left with really a pretty tiny thing.

[Airbozo]

I hate you!
When can I come see that in person?

Well, they moved my ship date to tomorrow...

That means I have to finish the cabling, install the OS and run burn in tests. Oh yeah, I still have labels to create for each end of every cable (indicating where it plugs in and where the other end plugs in, on each end...).

Looks like I will be pulling a late night tonight.


EDIT: Oh and don't be hatin' on me, I am just the builder. I don't actually get to play with this thing much. I was hoping to see what sort of PPD's I could get folding on this beast.

[Airbozo]

Heheh, veery nice. I love how some manufacturers have started moving over to units like that; cut out all the unnecessary cruft, and you're left with really a pretty tiny thing.

My main issue with the 2U quad node system is the power supply's. Right down the middle, hence the power cables running up the middle of the rack. I have to cable this thing so they can remove any node out the back without removing other cables. The other issue is the plastic airflow baffles they put in these things. They NEVER go back in properly and I have already mangled 2 of them to uselessness.

[Kayin]

God bless you. I remember those.

Pretty sick stuff, I like it.

[Airbozo]

Another cable update:

http://img153.imageshack.us/img153/8353/willowgarage1u019.jpg (http://img153.imageshack.us/i/willowgarage1u019.jpg/)

Next up?
Infiniband cables. This is the second hardest part of this build. These cables are thick and have specific radius tolerances that I must adhere to. Now to find out what they are. ;)

I am still hoping to power this rack up tonight.

[SXRguyinMA]

holy crap! I cant imagine the weight of that thing

[Trace]

Good luck dude, keep us posted!

[Airbozo]

OK, I came up short on my Infiniband cables, but I just did the smoke test and all but 2 of the nodes came online. I have to RMA some parts first thing in the morning and get 4 more cables and this thing will be ready for the OS...

I am beat. 14 hours today and counting.

[Luke122]

Slick!

[Kayin]

fscking sexy.

So, where is it you say you work? Just got laid off, could use a better job...

[Airbozo]

fscking sexy.

So, where is it you say you work? Just got laid off, could use a better job...

mce.com

Little company, mostly good people here, but the pay IS sub-par.

But hey, I get to play with some cool stuff and they even let me wear shorts and bring my dog to work... (they did get a little pissed about me marking up the nice shiny floors with my skateboard)


EDIT: BTW: I am loading ROCKS on this beast now.

[Trace]

More pictures?

[Airbozo]

More pictures?

Working on it.

I am starting to put all the skins and blanking panels on to make it pretty!

[Kayin]

Be nice if I didn't live in Alabama...

That's a serious setup, but why no Windows HPC client? That's been gaining a lot of ground... I'm sure it's dependent on workload, though. What is ROCKS again, anyway?