Building a super computer.

[plutoniumman]

Hey all

I'm considering building a small super computer. I plan on starting with an empty cabinet and then purchasing servers as I can afford them. The purpose of the cluster will be to render 3D models and to simulate physics.

So far I've got a pretty good idea on everything, except:
1. Cooling. Anyone have any ideas or suggestions on how to keep a cabinet full of servers cool in a home environment? ...In Arizona? Maybe I just won't use it much in summer lol. I suppose I should purchase a dedicated AC unit?

2. Electrical safety. I know the circuit breakers in the typical home supports around 2,000 watts before the circuit breaker is tripped. Would it be unwise to hire an electrician to increase that limit to 4kW (only for the servers' circuit) in a home environment?

3. Physical security. How to deter someone from stealing it? I'm not worried someone would insert a malicious flash drive to give it viruses or steal data (are they gonna steal my models & textures? lol). I'm more concerned someone breaking in, and stealing the hardware just because it looks expensive or something (there's a ton of hoodlum scum moving into my area). I suppose I could have the cabinet bolted to the floor, with the nuts welded to the bolts, and have each server locked into the cabinet. Your thoughts?



Is there anything else you all think I should probly know before trying this? For now it's just an idea Won't actually be buying anything for a while. Want to learn more about it.

 

[byteninja2]

Hey all

I'm considering building a small super computer. I plan on starting with an empty cabinet and then purchasing servers as I can afford them. The purpose of the cluster will be to render 3D models and to simulate physics.

So far I've got a pretty good idea on everything, except:
1. Cooling. Anyone have any ideas or suggestions on how to keep a cabinet full of servers cool in a home environment? ...In Arizona? Maybe I just won't use it much in summer lol. I suppose I should purchase a dedicated AC unit?

2. Electrical safety. I know the circuit breakers in the typical home supports around 2,000 watts before the circuit breaker is tripped. Would it be unwise to hire an electrician to increase that limit to 4kW (only for the servers' circuit) in a home environment?

3. Physical security. How to deter someone from stealing it? I'm not worried someone would insert a malicious flash drive to give it viruses or steal data (are they gonna steal my models & textures? lol). I'm more concerned someone breaking in, and stealing the hardware just because it looks expensive or something (there's a ton of hoodlum scum moving into my area). I suppose I could have the cabinet bolted to the floor, with the nuts welded to the bolts, and have each server locked into the cabinet. Your thoughts?



Is there anything else you all think I should probly know before trying this? For now it's just an idea Won't actually be buying anything for a while. Want to learn more about it.

Wow, must be a big buiessness. Security wise, I would have a dedicated room for servers, or the master bedroom, with some drawers with rackmount servers, and padlock the drawers.

 

[MyCattMaxx]

Not sure what your end plan is but this would be a good start:
4 x Opteron 6174
H8QGi+-F Supermicro motherboard
16 x 2GB 1333MHZ Cas 7 G.skill Ripjaws
4 x Hyper 212+ G34 mounting Mod

And it draws about 580 watts.

I saw a server A/C unit on CL just about a week ago but it was in Ohio.

 

[claptonman]

Just liquid cool it.

As for security, I assume you'll have to custom make a case for it. Just chain it down with high-grade chains.

 

[plutoniumman]

Thanks for the replies everyone

@ byteninja2:
No huge business here. Used servers can be purchased for fairly cheap. At the moment I'm looking at a server with 2 quad core Intel Xeons @ 2.33 Ghz, 8 GB RAM, and 3 10k RPM HDDs for $800. If I keep finding deals like this (one comes up every few weeks, often in bulk & cheaper per-unit price), I'll have quite a bit of CPU power in under a year; and cost less than a decent car.

What did you mean by "...with some drawers with rackmount servers"? Like installing an enclosed drawer on the rack and keeping the server inside it?

@ MyCattMaxx:
Thanks for the suggestion but each one of those CPUs is $1,200 :O I'd like to keep the cost for each server under $2,000. That way if something goes wrong with one of them (like if I drop it lol), it'll be less than 2,000 dollars wasted instead of like $8,000. Though it's tempting to go that route for the power efficiency & great space/core ratio.

@ claptonman:
I didn't know it was possible to use liquid cooling in servers. I just looked it up, and found that it is! Unfortunately dumping 4kW into a room will dramatically increase the ambient temperature, making it too hot for the computers to operate (even with liquid cooling). Not to mention all the liquid cooling solutions I found after a quick search would add $1,000 minimum per server. After 3 servers, it'd be cheaper to buy an AC unit intended for a whole house, dedicated to that one room. ...Yes I thought out how to cool the things a little better since my original post lol

Not a bad idea with the chains. Though wouldn't it be fairly easy to take bolt cutters and cut the chains? With my idea of bolting the cabinet to the floor, it'd be very difficult to cut the bolts because they'd be inaccessible.

 

[byteninja2]

Thanks for the replies everyone

@ byteninja2:
No huge business here. Used servers can be purchased for fairly cheap. At the moment I'm looking at a server with 2 quad core Intel Xeons @ 2.33 Ghz, 8 GB RAM, and 3 10k RPM HDDs for $800. If I keep finding deals like this (one comes up every few weeks, often in bulk & cheaper per-unit price), I'll have quite a bit of CPU power in under a year; and cost less than a decent car.

What did you mean by "...with some drawers with rackmount servers"? Like installing an enclosed drawer on the rack and keeping the server inside it?

@ MyCattMaxx:
Thanks for the suggestion but each one of those CPUs is $1,200 :O I'd like to keep the cost for each server under $2,000. That way if something goes wrong with one of them (like if I drop it lol), it'll be less than 2,000 dollars wasted instead of like $8,000. Though it's tempting to go that route for the power efficiency & great space/core ratio.

@ claptonman:
I didn't know it was possible to use liquid cooling in servers. I just looked it up, and found that it is! Unfortunately dumping 4kW into a room will dramatically increase the ambient temperature, making it too hot for the computers to operate (even with liquid cooling). Not to mention all the liquid cooling solutions I found after a quick search would add $1,000 minimum per server. After 3 servers, it'd be cheaper to buy an AC unit intended for a whole house, dedicated to that one room. ...Yes I thought out how to cool the things a little better since my original post lol

Not a bad idea with the chains. Though wouldn't it be fairly easy to take bolt cutters and cut the chains? With my idea of bolting the cabinet to the floor, it'd be very difficult to cut the bolts because they'd be inaccessible.

What did you mean by "...with some drawers with rackmount servers"? Like installing an enclosed drawer on the rack and keeping the server inside it?
Yes.

Also, build your own server. You really, really dont need a big server for a small buisness. And, a single xeon would be great unless you are pixar, and save you a lot of cash in cooling, motherboard price, and cpu price.

 

[spirit]

No huge business here. Used servers can be purchased for fairly cheap. At the moment I'm looking at a server with 2 quad core Intel Xeons @ 2.33 Ghz, 8 GB RAM, and 3 10k RPM HDDs for $800. If I keep finding deals like this (one comes up every few weeks, often in bulk & cheaper per-unit price),

That looks like a good server you've found there, $800 for a server like that is pretty good actually. If you get something like that you may want to upgrade the RAM to 16GB though, we had 8GB of RAM in our Exchange server at home and it was a it sluggish, so we upgraded to 16GB and it's been better.

 

[byteninja2]

That looks like a good server you've found there, $800 for a server like that is pretty good actually. If you get something like that you may want to upgrade the RAM to 16GB though, we had 8GB of RAM in our Exchange server at home and it was a it sluggish, so we upgraded to 16GB and it's been better.

Vistakid, im not sure if you meant this, but he is doing it for a 3D rendering server, not exchange. But, yes, this is one of those times were you need more than 8 GB ram, and for lots of 3D rendering, even more than 16, maybe 32.

 

[spirit]

Yeah I just saying in my family's Exchange Server we noticed 8GB to be a bit slow, so generally I'd recommend 16GB+ RAM for a server, and as you said, yeah 16GB or even 32GB for rendering would be good.

 

[plutoniumman]

Just to be clear, the servers will actually be the render nodes. The 3D models are created on the workstation, and then submitted to the manager. The manager assigns jobs to available servers to begin rendering.

For example: Say I need to render an animation which consists of 100 frames. If I have 50 servers, each server would be assigned 2 frames for rendering. In practice it actually works slightly different than that, but you get the idea.

The same concept can work similarly for single frame images. The manager splits the images into strips, and then assigns servers to begin rendering those portions of the image. It's later stitched together into one image.


To give an idea how long some of this stuff takes: I've once simulated water splashing into a container. The physics simulation (not including simulation of vorticity or foam/bubbles) was about 2 days for 5 seconds of video. Render time was an addition 4.5 days. This was on a 4 Ghz Quad core AMD Phenom II 955 (OCd from 3.2 Ghz) with 8 GB RAM. Up to 4 other computers were able to assist when they were otherwise idle. The other computers were Core 2 Duos ranging from 2 Ghz to 2.66 Ghz.

For modeling and setting the scene, those computers are plenty. When it comes time to rendering photorealistic video is when there's a problem (images are usually finished in a day or two). Which is why I want to start assembling a cluster of servers; to offload the work from the desktops, and possibly finish more than 5 seconds of realistic video in less than a week.