Gpu memory question

I am very close to pulling the trigger and getting my dream Blender system-
I am considering a system built around 3-4 GeForce GTX 580 Hydrocopper 2 cards…
The reason for the hydrocopper 2 choice is that it is supposedly truly a single pcie slot card, so I dont lose lots to an oversized card.
The problem is that for some reason the 3Gb Hydrocopper 2 cards are only available used- no longer being made new. Do I lose much by buying the 1.5 Gb card instead?

If my motherboard has 7 PCIE slots could I go over 4 cards?

Anyone that has experience with this please let me know…

I don’t know if Nvidia even supports more than 4 cards. I’m just using one 580 classified right now and don’t even want to THINK about the fiasco your going to have to go through to power those suckers.

Well only the memory on one card is usable, so I presume as long as one has 3gb then you should be fine - presuming you can specify or in some way ensure the card with the most memory is the primary card; no idea if you can.

As they shouldn’t be in SLI mode for cycles, then maybe it’s possible to use more than four; good luck powering them though. :slight_smile:

  1. You can install as many cards as you want. If you got a mainboard with 12 PCIe slots, you can install 12 cards. If they are not all x16, don’t worry, I’ve done the numbers in this forum often enought that the performance impact for cycles is minimal. Only SLI has a limit, depending on the mainboard chipset, either dual, triple or quad.

  2. Random example: MSI Big Bang Marshal with 8 PCIe slots.

  3. Power is no issue. Just use Server PSUs with 2.5kW or attach a separate PSU for the graphic cards.

  4. Watercooling is a waste of money for this system, just buy PCIe expander cables. A 20cm x16 cable costs 5.70 bucks. You can just use it for every second and third slot and build a small framework with cheap acrylic sheets to hold the cards.
    I’ve also seen all cards attached with cables somewhere, and installed that the DVI connectors point upwards which makes sense in terms of heat transportation.

The system I would plug these into is here:

Except I may put in 2 1kW PSU and run the two Power supplies off of different house lines (old wiring in the place I live)

Your answer is VERY helpful (Thank you so much!) but doesn’t answer the most urgent question- How limiting is 1.5 Gb vs the 3Gb option? If I render smaller tile sizes could that impact the amount of memory needed? I have one Air cooled 580 classified with 3 Gb memory and not run into a memory issue in render so far (Though it has crashed on the full screen viewport render when certain objects have a lot of subdivisions)

Now your suggestion about using PCIE extenders could save me a LOT of cash… if you have experience with this I would like to talk with you about the details… I had looked into this before and seen reports that people had trouble doing this… I also worry about cooling with 3 or more air cooled systems in one box… (It will be noisy too, but I have a way around that being an issue where I live now)

Thank you everyone for your help!:evilgrin:

So, as a test I ordered a few different lengths of pcie 16x cables, I will use them in my current tiny computer with one GTX 580 to extend the card out and then see if it performs normally… if that works smoothly then I may forgo the water cooled cards and go the cheap route with extenders and air cooled cards… I just worry about putting 3 aircooled 580’s in one (albeit big) box…

Also, I contacted EVGA and asked about the power needs for multiple 580 cards, they said you need 600W for the first card (card+rest of system) then 300W for each additional card… soooo 3 cards would then seem to need 1200W supply…

There’s really no right answer here. I have a box with 4 watercooled GPUs at 1.5GB, and another box with a single 4GB card in case I can’t squeeze the scene into 1.5GB. I recommend trying some scenes that are typical for your use and measure their RAM usage.

There is however a very important trick that I use in the 4xGPU setup, and that it to make sure I use a fifth GPU for viewport - in my case the built-in GPU of the Intel i7. This will save you a lot of RAM and in my experience often makes the difference. This is also mentioned in the link above.

If you end up squeezing it all into one box without PCIe extenders/expanders, you’ll need watercooling of course. Just be aware that you will need some serious watercooling for 4GPUs. I built a setup for 4GPUs, and I had to expand it twice to get temperatures down to a reasonable level (I now have two pumps in serial as well as two radiator with additional fans attached). If you want to do this by air cooling/expander I should warn you that noise could get to a rather discomforting level. In my case so bad the wife complained it sounded like an airport from two rooms away :wink:

Oh, and about the PSU: I measured the load with 2xGTX-580 and 1xGTX-590 yesterday and it hit 900W totals sustained wattage. With peaks you can’t really be safe with anything less than a 1200W PSU. On the upside you’ll never be cold in the winter again :wink:

This will be my first foray into water cooling and I dont know much about it- but I do know it will be costly if I do it wrong…

I just got 3 used 3GB gtx 580 Hydrocopper 2 cards for about what new air cooled cards would cost- I will go with a 1500W or 1600 W PSU (OR possibly TWO 1kW power supplies run off of different house lines?)in case I want to add a 4th card. From what you are saying, it might be problematic to do more than 3 cards without doing serious mods to the cooling system…

This is the cooling system it sounds like I should get(?):

Dual pump, large reservoir…

No problem :slight_smile:

The difficulty with watercooling is that when you first start out, unless you buy a complete kit, all the terminology and stuff people simply expect you to know already can be a bit overwhelming. I have quite a few bits and pieces that I’ll never use due to them not fitting perfectly or other issues. You really should fire up the coffee maker and start reading a few warercooling FAQs - there are many of them out there.

The biggest problem is that most kits are geared towards the 1 CPU + 1 GPU combo, and with 3+ GPUs you’ll need a lot more than that. I live in Europe and I got in touch with Aquacomputer in Germany, asking enough questions to seriously test the limits of their very nice customer service reps’ patience, but in the end they helped me getting something that works.

In your case I’m worried the radiator is a bit too puny. My setup is geared towards a 3x580 + 1x590 setup, and I used their Aquaduct 360 eco+ external watercooling system and 12 V pump. This setup overheated so quickly I had to rush to shut it down. After discussions with client service I ended up adding another pump and this beast of a radiator. It has space for 9x12cm fans. So far I only put 3 fans on it, as it seems I’m finally keeping all the GPUs at a solid 60C (more or less) at sustained full load.

This might sound like overkill, but it really wasn’t. My first purchase was approx. the same as you suggest in your setup (3x12cm fans sized radiator), and it would’ve melted if I hadn’t stopped it in a hurry. So I think you need to scale up.

About pumps, the flow rate is less important than people think, but you need extra pumping power to overcome the flow resistance in your parts. Which means that if you were to buy the kit you suggest and then add another (external) radiator, it might make the pumps work too hard. I’ve also ordered a flowmeter. This is not because I need super fast flow, but because if for some reason the flow stops, I would like to stop the GPUs from creating a small bonfire in my PC case.

One final point: your fans and pumps are normally powered by your case PSU. I went and got myself an external power supply for the 360 eco, as when the system started heating up, it started sucking serious wattage out my PSU. Just something to be aware off. And remember that mechanical components like pump are more prone to produce wattage spikes (which will have nasty consequences if they peak above your PSU’s max), so be careful that you don’t move too far to your limit.

Hope this helps!