Rendering at bed time.

I wonder if anyone has this problem, or any ideas for a solution.

To get around my crashing problems I now have the side off the pc and a desk fan blowing air at the 3 GPU’s. This is keeping the temperature down a bit, although card 2 reports running at its limit around 50% of the time. I’m going to swap the card positions today and see if the problem is the same, or if I have a card that’s running hotter than the others.

However, that’s not the problem.

Every evening I start my animation render, it’s getting through a frame every 2 and half minutes so it’s fairly long winded. I sit here for a while watching the temperatures rise and fall, then settle down to a steady level and it’s fine. I can check back in a couple of hours and everything is still running great. However, the moment I decide it looks stable and go to bed, the bloody thing crashes!

Suggestions on a post card please!

a)
What kind of graphic cards are those? Some graphics card vendors use (especially on factory-overclocked cards) non-reference air cooling (“TwinFrozr”, “Windforce” etc.) which is fine for single card setups and is likely to be more silent, but is not ideal when installing more than one card, because the cards severely block each other’s airflow and the hot air is blown back into the casing.
For multiple card setups it’s better to stay with cards in NVidia’s reference design for the air cooling, which is geared towards blowing the hot air out of the casing.

So, swapping the cards’ positions might gain nothing.

b)
Get a decent cooling solution for your machine. And no, having the case open with some external ventilation does not qualify as “decent”. If the cards get so hot that the machine crashes, you will significantly lower the lifetime of all overheating components.

c)
If your machine can’t handle the strain of rendering animations, have you ever thought of using a commercial render farm service? Might save you from getting crazy over you crash problems.

Underclock each of the cards a bit.

If you are monitoring using MSI Afterburner you should also check the power usage. Does your power supply have enough watts to drive three cards at full load?

make sure there’s nothing in the scene that’s making it crash, I had a problem with a scene crashing cause I had too many hair particles on a character and it would render some frames but choose to crash on a random frame. If it’s not the scene than make sure your power supply is able to support 3 gpu’s. You probably need something over 1000watts (though I’m not exactly sure on this) and if you can, put it as close to an AC vent as possible to keep it cool.

Thanks folks.

The cards are GTX780ti’s, as mentioned not perfect in this setup as the do vent all their heat straight onto the next one in line. I chose them simply for the cuda count. They’re very good, I just seed to sort out the cooling.

No, case off and desk fan pointed inwards is not at all the ideal solution!! I am investigating a water cooled setup, costly, but it may well be the way forward until I can afford a crazy 8 gpu board, at which point I’d probably go for the K series cards.

The power supply is a good 1200w unit, that’s certainly not the problem.

I have looked at render farms, but the cost is actually pretty high. I worked out my two longish animations so far would have cost as much as one GPU, and I’m only doing this as a hobby to be honest. I don’t like the idea of the community based render farms, I don’t really want tiles/frames of my work being thrown all around the globe out of my control!

Thanks again for the suggestions.