GPU memory management issue when rendering in cylces.

I’ve run into an issue trying to render out a very simple scene using the GPU. I can’t seem to render many frames before the GPU memory gets full and the process slows way down, to eventually fail.
The scene is very roughly modeled and not too complex, has simple textures, not that many polygons, and a few rigid body objects. No geometry transformations or light changes during the scene, just a simple camera move.


If I look at the graph of the GPU usage I can see how on very frame that gets rendered the memory usage goes up. Once it reaches the end of the available memory it keeps rendering, but at a much slower pace in which it takes a while to prepare the scene and then renders. This cycle goes on for a few more frames an then eihter it stops rendering completely or starts saving black frames with no information…

Here’s a screen grab the GPU usage, (the yellow and red numbers are the frames being rendered):



It shows the progression of memory usage and how the gpu is being used. You can clearly see at which point the memory saturates and the rendering start to slow down. The actual rendering of the frame corresponds with the flat lines at 99% fo the GPU usage. The more erratic lines with less than 40% usage show just before the frame starts being drawn on the image editor window.

Strange thing is that if I quit blender and render again, starting from the point at witch it failed, It will render those frames very fast again until the memory fills up.

I’ve tried changing the tile sizes, BVH cache settings, memory cache limits, but the result is the same everytime. I can only render a few frames at the time.

Is there a way to instruct blender to flush the memory from the frames already rendered and saved?
Any other setting I should try or thath I’m completely overlooking?

Using Blender v2.69. The system is running 2 GTX780 with 3 GB of memory, i7 930 w 24 GB ram…
I’ve rendered way more complex animations on this system with no errors and this one is giving me headaches.

Attachments

washer.blend (1.38 MB)

Cant reproduce here using a buildbot version… works as expected…

Do you run into the same problem if you are running just one GPU instead of 2/3

Does this problem happen on different scenes

does this problem happen with different versions of blender? try a buildbot version – http://builder.blender.org

I did try rendering on just one GPU and the results were the same (only twice the time to render and fail).

I’ll try the build bot.

I just tried with the buildbot version (for windows 64). same result: By the eight frame the memory fills up and the whole thing slows down…


SOLVED!
After Uninstalling and Reinstalling NVida Drivers, reinstalling blender from Graphicall.org things were not going any better. So I reloaded the factory settings in blender and that did it! The memory is released after each frame.

Normal behavior again:

I still don’t know what caused the Issue (If someone does please feel free elaborate), but now I’m a happy blenderer again.
:slight_smile:

both Cuda and OpenCL are driver-based technologies, Cuda is also a closed-source, proprietary, framework; in a nutshell it’s impossible to detect and debug your application to the last bit of execution.

There is nothing to say, it can happen anytime and even a developer can’t do that much to investigate or fix the issue.