CUDA error on GTX 970

Hi everyone
First of all I’m not entirely sure this is the right place for this question so if there are any problem let me know!
I’m having the classic CUDA error when I try to render on GPU a new character I’m making, as always no problem if I try with the CPU.

The strange thing is that in all the past projects I’ve rendered in GPU some really heavy scenes without problems (I almost always work with 4k textures, pbr, volumetric and subsurface shaders, different materials and million of polygons for the final renders of characters) but this time I’m trying to render a single character with only 6 materials and 24 4096x4096 png textures, no volumetric or SSS, only 51,000 faces, ultra simple lighting and even in the viewport render I got this error:
CUDA error: Out of memory in cuArrayCreate(&handle, &desc)

What I’m doing wrong this time?
When in the past I’ve got the CUDA error I was sure to have exagerate with the scene but this time I can’t really explain what is the problem, I’ve even tried to re-launch a render on a old heavy scenes with a character with 31 different materials and almost 3 different 4k textures each, heavy volumetric fog and sss for some materials and still got no problems with the GPU.
For this character instead I’ve yet searched for some problem in the topology/shaders, all the textures are made in Substance Painter but I haven’t got any problem in the export.
This is my pc spec:
OS:Windows 10(OS and blender launched on the main SSD)
RAM:16GB
Processor: Intel I7
GPU: MSI GTX 970
Blender version: 2.78
Thanks in advance for anyone would try to help me!

Hi.
Well, without seeing the scene others can only guess things.
Are you using GTX 970 to handle the display, right? Have you analyzed the vRAM usage with GPU-Z?
Open GPU-Z and you’ll see how vRAM is using Windows 10. You subtract that to your 4GB and that’s what you’ll have available for Cycles. So now you can also open the scene in Blender, put GPU-z in the foreground, and test the scene while you see the vRAM usage.
If your intel has iGPU, you can set your iGPU to handle the display, so you save the vRAM that Windows 10 uses on your GTX 970.

I’ve run into similar problems.

There are five things I can think of that may be producing this error:

  • the 31 textures you used before were 8-bit (or 16-bit) and the 24 this time are 32-bit, therefore taking up more GPU RAM; or it may be that the textures you’re using now are simply larger and you’re running out of RAM,
  • one of your meshes this time around is far more complex that you realize,
  • more mesh area is visible to the camera this time compared to the last,
  • One of your shaders has set up a race condition that garbage collection can’t keep up with, or
  • one or more of the meshes in your current file has caused an anomaly. If none of the first three seem to be the problem, this may help:

[LIST=|INDENT=2]

  • start a new file and append each element of the scene, doing test renders after each append, until you hit the error.
  • start another new scene and append each element again, starting with the last one you appended before the error occurred. If the error doesn’t crop up this time with that single element, work backwards through the list of elements you appended the first time until you hit the error again.
  • Restart from scratch a third time, and this time start by appending the last element from the first run and the last element from the second. If that mix of elements doesn’t cause the error, it may be that your entire scene is simply too big in some way that’s hard to track down.
    [/LIST]

Thanks to both of you for the response

-YAFU I’ve never use gpu-z before but I’ve downloaded it right now, very useful, thanks!
By the way the GPU even when give the CUDA error seems to never reach the maximum of memory usage apparently…

-rontarrant I’ve tried your suggestion of restarting multiple new file and append the various part of the scene, “apparently” the problem are the number of textures loaded, no matter what type of textures are (base color-rough-normal-metallic ecc…) when reached 13 textures I got the CUDA error, I’ve even tried with the lowest LOD of the model that have only 9,741 faces or replace all the shaders of the 6 materials with a simple diffuse.
The textures out of Substance Painter are all png 16 bit as always in my project.
At first was sure enough that the problem was the complexity of the shaders or something to do with the geometry but this thing of the textures seems too strange, I’ve even rechecked another model I’ve made months ago and have shared with a company and still work on GPU rendering even if have 21 materials and 51 full 4k textures!
PS:the model even with different LODs work fine and smooth in Substance Painter even with all the textures in 4k in the viewport.

The total of your texture file sizes must be the issue and I’m curious what that is compared to the other scene you mentioned. Despite how many textures you have or their 4k resolution, it just sounds to me like you’re somehow taking up more RAM.

And if that’s not the problem, you might check:

  • render settings (light path bounces, samples, etc.)
  • have you been trying micro-displacement with this newer scene? That crashes my GPU unless I carefully tweak levels of adaptive sub-divisions.
  • Load your texture files into Photoshop or GIMP and re-save them (in case SP has left some weird anomaly behind)

If it’s none of those things, I’m stumped.

Can the scene start rendering by showing a tile? Or does it give error before displaying a tile?
How close is that scene of filling the 4GB vRAM? (seeing vRAM usage from GPU-z while render). If you are close to 4GB, in the system there may be very fast peaks in memory usage that can trigger CUDA out of memory error.

Out of curiosity, what tiles size are you using? You are not using Denoiser feature in Blender from master/buildbot, right?

I try to respond at any of your questions:
-I’ve compared the two textures folders of the new and old characters: the new one is 510mb and the old one is 2,49gb
-regarding the render settings, I almost always left everything as default the only things I change for the differents situations are the samples (both at 1000) and the render tiles using the auto tile size (obviously different between the projects using different render hardware)
-no displacement at all, in any of the two scenes
-For the moment I’ve re-saved all the textures with Photoshop, later I will try out if omething change (for the moment the only difference is that all the textures now are 590mb)

-No tile shown, but I remember that blender shows a tile and then failed the render when I tried time ago with a very heavy scene (that I was sure was too heavy for my GPU, only for the sake of test).
-Now there are a difference in the memry used by GPU, when in default mode there are 1695mb used and when I try to render go up to 3809mb, I don’t know why this difference, obviously only with blender and GPU-z opened.
-Tile size in GPU are 240x256 for 2160x3840 pixels of render resolution (I still got the CUDA error even at very low render resolutions.
-No I’m not using the Denoiser, only thing changed is the Filmic in color managment but I’ve lready tried the scene in two different versions of blender 2.78

For rontarrant: I’ve tried the re-saved textures from Photoshop but still have the CUDA error…

I’m out of ideas. Sorry.

As I had said before, without being able to see or know anything about the scene, people can only guess things. It is up to you to do tests and monitor vRAM usage on GPU-z while rendering. Testing by removing objects from one at a time, or lowering levels of subdivision into subdivision surface or multiresolution modifiers. You have “Simplify” item in Scene tab to globally do those things (to the right, Render part).

When you start rendering is when vRAM starts to be filled. I guess that of those 1695mb, most memory corresponds to Windows display part.
So you have just over 2GB of vRAM available for cycles. You can go back to Windows 7 or use some version of Linux to reduce use of vRAM. Also keep in mind what I said about using intel iGPU to handle the display.

By the way, I’m not sure if I understand what you mean by Default Mode.

Your textures are in a compressed format - that’s why you do not see the size they take in GPU memory when you check the size of the folder.

One 4k 16bit texture is 96MB. So 24 x 96 = 2304MB. Are you using HDRI lighting? 32 bit high resolution HDRI can easily be extra 500MB. So you are also using the same GPU for work, maybe you enable the nice Material mode in the viewport as well…
It is not surprising you are runnig out of memory.

For GPU rendering you must think of the texture size. 4k 16 bit textures are not going to be a good choice with GTX 970. I wish it was - I use the same GTX970 for rendering myself, but 4GB is just not enough for that. Do all of the texture have to be 4k? Do they all have to be 16 bit? I know the answer already. No. It might be usefull to convert your black and white textures like roughness or bump maps to greyscale - 3 times less data instantly without any loss. You should consider using another GPU for the monitor as well - does the computer have integrated graphics card? Then think of what can be reduced.

If you are just working with GTX970 with another rendering solution in mind that is able to hande the high resolution textures, you can grab yourself a daily build, and use Simplify functionality that will reduce the size of the textures for you while you work. https://s21.postimg.org/cmrw7yzqf/Capture.png

Sorry in the previous messages I totally forgot to ask you this, what is iGPU? I’ve searched on google but I admit to not completely understand if has something to do with an Intel integrated GPU and how can I find if my CPU have it (rookie question I know…)
Regarding the default mode in viewport, obiviously was a my mistake, I was referring to the Solid mode, my fault sorry!

What cpu do you have ? You can use this free software to check it : https://www.piriform.com/speccy

Thank you very much for your response, it’s very informative! I’ll try to not ask you too many question and answer to everything.
-Didn’t know about the real size of the 16bit texture, thanks for the tip!
-In viewport i try to work only in solid and texture mode when I can, rendered only for trying the various light setup (for nice material mode do you reffer to the material viewport shading?)
-Regarding the texture size obiviously I agree with you, not everything must be in full 4k, I’ll be totally onest I almost always skip the “optimization” part of my scene for a matter of time, really big mistake I know but until now I never have problems whit my current pc setup even if I already know isn’t top tier, not knowing the real size of my files texture I was curious to know what was sucking up all my GPU memory :slight_smile:
-I have tried the Simplify functionality of a daily build I was trying for the new principled shader and the GPU rendering go without problem on a lower resolution, didn’t know this functionality, very useful thanks!
-Last important question, you said “You should consider using another GPU for the monitor as well”, for the moment I have only the 970 installed but I’m still have another MSI 750Ti unused at home, can I mount it with the 970 and using it only for the rendering in blender? Can I have some benefit from it or will simply not work?

Intel Core i7 4790 specifically, thanks for the software’s link, can be useful!


You can try to use the gtx 750 ti for only your monitor and use the gtx 970 only for the render :slight_smile:

And this is why we need technically-minded, analytical minds as part of this community. Artists just don’t think of these things.

Well spoken, Martin. This should give the OP a real leg up to solving this.

Yes, some intel CPUs have integrated graphics (iGPU). CPU-z (not GPU-z) is the software to read CPU information. If your CPU have iGPU, you must configure the BIOS to select iGPU as the main display and connect your monitor to one of the outputs of the motherboard.
But I did not know you had another nvidia available. This is better. Connect your monitor to the 750ti and set it to handle the display. Then from Blender do what Benny G has explained to you.

Edit:
This screenshot corresponds to Blender from master/buildbot (or 2.79 Testbuild). In 2.78c this may appear a bit different, not checkboxes.
Download the one that corresponds from the first links, ‘2.78’ should appear in filename:

Edit 2:
And anyway you keep in mind the good practices that people have explained to you so that rendering does not take much vRAM.

Thanks for the tip, if for some reason I can’t use the other Nvidia I have I’ll try it thanks!
However the 970 is already set in the system preference of blender from time ago, no problem :slight_smile: