Multiple Importance Sampling in World settings = Cycles GPU render fail

Hi All – I’m seeing strange behavior from Blender recently that hopefully someone can shed some light upon…When I enable Multiple Importance Sampling (MIS) in Blender World settings, Cycles GPU rendering fails.

Issue reproduction steps:

  1. Open Blender 2.70 default scene
  2. Switch to Cycles render engine
  3. Switch from CPU to GPU Compute in Scene Render settings
  4. Press Shift-Z – Render of cube works OK
  5. Switch to World settings and check Multiple Importance Sampling under Settings
  6. Press Shift-Z, render fails with the following Console output:

[SUB]CUDA error: Launch failed incuCtxSynchronize()[/SUB]
[SUB]Refer to the Cycles GPU rendering documentation for possible solutions:[/SUB]
[SUB]http://wiki.blender.org/index.php/Doc:2.6/Manual/Render/Cycles/GPU_Rendering[/SUB]
[SUB]CUDA error: Launch failed incuMemcpyDtoH((uchar*)mem.data_pointer + offset,(CUdeviceptr)((uchar*)mem.device_pointer + offset), size)[/SUB]
[SUB]CUDA error: Launch failed incuMemFree(cuda_device_ptr(mem.device_pointer))[/SUB]
[SUB]CUDA error: Launch failed incuMemFree(cuda_device_ptr(mem.device_pointer))[/SUB]
[SUB]CUDA error: Launch failed incuMemAlloc(&device_pointer, size)[/SUB]
[SUB]CUDA error: Launch failed incuModuleGetGlobal(&cumem, &cubytes, cuModule, name)[/SUB]
[SUB]CUDA error: Launch failed incuMemcpyHtoD(cumem, (void*)&ptr, cubytes)[/SUB]
[SUB]CUDA error: Launch failed incuMemAlloc(&device_pointer, size)[/SUB]
[SUB]CUDA error: Launch failed incuModuleGetGlobal(&cumem, &cubytes, cuModule, name)[/SUB]
[SUB]CUDA error: Launch failed incuMemcpyHtoD(cumem, (void*)&ptr, cubytes)[/SUB]
[SUB]CUDA error: Launch failed incuMemAlloc(&device_pointer, size)[/SUB]
[SUB]CUDA error: Launch failed incuModuleGetGlobal(&cumem, &cubytes, cuModule, name)[/SUB]
[SUB]CUDA error: Launch failed incuMemcpyHtoD(cumem, (void*)&ptr, cubytes)[/SUB]
[SUB]CUDA error: Launch failed incuMemAlloc(&device_pointer, size)[/SUB]
[SUB]CUDA error: Launch failed incuModuleGetGlobal(&cumem, &cubytes, cuModule, name)[/SUB]
[SUB]CUDA error: Launch failed incuMemcpyHtoD(cumem, (void*)&ptr, cubytes)[/SUB]
[SUB]CUDA error: Launch failed incuMemAlloc(&device_pointer, size)[/SUB]
[SUB]CUDA error: Launch failed incuModuleGetGlobal(&cumem, &cubytes, cuModule, name)[/SUB]
[SUB]CUDA error: Launch failed incuMemcpyHtoD(cumem, (void*)&ptr, cubytes)[/SUB]
[SUB]CUDA error: Launch failed incuModuleGetGlobal(&mem, &bytes, cuModule, name)[/SUB]
[SUB]CUDA error: Launch failed incuMemcpyHtoD(mem, host, size)[/SUB]

If I swap steps 5 and 4 above (perform first render with MIS World setting enabled), GPU render works OK.I’m willing to raise a bug in the tracker but I would like for someone to confirm that this IS a bug so I don’t waste anybody’s time.

My system configuration:
Blender 2.70 64 bit (I’m seeing the same on 2.69)
Windows 8.1 64 bit
NVIDIA GeForce GTX Titan x 3 (SLI is disabled)
NVIDIA Driver 335.23 (latest as of this post)

ANY insight is greatly appreciated.

Strange… I get the same error if I enable 3 graphics cards. If I go with 2 GPUs the error is gone… (no error if I go with just one GPU either)

cegaton - I’m relieved to hear it’s not just my system. I do not see the issue either if I enable only one graphics card via User Preferences. The issue appears when Blender is configured for multiple GPUs. What is your OS and driver version?

I’m on blender 2.7
Linux Mint 16 64bit
driver version 319.32

I found that lowering the map resolution form the default 256, makes The Multiple Importance sample work again with all 3 card enabled. The max size I’ve got is 180…

If you use GPU-z. how much ram is each card using?

I’m seeing the same, map resolution at 180 or lower allows render with 3 cards selected. 181 or higher produces CUDA error.

During CUDA fail using repro steps above GPU-Z reports video card memory used:
GPU1: 1707 MB
GPU2: 0 MB
GPU3: 0 MB

This is entered into the tracker: https://developer.blender.org/T39459

cegaton - Running the repro steps above with SLI enabled produced a hard lockup of my system (first I’ve seen). If you’re feeling curious and don’t mind a potential system reset (after saving work), let me know if you see that also. I’d be impressed if the issue could kill Linux Mint.

Sorry, I don’t have the connector for SLI, plus only two of my GPUs are the same kind.
As far as I know SLI is not recommended for blender.

how much memory is used when the map resolution is at 180

GPU-Z video card memory usage after Control-Z render and MIS Map Resolution at 180:

GPU1 “Memory Used”: 1851 MB
GPU2
“Memory Usage (Dedicated)”: 0 MB
“Memory Usage (Dynamic)”: 703
GPU3:
“Memory Usage (Dedicated)”: 0 MB
“Memory Usage (Dynamic)”: 665 MB

I’m not sure why GPU-Z is displaying memory usage differently (e.g. Used vs Usage Dedicated/Dynamic) for that first card.

cegaton - Correct, SLI slows Cycles. I use it for gaming and enabled it only for troubleshooting.

For posterity - This bug appears to have been fixed via duplicate bug https://developer.blender.org/T40228
I’ve tested with BuildBot blender-2.71-b49e6d0-win64.zip and I’m no longer seeing the issue. Props to bug fixing Blender devs.