i believe yours is a different case than mine.
For mine, i am trying out the adaptive subdivision. I am addicted to crazy displacement map for the last few weeks and did a lot of experiment with it
adjusting dicing rate (adaptive sub-d) in 2.92beta too low will break the render and crash to desktop.
While in 2.91, i will see my gpu shared memory filled up and my system RAM grew to 30GB± and in worst case, system become unresponsive instead of crash, which is expected.
for the out of core, i did it in a crude way,
fire up my task manager, take note at ‘shared GPU memory’ stats just right after the render started.
for my scene, cycles crash when my ‘shared GPU memory’ started going up, in other words, the VRAM onboard is not enough and the scene started to load to system RAM.