Why does my frame look worse the longer it renders? (Cycles)

I have a strange problem that I can’t figure out for the life of me.
I thought that generally in cycles, the longer you let it render, the better it looks.
I’m having strange behavior on mine though.

This picture was after only 3 Path Tracing Samples.

This picture however was when it had completed all 32 samples.

Does anyone have any insight as to why it’s doing this?

Might be some caustic effect. Those need forever and then some to converge in Cycles, so they might actually become more and more prominent the longer the render time.

Try disabling refractive and reflective caustics in the render tab.

That is a specular highlight. It’s a reflection of the sun light or hdri you are using. Those sorts of reflections take a few more samples to converge. 32 samples isn’t much for a path tracer, try giving it 128 samples and see how that looks.