Diffuse baking artifacts?

I’m baking the purple object onto the green, but the result is wacky. I can clearly see the objects correctly baked on the image, but then there’s additional color just everywhere. I’ve tried this on multiple versions of blender, I’ve applied scale/etc, I’ve messed with ray distance and modifiers (both objects currently have none.) and nothing works.

I’m trying to bake these tassets onto an object for cloth physics, instead of each one being its own object.
Pls help.