I think this is a engine limitation. Cycles render noisy shadows in image output. To fix that enable Denoising Data at your View Layers Properties, them Denoise the alpha channel and use as a fac input to a Color Mix Node and set to Multiply, first color will be your background (also works using images). Denoising the actual shadow RGBA pass doesn’t work.
I also had a similar issue with quality using shadow catcher: Shadow Precision Issue - Cycles