Hey, fun concept, let me give a few pointers I’ve learnt from working on a feature film with blender.
Sampling
I’ve done many many tests in this area, the best place I’ve found to start is with “Branched Path Tracing”, “Path Tracing” while faster if you are using the same amount of samples, is significantly noisier than “Branched Path Tracing”. Example, one render with your scene, “Branched Path Tracing” at 32 squared samples (I prefer using squared samples, but not really necessary), took about 1 min 41 seconds, “Path Tracing” at the same samples only took 46 seconds but looks incredibly noisy, even if I double the samples to 64 squared, the render-times are now 3 mins and the 32 sample 1min 41sec “Branched Path Tracing” STILL looks cleaner.
ALWAYS USE "Branched Path Tracing"
Next, as mentioned by Ikari, you need to scale your scene down, I scaled your scene down to 10% it’s original size, this helped with the refraction being less crazy. (I also suggest an IOR of 1.55 for glass which is the “proper” ior)
There are some better glass setups I could suggest, mainly using a glass shader and a transparent shader plugged into a mix shader (kinda like how you have it now) but use a light path node and plug the shadows into the factor.
bunch of other small things, don’t use Caustics (too noisy currently in cycles), your light sampling threshold is way too high (try 0.05), clamp settings I commonly use is 10 for direct and 2 for indirect. For the most part that will get you there, there are some other minor things but it’s difficult to go into detail as I’m currently at work and can’t upload images and such.
As for why the house is noisy with the glass as opposed to without, when the rays are entering the glass they are becoming a single “transmission” ray, so your ray could is reduced a lot through the glass. One way to specifically target this is in the branched path tracing just bump the transmission rays to 2, but with the settings I suggested above you should be fine without that.
Hope that helps