… ok, Its fairly hard to tell what the problem could be without more info , screens or a .blend.
for example, how many substeps does the simulation use. How “complex” are the “colliders” in the scene and how many collision objects are there. is the object emitting the particles also marked as a collision mesh? and are any of the collision objects animated, or modified with, for example subdivision surfaces…
If i had to guess, I would say the collision meshes are WAY to complicated… to check this, turn them all off… see how it runs without anything for the particles to collide with.
also … check the order of the modifiers for each collision mesh… putting a subdivision surface in the wrong place will REALLY start to slow the simulation speeds, as the collision mesh will be much more complicated… so check that collision modifier is at the BOTTOM of the modifier stack of anything with a collision modifier.
Ensure the substeps for any particle systems are set to 1, and increase IF U NEED TO… don’t increase it unless you really have to for the sake of accuracy. keep in mind a substep setting of 10 is like having 10 “subframes” for each actual frame… hence it can make the simulation bake 10 times slower than usual.
and for the last question, things which aren’t involved in the simulation (such as meshes without a collision modifier) will not effect the speed of the simulation… it may effect the playback speed, but not the bake speed…
If you try these things , and still no luck… try uploading the blend file… its easier to find a needle in a haystack, when we have access to the haystack