I think this will always happen because of your topology. You might find it easier to model the shorts more similarly to how shorts are sewn together in real life. Otherwise you could try very strict weight painting like I did here. One short leg at a time.
It states I need to ask permission to access the file.
To add to this, if it helps, I found this video quite useful for IK chains. It is short and would be sufficient for simple rigs.