It’s because of the seam at the bottom of the legs. It’d be clearer if you post some wires of the model, but it’s likely that the rest of the mesh has fairly large quads compared to the narrow ones at the seam. Your cloth settings are making the seam quads try to expand, and some are, but not others, instead they’re squeezing together. If you’re using Edge Split on the seams it’s also probably contributing to the problem.
For a more even cloth motion you need a more even mesh in terms of quad size. This makes detail like seams difficult to include in the mesh itself – maybe better to use texturing or normal mapping for a more “implied” seam? I’ve never seen boxers with that heavy a seam, anyway, looks almost like a cuff.