How do I use border select (B) to deselect some vertices from the current selection?

We all use border select (B) a lot. But how do I use it to deselect some vertices from the current selection?

B and drag middle mouse button

Thank you!