If i understand you correct you want to rotate around the yellow points without leaving object mode, right?
For this:
- set pivot point to 3d cursor
- set snapping to vertices or edge
- hold left mouse button, move your mouse, then hold ctrl too -> so the 3d cursor starts snapping
- then you can rotate and scale (keys [r|s]+[x|y|z]+(mouse move | [-]number))