Check if mouse is on custom handle

I am trying to code a simple modal operator with some guidelines in the 3D viewport like the move/translate operator. When the user clicks on one if this lines he can edit the object in some way. My question is how can I check if the mouse is over one of these guidelines. I have drawn them via the bgl functions. The user is able to rotate, zoom and pan in the viewport, so the position, size and shape of the guidelines change (like the ones from move/translate).

Is there some way how I can identify when the user clicks on of them?

I would say only if you keep the data you use for drawning stored somehow and check for intersections (in case of lines), another approach would be to sample the color and if it’s the color of what you drew earlier, then it’s a hit (not sure how well this works if done in python).