How To: convert mouse cursor position to 3D scene space

Game Engine Game Engine Resources
#1

Hi all,

we often see questions like:

  • how use mouse movment to aim up and down?
  • Track to mouse position?
  • mouse parent?
  • Moving an object with mouse?
  • mouse movement?
  • Mouse look?

In a lot of this questions there is a lack of understanding what the mouse position is and how it belongs to a 3D scene.

The mouse
The mouse is an input device.
There are several different device types e.g. Mouse, Trackball, Tablets, Touchscreen
They use different methods and are input devices. For simplification they are called mouse within this thread.

The mouse provides tracking information of at least 2 axis and some buttons.

The mouse resides usually on your desk. This makes it really hard for a game character to follow it.

Requests like: “move the object to the mouse” -> are are physical not possible ;).
I’m pretty sure this is more an issue of how to express the requirements :evilgrin:.

The user really meant: move the object to
The system mouse cursor
is the visual representation of the pointing device. It is a helper that shows the converted mouse position at the screen. By moving the (physical) mouse the position of the system mouse cursor changes.

Usually if someone talks about “mouse position” he means “position of the system mouse cursor on screen”.

The request is: “move the object to the position of the system mouse cursor”
Unfortunately this is still not specific enough.

Why?
The system mouse cursor lives in 2D space of the screen.
Our Scene is (virtually) 3D.

We need to convert the position of system mouse cursor to
A position in 3D scene space

If you (virtually) look from the side at your scene you can see under the system mouse cursor is not just a position. There is an unlimited number of positions. To be more specific: there is a line of positions.


You can pick any of them and you met your request.
The question is: which one?

This explains the next post …

#2

There are a lot of methods to choose a position under the system mouse cursor. I will present the most common once.

constant axis


It means you simply use the system mouse position (2 coordinates) to calculate two axis of the 3D scene. The third axis gets a constant. This makes means the chosen position is always on a flat plane.
E.g.:
mouse X -> 3D Y
mouse Y -> 3D Z
0.0 -> 3D X
This is very simple. The conversion can be in world space or relative to the camera (or any other object).
Unfortunately there is no build-in logic brick which does this for you.

Constant distance


It means the position has always the same distance to the camera. The position would always be on a (partial) sphere around the camera.
This can be easily achieved via parenting.

Hit Point


Earlier we saw that users tend to assume the position of the system mouse cursor when they are talking about “mouse position”.

Here the user usually assumes the “object that I see under the position of the system mouse cursor” when he talks about “to the mouse position”.

We choose the position of the nearest pixel of a face that crosses the “system mouse position - line”.

The BGE’s mouse sensor provides this position. It basically measures a ray from a point close to the camera to a point far away from the camera. The the hit point of first face blocking the ray is returned. Again only in Python.

Example
If you do not know Python, this is no problem you can use the S2A to place whatever object you want at this position.

Grab the S2A from the S2A blend file.

Apply following logic:
Mouse Sensor “Over Any”
Mouse Sensor “Movement” (True Pulse)
–> Python controller Module: S2A.hitPositionToPosition
-----> Any actuator at the object to be placed at the hit position

#3

What if there is no face under the system mouse cursor?

In this case I recommend a combined method.

E.g. when there is an object under the system mouse cursor use the hit object method. Otherwise use the constant distance method.

There are several other options to combine this methods.
Which one you prefer is up to you ;).

I hope this little explanation helped you

Monster

#4

hi ,
if I know the Z position (is thath wich lack to the mouse) , this is => 0
I know the orientation of camera(own.orienation[2]*-1)
how found the “intersect point”? (the same point which can found with “mouse over any”…)

vec = own.orienation[2]*-1
z = 0
intersectionPoint = ?

PS: I have alway see somethings , but there also other things as mouse.x, and mouse.y which make me confusion

thanks in advance (i preffer CtoC approach ;D )

#5

like this?

http://blendercave.tuxfamily.org/?page=objet&id=251

#6

So how do I get the space the mouse is pointing at a constant distance?
If I can get a vector from the object to a point in space, I could get a simple mouse look! Or set a destination point, etc!

#7

The constant distance method can be achieve with parenting:

  • have an empty (joint) at the position of the camera.
  • parent the joint to the camera (so it inherits the camera’s motion)
  • parent another object (e.g. 3D cursor object) at the joint with the distance you like.
  • let the joint track any point on the line under the system mouse cursor (e.g. mouse over any ray source or ray target)
#8
  • let the joint track any point on the line under the system mouse cursor (e.g. mouse over any ray source or ray target)

That’s the core of my question! What code lines do I use to get that?
Do I just take the cursor’s position and use it as x and z coordinates for the 3D cursor?

#9

Maybe this script might help:

import bge
from bge import render as render
from bge import logic as l

cont = bge.logic.getCurrentController()

own = cont.owner

m = cont.sensors[“Mouse”]

bge.render.showMouse(0)

sensitivity = 0.005

w = bge.render.getWindowWidth()
h = bge.render.getWindowHeight()

w2 = w//2
h2 = h//2

x, y = m.position

x = (w2 - x)*sensitivity
y = (h2 - y)*sensitivity

own.applyMovement([x*-1.0, 0.0, 0.0], True)
own.applyMovement([0.0, y, 0.0], True)

render.setMousePosition(w2, h2)

I am currently using Blender 2.61. This is the code I will use in my game Icarus. I learned this code in youtube (I didn’t copy and paste, I coded it from the instructor’s teachings, so I think I own the code?). Though it originally was for mouse look, I tweaked, changed and manipulated it to make it as a door opener in Icarus. When the 3d object runs it, it automatically acts like a camera. It’s use is to copy the mechanics shown in Amnesia: The Dark Descent where you use the mouse to swing open doors, or other hinged objects. I hope this helps a lot!

#10

When you post code please use the code tags.
This code does not find a position under the system mouse cursor.
It uses relative mouse motion. Which means it takes the change coordinates of the system mouse cursor and applies it to a motion along x and y axis of an object. This is a transformation too. But it does not try to find an position under the system mouse cursor.

The code will conflict with absolute motion of the system mouse cursor as it resets the system mouse cursor to the middle of the game window all the time.

This is a different approach to use the mouse as input device unfortunately it is incompatible with the described methods above (at least with the proposed implementation).

#11

This code does not find a position under the system mouse cursor.
It uses relative mouse motion. Which means it takes the change coordinates of the system mouse cursor and applies it to a motion along x and y axis of an object. This is a transformation too. But it does not try to find an position under the system mouse cursor.

The code will conflict with absolute motion of the system mouse cursor as it resets the system mouse cursor to the middle of the game window all the time.

This is a different approach to use the mouse as input device unfortunately it is incompatible with the described methods above (at least with the proposed implementation). <Monster>

Yes. It only makes an object move in the 3d space. Sorry for that. But I do know that this script involves making the system cursor send pulses to the 3d space in blender. So it might/ can be implemented (in other words, modified) to track the mouse’s current position and translate it into 3d space. Correct me if I am wrong.

When you post code please use the code tags.

Sorry, I am a new member. I don’t know how to do that yet.

#12

You can edit your own posts.
In advanced edit mode you have the tool # to add the

/[/ code] (without the spcae) tags similar to [quote].

Yes, you can calculate a system mouse position from relative movement. I did that with another project. In this case a custom mouse cursor is recommended as the system mouse cursor constantly jumps to the middle.
;)
#13

Man, your modules / script help me a lot! thanks. But it’s not easy for me to understand your python library. ( one day for this one, i’m a really noob in python).
S2A.hitPositionToPosition is good for my project.
I’ve a question >> how can i ignore some objects? I want to put an objet on another object with S2A.hitPositionToPosition, but i want to ignore some other objet in front of the first one. How can i do that without make it in “no collision?”
Thanks

#14

You can do this only with a ray sensor. The ray sensor has a x-ray switch so it can detect through other objects.

It might be easier to write an own Python code that casts a ray from mouseSensor.raySource to mouseSensor.rayTarget with enabled X-Ray parameter.

#15

Indeed, thanks … this is my next step :wink:
Have you got a good EASY tuto for Ray sensor? it’s difficult to understand.
Thanks!

#16

rayCaster.py:


'''
rayCaster.py
============
Provides entry points to casts rays 
that are not supported by default sensors.
'''

__version__ = '1.0'
__author__ = 'Monster'

#Configurational Properties
Pxray = "xray"
'''Property to block the xray'''

XRAY_ON = 1
IGNORE = 0

#BGE callable
def xrayHitPositionToPosition(cont):
    '''
    Fires a ray from sensors rayTarget to raySource.
    The ray uses X-Ray with the property name 
    defined in property "xray".
    Places the owners of all connected actuators 
    at the detected hitpoint.
    Actuators will not be activated by this controller!
    '''
    detector = cont.owner
    
    rayParamProvider = getRaySensor(cont)
    
    ray = detector.rayCast(
        rayParamProvider.rayTarget,
        rayParamProvider.raySource,
        IGNORE,
        detector.get(Pxray,""),
        IGNORE,
        XRAY_ON
    )
    
    hitPosition = ray[1]
    
    if hitPosition is None:
        hitPosition = rayParamProvider.rayTarget
    
    for actuator in cont.actuators:
        actuator.owner.worldPosition = hitPosition
 
# internal   
def getRaySensor(cont):
    for sensor in cont.sensors:
        try:
            sensor.raySource
        except AttributeError:
            continue
        return sensor

Properties:
“xray” string value: your xray property

Logic bricks:
Mouse sensors True-Pulse Mode:Mouse over any
–> Python Module: rayCaster.xrayHitPositionToPosition
----> Any actuator on each object you would like to relocate (e.g. a property actautor)

Remarks:

  • connected actuators will not get activated by this controller
#17

Thanks for suggesting that S2A library, it’s awesome!

#18

This is very interesting Monster. After every thread of yours I read I feel my knowledge is expanded just that little bit more.

I’ve been thinking about this subject a bit lately so I don’t know how I missed this thread!

What I came up with are 3 options for constant distance (IMO generally speaking the most useful)

  • You could as you say create a sphere with a radius of the constand distance desired. Parent the sphere to the camera, set it to ‘no collision’ and ‘invisible’. Then shoot a ray from the camera and read the hit position.
  • Similar to the first, place a plane in front of the camera and shoot a ray from the camera into it. Once again set the plane to ‘no collision’ and either make it infinitely small (eg scale on all axis 0.001) or ‘invisible’. Again get the hit position.
  • Use maths. Surely there would be a way of writing an algorithm that uses the current camera position, angle & frame dimensions to get a point in 3D space a certain distance from the camera.
    Personally I would use the 3rd method, because you’re not using a ray sensor it theoretically would be less taxing on logic (and it seems like an inefficient way of doing things) but my knowledge of maths is insufficient, or maybe I’m just not trying hard enough :stuck_out_tongue:

I’m thinking about taking some mathematics courses as I’d love to be able to code like you guys do and I think it’d make a big difference.

Once again though monster you get the old grey matter working and answer a few questions at the same time so thanks :smiley:

#19

I would not create a sphere, but it is possible. The point was if you choose a point under the cursor with a constant distance you get a sphere. This is more a mathematical description. You know you can describe a sphere in 3D space as point + radius (constant distance). This is just a description. It is not a visible model ;).

As there are surely multiple methods to implement this, I found a pretty good solution with parenting. You parent your “3D Mouse Cursor” at the camera (= constant distance). With each mouse move you turn the “3d mouse cursor” that it fits under the system mouse cursor.
The advantage is the “3D Mouse Cursor” will always face the same side to the camera with the same distance. If you use the “hit-a-plane” method (described next) you will see the 3D Mouse cursor also from the sides (depending where it is on screen). The cursor might become small as further it is from screen center.

If you use NO Collision the ray can’t detect it ;). But this is a very good method to place a 3D Cursor under the System mouse cursor in an Overlay scene. Please have a look at 3D Cursor. It does exactly that.

Regardless of the method you always need math. But before you go and create wild formulas, you should be clear in what you want to calculate.
The BGE can provide you with a lot of options to avoid to run your own math. Which one is the best for you depends on your situation.

(BTW: when using math in 3D space try to avoid angles. They are hard and inefficient to calculate. Angles are just easier to ready by humans. Better stay with vectors as this is what you get from the BGE already. And vectors is what you have to return to the BGE. You can indeed use any intermediate form (matrices, scalars, angles, bread crumps …) you like as long as you can convert it to vector.).

If you can choose mathematical courses go for Matrix/Vector calculation as this is how current 3D engines work. Please be aware this is hard stuff. If you keep in mind what you want to do with that, it might be easier to learn.
BTW: The subject “Geometry” might not really help as this are more “human” methods (using ruler and pen). It still can be useful (e.g. if you want to build a right angle in your garden ;))

#20

OK. So this by itself works. I have 2 questions:

1 -How would you ignore some objects, so that the MouseRay goes through them and intersects with anything behind them? Like with the X-Ray option but instead of ignoring objects that don’t have the property, ignoring the objects that do?

2 -How do you change from the hitPositionToPosition method to the Constant method? (when there is no face to intersect with)