Attempting to create a multi touch addons (Update : How to properly combine Matrix and Quaternion?)

[mercuryraven]

What the title said. I thought I would try to create a multi touch addon and sell it but uhh…has been utter fail rofl

so I just want a general idea on what else should I look into before starting this thing again.

Honestly most of my problem is trying to derive from the examples and not reading the manual…copying coding without thought from example.

I learned programming long ago with visualbasic,c# and python and have made unity game in the past. But uhh all of my skills just boils down to if statement…so I dont really know anything…
its really weird…i feel like i need to go over what is function and method for python

Anyway I have been looking at kivy and pygame and also pyMT…to achieve this.
pyMT seems to be outdated because it says it requires python 2 < python 3
pygame also appar

But Kivy is not exactly a library.(I dont know what framework means) I did manage to install kivy on the python that comes with blender…and managed to create a script where blender manages to run a separate application.

kivy supports multitouch…it has a feature called scatter which the user can move around with multitouch. Problem is I dont know where to get the position of the scatter object during runtime.

…I dont really feel like posting code right now…like I said i dont know anything…might be calling double method for no reason.

any advice?

[const]

You can mix any Python library with Blender and create hybrid applications. I have seen it many times back in the old 2.7 days.

https://kivy.org/doc/stable/examples/gen__demo__touchtracer__main__py.html

For example in this method def on_touch_down(self, touch): as of having imported the bpy as well you simply do a call to an operator like this bpy.ops.z.z() and it should work.

[mercuryraven]

oh man…i tried searching for an example like this but I am really bad at it so thanks!

I just have one problem…anytime the script attempts to use bpy.ops.z.z() based on Kivy’s app just shuts down…right there and then

so like something based on modifying the example app

 def on_touch_move(self, touch):
        if touch.grab_current is not self:
            return
        ud = touch.ud
        ud['lines'][0].pos = touch.x, 0
        ud['lines'][1].pos = 0, touch.y

        index = -1

        while True:
            try:
                points = ud['lines'][index].points
                oldx, oldy = points[-2], points[-1]
                break
            except:
                index -= 1

        points = calculate_points(oldx, oldy, touch.x, touch.y)
        if oldx > touch.x:
            bpy.ops.view3d.view_roll(angle=+0.261799)

the last line shuts down blender…is this some sort of security defense mechanism?

[const]

See if you get some info by starting Blender with debug command line argument
https://docs.blender.org/manual/en/latest/advanced/command_line/arguments.html

[mercuryraven]

(sorry for long post…I dont know which part is the debug part or random messages…)
hmmm if I start with blender --debug this is the error I get

 Traceback (most recent call last):
WARNING:kivy:stderr: Traceback (most recent call last):
   File "\kivyblendtouch.py", line 230, in <module>
WARNING:kivy:stderr:   File "\kivyblendtouch.py", line 230, in <module>
   File "C:\Program Files\Blender Foundation\Blender 2.82\2.82\python\lib\site-packages\kivy\app.py", line 855, in run
WARNING:kivy:stderr:   File "C:\Program Files\Blender Foundation\Blender 2.82\2.82\python\lib\site-packages\kivy\app.py", line 855, in run
     runTouchApp()
WARNING:kivy:stderr:     runTouchApp()
   File "C:\Program Files\Blender Foundation\Blender 2.82\2.82\python\lib\site-packages\kivy\base.py", line 504, in runTouchApp
WARNING:kivy:stderr:   File "C:\Program Files\Blender Foundation\Blender 2.82\2.82\python\lib\site-packages\kivy\base.py", line 504, in runTouchApp
     EventLoop.window.mainloop()
WARNING:kivy:stderr:     EventLoop.window.mainloop()
   File "C:\Program Files\Blender Foundation\Blender 2.82\2.82\python\lib\site-packages\kivy\core\window\window_sdl2.py", line 747, in mainloop
WARNING:kivy:stderr:   File "C:\Program Files\Blender Foundation\Blender 2.82\2.82\python\lib\site-packages\kivy\core\window\window_sdl2.py", line 747, in mainloop
     self._mainloop()
WARNING:kivy:stderr:     self._mainloop()
   File "C:\Program Files\Blender Foundation\Blender 2.82\2.82\python\lib\site-packages\kivy\core\window\window_sdl2.py", line 479, in _mainloop
WARNING:kivy:stderr:   File "C:\Program Files\Blender Foundation\Blender 2.82\2.82\python\lib\site-packages\kivy\core\window\window_sdl2.py", line 479, in _mainloop
     EventLoop.idle()
WARNING:kivy:stderr:     EventLoop.idle()
   File "C:\Program Files\Blender Foundation\Blender 2.82\2.82\python\lib\site-packages\kivy\base.py", line 342, in idle
WARNING:kivy:stderr:   File "C:\Program Files\Blender Foundation\Blender 2.82\2.82\python\lib\site-packages\kivy\base.py", line 342, in idle
     self.dispatch_input()
WARNING:kivy:stderr:     self.dispatch_input()
   File "C:\Program Files\Blender Foundation\Blender 2.82\2.82\python\lib\site-packages\kivy\base.py", line 327, in dispatch_input
WARNING:kivy:stderr:   File "C:\Program Files\Blender Foundation\Blender 2.82\2.82\python\lib\site-packages\kivy\base.py", line 327, in dispatch_input
     post_dispatch_input(*pop(0))
WARNING:kivy:stderr:     post_dispatch_input(*pop(0))
   File "C:\Program Files\Blender Foundation\Blender 2.82\2.82\python\lib\site-packages\kivy\base.py", line 286, in post_dispatch_input
WARNING:kivy:stderr:   File "C:\Program Files\Blender Foundation\Blender 2.82\2.82\python\lib\site-packages\kivy\base.py", line 286, in post_dispatch_input
     wid.dispatch('on_touch_move', me)
WARNING:kivy:stderr:     wid.dispatch('on_touch_move', me)
   File "kivy\_event.pyx", line 707, in kivy._event.EventDispatcher.dispatch
WARNING:kivy:stderr:   File "kivy\_event.pyx", line 707, in kivy._event.EventDispatcher.dispatch
   File "\kivyblendtouch.py", line 125, in on_touch_move
WARNING:kivy:stderr:   File "\kivyblendtouch.py", line 125, in on_touch_move
   File "C:\Program Files\Blender Foundation\Blender 2.82\2.82\scripts\modules\bpy\ops.py", line 201, in __call__
WARNING:kivy:stderr:   File "C:\Program Files\Blender Foundation\Blender 2.82\2.82\scripts\modules\bpy\ops.py", line 201, in __call__
     ret = op_call(self.idname_py(), None, kw)
WARNING:kivy:stderr:     ret = op_call(self.idname_py(), None, kw)
 RuntimeError: Operator bpy.ops.view3d.view_roll.poll() failed, context is incorrect
WARNING:kivy:stderr: RuntimeError: Operator bpy.ops.view3d.view_roll.poll() failed, context is incorrect
Error: Python script failed, check the message in the system console

Error   : EXCEPTION_ACCESS_VIOLATION
Address : 0x00007FF740BA4402
Module  : C:\Program Files\Blender Foundation\Blender 2.82\blender.exe

but if I do --debug–all the script craches as soon as I start the

wm_event_do_handlers: Handling event
wmEvent  type:1 / LEFTMOUSE, val:2 / RELEASE,
         shift:0, ctrl:0, alt:0, oskey:0, keymodifier:0,
         mouse:(728,409), ascii:' ', utf8:'', keymap_idname:(null), pointer:00000154BBEC4F28
[INFO   ] [Logger      ] Record log in C:\Users\Mercury\.kivy\logs\kivy_20-08-31_5.txt
INFO:kivy:[Logger      ] Record log in C:\Users\Mercury\.kivy\logs\kivy_20-08-31_5.txt
[ERROR  ] [Core        ] option --debug-all not recognized
ERROR:kivy:[Core        ] option --debug-all not recognized
Kivy Usage: blender [OPTION...]::

        -h, --help
            Prints this help message.
        -d, --debug
            Shows debug log.
        -a, --auto-fullscreen
            Force 'auto' fullscreen mode (no resolution change).
            Uses your display's resolution. This is most likely what you want.
        -c, --config section:key[:value]
            Set a custom [section] key=value in the configuration object.
        -f, --fullscreen
            Force running in fullscreen mode.
        -k, --fake-fullscreen
            Force 'fake' fullscreen mode (no window border/decoration).
            Uses the resolution specified by width and height in your config.
        -w, --windowed
            Force running in a window.
        -p, --provider id:provider[,options]
            Add an input provider (eg: ccvtable1:tuio,192.168.0.1:3333).
        -m mod, --module=mod
            Activate a module (use "list" to get a list of available modules).
        -r, --rotation
            Rotate the window's contents (0, 90, 180, 270).
        -s, --save
            Save current Kivy configuration.
        --size=640x480
            Size of window geometry.
        --dpi=96
            Manually overload the Window DPI (for testing only.)

Saved session recovery to 'C:\Users\Mercury\AppData\Local\Temp\quit.blend'
Writing userprefs: 'C:\Users\Mercury\AppData\Roaming\Blender Foundation\Blender\2.82\config\userpref.blend' ok
Error: Not freed memory blocks: 22, total unfreed memory 0.030342 MB
GPUShader len: 32 00000154A7F19528
GPUShaderInterface len: 6352 00000154A7A66C78
name_buffer len: 24 00000154A7EFD738
GPUShaderInput Attr len: 32 00000154A7F18958
GPUShaderInput Attr len: 32 00000154A7F18BF8
GPUShaderInterface batches len: 128 00000154A7D38118
GPUShader len: 32 00000154A7F18C68
GPUShaderInterface len: 6352 00000154A82B5938
name_buffer len: 36 00000154A7F186B8
GPUShaderInput Attr len: 32 00000154A7F19B48
GPUShaderInput Attr len: 32 00000154A7F18798
GPUShaderInput Attr len: 32 00000154A7F19598
GPUShaderInterface batches len: 128 00000154A7D39158
_build_translations_cache len: 64 00000154A8328128
ghash_buckets_resize len: 40 00000154A9F120A8
memory pool len: 48 00000154AA001458
BLI_Mempool Chunk len: 2024 00000154AA0C1E08
IDProperty group len: 128 00000154B01562C8
TEXT_OT_run_script len: 168 00000154AF897A38
wmOperatorPtrRNA len: 24 00000154BBDDB4F8
idp_generic_copy len: 128 00000154B0156EF8
wmOperatorReportList len: 40 0000015494E4BD58

[const]

Most important thing that the error message (usually is the last one) says is:
Operator bpy.ops.view3d.view_roll.poll() failed, context is incorrect

I don’t know exactly what your script looks like but I can guess that you try to invoke an operator on top of a running operator. While the modal operator runs there is no way to run a second modal.

The only way is to replicate again the same features in your own code, such as the view_roll.

P.S. I went with a lot of guessing in this answer just to save time, but if you try to do something else other than I mentioned, you can correct me.

[mercuryraven]

Decided to go completely different way with this concept here is a video of all the work I have done (it is really just cobbled code of different people’s code)

I am probably not gonna continue with this. But just wana showcase the concept of what I intended.

WIP, BlenderCameraController Addon, an alternative way for touchscreen on blender

I am pretty sure this is redundant particularly as there is already Blender AddRoutes.

[const]

I wonder if using a modal operator with pass-through can do something for this. If the results are the same.

Perhaps there is the alternative way to use timers. Such as for example Javascript codes in HTML, never run a blocking loop, but schedule the next update for the future. This seems like a much more feasible update model rather than multithreaded.

https://docs.blender.org/api/current/bpy.app.timers.html

I think though that the effort is really good though, not many people attempted these things so far and is a bit of unexplored territory. But it holds lots of possibilities.

[mercuryraven]

Hi,

I initially had a hard time understanding how it worked…but after looking at the template (idk how I missed it) I think I managed to remake it to fulfill the purpose.

On the other side though, its not quite what I envisioned…I wanted to create an application/controller that would have a floating gizmo on top of every other application.

I switched to making the controller with Unity instead of Kivy because I was accustomed to Unity3d. I initially tried making the controller with Kivy before but kivy’s position changes when you scale it, there was a way to fix it but I couldnt make it work.

I heard Godot also has the ability to make transparent window with clickthrough…but I dont think godot supports multitouch on desktop os.

I am using pheonise unity project on github. While it works well with mouse I just couldnt make it work with multitouch…Unity somehow fails to detect touchscreen input when it comes to transparent window mechanic.

But I am releasing what I have…for now…Unfortunately it is very prone to crashing…the panning movement is jarring…but I am just really tired…been hitting this problem like beating a deadhorse

Thanks for everything.

[const]

An idea is to run the modal operator with an update rate of 1/60.0f however you need to store the current value and the new value, an interpolate between them.

self.currentpos = 0
self.newpos = 0
self.currentpos = self.lerp(self.currentpos, self.newpos, 0.5)

    def lerp(self, start, end, amount):
        return start + amount * (end - start)

Interpolation is done at 60fps, however server can update even at 1/4 of a second. You will have super smooth motion, but since the newvalue is updated in a slower rate, you might have a little slide here and there.

Also in order to prevent crashes. I consider that is OK to start the server inside a thread.
Provided that you use only intermediate variables (copies) and do not access the bpy directly. That way you won’t even touch the bpy and have thread-crashes.

https://www.geeksforgeeks.org/socket-programming-multi-threading-python/

So you end up with something like this:

# inside the thread
data = s.recv(1024)
gdata = data
# ...
# and now inside the modal operator

# the object position is set at a constant 60 fps rate
context.scene.objects['Cube'].location.x = self.lerp(self.thisdata, self.newdata, 0.5)

# new data updated only when they are fresh
# or even you can delay the check even more (if needed)
if self.newvalue != gdata:
  self.datavalue = gdata

However for sure lots of tests and even more attempts need to be made. Is not exactly sure how things work better, there are many techniques and hacks around.

[mercuryraven]

Wow, its been a year. I have progressed so much with the addon, I have been attempting to upgrade it and I came across some problem. Sorry I wasnt sure If I should make a new topic or revive this thread. Decided to revive it. I actually asked this on the Blender Dev and also Stack Exchange but wasn’t getting any answer.

Anyway here is my copy and pasted post from the dev talk forum:

…

Hi,

I am working on an addon that uses external application’s touch gesture to send information to a socket, which then is used to navigate the viewport through Python’s api and timer modal. I think I am almost there but I just need to understand how some of the math works . Just one thing I need to fix to get it working exactly the way I intended.

So first I have the rotational values that is changing the viewport/RegionView3d:

#self.defq is the initial rotation of the viewport before timer starts...r,rx,ry are all values from socket
quat_r = mathutils.Quaternion((0, 0,-1*srvrprop.rtouch_speed), math.radians(r))
quat_x = mathutils.Quaternion((1*srvrprop.rtouch_speed, 0, 0), math.radians(rx))
quat_y = mathutils.Quaternion((0, 1*srvrprop.rtouch_speed ,0), math.radians(ry))
quat = self.defq @ quat_r @ quat_x @ quat_y                            
self.region.view_matrix = quat

If I put it though the view_rotation. It works as intended. It starts rotating from user’s perspective

I also have the positional value that is directly being put through the copy of the view_matrix:

#self.matx is the initial view_region's copy before timer starts. lr,ud,ss are all values from socket
self.matx.col[3][0] = lr
self.matx.col[3][1] = ud
self.matx.col[3][2] = ss

Initially it was just using “view_location” to change position but then I rewrote it to directly change matrix values so that it would change the position from user perspective and not the Cartesian value of the grid.

My current problem is that I need to put them together . So far

self.region.view_matrix = self.matx @ quat.to_matrix().to_4x4()

1. if I multiply them and put it into a view matrix, it resets the view port to top view before any changes being made to the view port.

self.region.view_matrix = quatm
self.region.view_rotation = self.defq @ quat_r @ quat_x @ quat_y

2. If I put them separately, It starts being more laggy…and the location change works like a cartesian grid instead of going forward from the user/viewports perspective

self.region.view_matrix = mathutils.Matrix.LocRotScale(vec2,quat,None)

3. If I use LocRotScale, the “r” value only orbits at the centre instead of rotating from the user.

I am sure there is an easy answer to this that I don’t know about since I am not familiar with math. But any help would be appreciated.

[manufanki (Manuel Fankhänel)]

Hey,

I used multi-touch for my own blender addon, this and many other posts where extremely helpful.
I first tried kivy as recommended by @const, but i could not run it with a modal operator. I tried threading but this was very unstable so i switched to pygame.

Here you can see a code example:

Blender multi-touch addon with pygame

Hello everyone, I have created multi-touch support using Pygame for my own Blender addon. This is sample code for anyone trying to do the same. You will need to install the pygame dependency inside blenders python. To do that, check out this link: https://blender.stackexchange.com/questions/168448/bundling-python-library-with-addon This code creates a small window that you can touch with multiple fingers. The console will display many prints with the finger ID and position in relation to t…

[joseph (J)]

I’m curious now- Pygame has its own, notoriously slow main loop, and Blender has a main loop as well- how does running a Pygame instance in Blender affect performance? I’d guess at least a 20% hit to FPS

[const]

Eventually I did try something like this, which is much more simplified, compared to threading and synchronization of data among threads etc…

Provided that you can limit blocking time somehow to less than 5ms you would not experience anything negative in terms of lagging.

The the real point is to consider that you must not nag the data receiver continuously with requests, since you get more blocking that way. So you can split the the receiving part from the updating part and kinda get away with it.

Since threading is quite unstable, it might not be the best choice. Though there are lots of workarounds (queue, multiprocess) perhaps at some point another technique can be used, according to requirements and other use cases.

import bpy
import time
import random

class DataReceive:
    def __init__(self):
        self.flip = 1
    def receive(self):
        self.flip *= -1
        return self.flip * random.randint(5, 10)

class Interpolate:
    def __init__(self):
        self.this = 0.0
        self.new = 0.0
        
    def lerp(self, a, b, blend):
        return (blend * (b - a)) + a
    
    def update(self):
        self.this = self.lerp(self.this, self.new, 0.2)
        

class ModalTimerOperator(bpy.types.Operator):
    bl_idname = "wm.modal_timer_operator"
    bl_label = "Modal Timer Operator"
    _timer = None

    def execute(self, context):
        print('execute')
        
        # validate
        if 'Cube' not in context.scene.objects:
            print('not found Cube')
            return {'CANCELLED'}
        self.object = context.scene.objects['Cube']
        
        # prepare modal
        wm = context.window_manager
        self._timer = wm.event_timer_add(0.01, window=context.window)
        wm.modal_handler_add(self)
        
        # to receive data
        self.data = DataReceive()
        
        # to interpolate
        self.interp = Interpolate()
        
        # to receive data periodically
        self.prevtime = time.time()
                        
        return {'RUNNING_MODAL'}

    def modal(self, context, event):
        # cancel with ECS
        if event.type == 'ESC':
            self.cancel(context)
            print('cancelled')
            return {'CANCELLED'}

        # timer update
        if event.type == 'TIMER':
            # receive each second
            if time.time() - self.prevtime > 1.0:
                self.prevtime = time.time()
                self.interp.new = self.data.receive()
                print('new value', self.interp.new)
            
            # update position
            self.interp.update()
            self.object.location.x = self.interp.this
                
        # return pass through
        return {'PASS_THROUGH'}

    def cancel(self, context):
        wm = context.window_manager
        wm.event_timer_remove(self._timer)

def register():
    bpy.utils.register_class(ModalTimerOperator)

def unregister():
    bpy.utils.unregister_class(ModalTimerOperator)


if __name__ == "__main__":
    try: unregister()
    except: pass
    register()
    bpy.ops.wm.modal_timer_operator()
 

[manufanki (Manuel Fankhänel)]

I cant see a major performance decreese. I used a model operator with a timer and even with the timer set to 0.01 sec update rate the instance runs smooth.
I do not use the pygame update loop I replaced it by the modal operator.

If you want to try it by your own check out the code I have provided by my post.