Running background jobs with Asyncio

(emu) #1

Over the last few days I’ve been coding a script that silently runs in the background and only from time to time, it decides to download some objects from the Internet. It was a job assignment, details are not important. You will face the same issue if you make any interactive network communication in Blender.

There seems to be a recommended approach to this in Python, and it is the async/await scheme. Asynchronous download of a webpage may look like this:


<b>import</b> asyncio
<b>import</b> aiohttp

<b>async def</b> fetch_page(url):
    <b>with</b> aiohttp.ClientSession() <b>as</b> session:
        <b>async with</b> session.get(url) <b>as</b> response:
            content = <b>await</b> response.read()
            <i># do something with the content</i>

task = asyncio.ensure_future(fetch_page("http://blender.org"))
asyncio.get_event_loop().run_until_complete(task)

The beauty of Asyncio is that multiple such requests can be handled in parallel, although there are no threads involved.

If you would run exactly this code in Blender, it would block the user interface until all pages are downloaded. To make it work, we need a bit of magic – namely:

  • Tell Blender to go through our task list several times a second
  • In each go, handle web pages that have just been downloaded

A nice solution to (1) is to have a modal operator and set a timer to it, with period much below a second. Like this:

wm.event_timer_add(0.01, context.window)

The trick in (2) is to let Asyncio just once read through the list of pending tasks, and return as soon as possible. This surprising snippet does just that:


loop = asyncio.get_event_loop()
loop.stop()
loop.run_forever()

I learnt this elegant idea from the Blender Cloud plugin. It is actually a documented feature, but a bit hard to find: despite the loop is stopped already before starting, it will visit each task once.

All combined together into a handy package:


<b>import</b> asyncio
<b>import</b> bpy

timer = <b>None</b>

<b>class</b> AsyncLoop(bpy.types.Operator):
    bl_idname = "asyncio.loop"
    bl_label = "Runs the asyncio main loop"
    command = bpy.props.EnumProperty(name="Command",
        description="Command being issued to the asyncio loop",
        default='TOGGLE', items=[
            ('START', "Start", "Start the loop"),
            ('STOP', "Stop", "Stop the loop"),
            ('TOGGLE', "Toggle", "Toggle the loop state")
        ])
    period = bpy.props.FloatProperty(name="Period",
        description="Time between two asyncio beats",
        default=0.01, subtype="UNSIGNED", unit="TIME")

    <b>def</b> execute(self, context):
        return self.invoke(context, None)

    <b>def</b> invoke(self, context, event):
        <b>global</b> timer
        wm = context.window_manager
        <b>if</b> timer <b>and</b> self.command <b>in</b> ('STOP', 'TOGGLE'):
            wm.event_timer_remove(timer)
            timer = None            
            <b>return</b> {'FINISHED'}
        <b>elif not</b> timer <b>and</b> self.command <b>in</b> ('START', 'TOGGLE'):
            wm.modal_handler_add(self)
            timer = wm.event_timer_add(self.period, context.window)
            <b>return</b> {'RUNNING_MODAL'}
        <b>else</b>:
            <b>return</b> {'CANCELLED'}

    <b>def</b> modal(self, context, event):
        <b>global</b> timer
        <b>if not</b> timer:
            <b>return</b> {'FINISHED'}
        <b>elif</b> event.type != 'TIMER':
            <b>return</b> {'PASS_THROUGH'}
        <b>else</b>:
            loop = asyncio.get_event_loop()
            loop.stop()
            loop.run_forever()
            <b>return</b> {'RUNNING_MODAL'}

<b>def</b> register():
    bpy.utils.register_class(AsyncLoop)

<b>def</b> unregister():
    bpy.utils.unregister_class(AsyncLoop)

<b>if</b> __name__ == "__main__":
    register()

The asynchronous web download then looks much like above, just with an extra operator call:


asyncio.ensure_future(fetch_page("http://blender.org"))
bpy.ops.asyncio.loop()

The difference is that the former code did exit as soon as the page was downloaded, whereas this will silently loop forever. Fortunately, you can call bpy.ops.asyncio.loop(command=‘STOP’) as soon as the page is downloaded, and get the same result.

btw, I marked this thread as Solved because I am just happy how I solved the task :slight_smile:
Another note: the aiohttp library is great, but has to be installed. By default, Asyncio only allows you to read/write files and raw TCP/IP connections.