Modern Way of Reading Pixel Data from Current Video / Movie Clip Frame

Support Python Support
,
1

I’ve been trying to read the pixel data for each frame in the currently loaded MovieClip. In an ideal world I’d like to be able to do something like this:

for frame_idx in range(context.scene.frame_start, context.scene.frame_end):
  context.scene.frame_current = frame_idx
  pixels = context.edit_movieclip.frame.pixels
  do_something_with_pixels(pixels)

  # OR...

  img = bpy.data.images.load(clip.filepath, frame=frame_idx)
  pixels = img.pixels
  do_something_with_pixels(pixels)
  img.user_clear()
  bpy.data.images.remove(img)

But it seems there’s no real way to specify the frame number in the image load. Tweaking the current frame via context.scene.frame_current = x doesn’t actually change the frame that’s loaded, nor does changing the frame offset. It appears the best method is still a decade+ old thread that involves a roundabout hack of loading the sequence into an ImageEditor: How to get particular frame of video "image"? and How to load a particular frame from video

Summarizing in case of link-rot:

  • Go through the context.screen.areas and find one which has type IMAGE_EDITOR.
  • Ensure the viewer space inside the current area also has type ‘IMAGE_EDITOR’.
  • Load the movie clip via bpy.data.images.load(clip.filepath).
  • Set the viewer_space’s image_user.frame_offset to change the frame and access the pixel data at context.screen.areas[whatever].spaces[whichever].image.pixels[…]

This feels like I’m missing something obvious, but I can’t seem to find (or properly decode) the documentation. https://docs.blender.org/api/current/bpy.types.MovieClip.html#bpy.types.MovieClip The data seems like it should be right there.

Have there been any advances in this approach in the past decade or so? Or is this really still the best way to handle reading pixel data?

To my untrained eyes, there’s an internal image accessor of sorts, but I find no corresponding op/type/prop exposed in Python: https://projects.blender.org/blender/blender/src/commit/43216b03ebbd2a4f642404ef74fe94bf0f15eca0/intern/libmv/libmv/autotrack/autotrack.cc

2

You need to use img.gl_load(frame=frame_idx)
It will load the current frame into an OpenGL texture
Then you can use img.pixels

To save a frame from a video:

img = bpy.data.images.load(filepath)
img.gl_load(frame=frame_idx)

w, h = img.size
result = bpy.data.images.new(f"frame_{frame}", width=w, height=h)
result.colorspace_settings.name = 'Linear Rec.709'
result.pixels = img.pixels[:]

img.gl_free()
bpy.data.images.remove(img)

Took a while to figure out, but works great.