Each object is rendered to a different RenderLayer, with the power of node compositing you can have layers interact with each other and result to a proper masking, making the integration between the background and the 3D objects almost great.
Procedure of creation:
Setup a background, in this case I have set it as a texture to a 3D plane mesh.
I added a Monkey mesh in the the scene for testing.
In this case I create a Bezier Curve and trace the outline of the tree which is in foreground. Actually, any rendered object can work but Bezier Curve is very easy to use for outlining. The Bezier curve could be set to 2D (for better handling) and make it Cyclic U (to form an enclosed surface).
Move objects in different layers
Layer 1: Mask (Bezier Curve)
Layer 2: Monkey (along with some lights for proper rendering)
Layer 3: Background
Create new RenderLayers
While creating a new RenderLayer we will have to set which exactly scene Layers to match.
Here’s the setup:
Mask: Layer 1
Monkey: Layer 2
Background: Layer 3
Follow the instructions of this image or study the blend file, I found out about it based on experimentation and luck. It works nicely for now, I hope that it will be reusable for many other occasions on your projects.
Download zip file containing blend source here: