Object outline as transparency mask

[aerocoptic]

Hi all, I’m developing a watercolor look for a music video.

I’m getting some good results, and with some post compositing in AE it’s starting to look solid. Here’s a quick test I did:

The issue is that there’s a large piece missing. I need to noise and alpha up the edges of each object to let them blend together as if they’re actually hand-painted. I’ve got to do this in the shader editor and not the compositor, because this needs to operate on a per-object basis and creating an ID mask for scenes with hundreds of objects is not a viable solution. In After Effects, I can use the ‘Roughen Edges’ effect to produce the same thing basically, but unless I render out each object individually (which I need to avoid for a number of reasons, namely lighting), I can only really roughen the alpha of the whole scene in front of the background.

Below is using the Layer Weight > Facing factor to drive the noisy outlines on the torus, which looks good from some angles and bad from others. See below (I can’t embed images in the post since I’m a new user, sorry!)

[Album] Watercolor rough edges

I’ve scoured everywhere for solutions to this. Freestyle, Line art, the solidify modifier method, are all limited and can’t really produce the effect I need at scale, unless I’m missing something.

But just selecting an object in blender gives me the outline I want, so obviously the functionality is built-in. I have no experience in python or developing for Blender, but maybe that’s the ticket? Hopefully someone can point me in the right direction here. I’d be happy to pay someone for their time working on this, too.

Thanks in advance!

[joseph]

Welcome to BA

You can do this pretty simply with a node setup something like this, if I’m understanding your goal correctly. Just change the Diffuse BSDF for your shader.

Nice work on your watercolor shader by the way, it looks awesome!

Another advantage of this method is that your outline will change with the camera, which really contributes a watercolor look:

Gyazo Screen Video

It looks a lot better shaded smooth, I just wanted to emphasize that this is just a normal cube with some subdiv.

Make sure to set your Material settings to Alpha Blend and turn Backface Culling on:

Here’s a more organized node setup:

One more thing, you’ll get really bad results without a subdivision modifer:

Suzanne on the left has a level 2 subdiv, Suzanne on the right has no subdiv. In my quick tests, even just a level 1 subdiv clean ups the edges enough. Just make sure your objects all have that

EDIT: I saw that you’ve tried a Facing layer weight solution that didn’t work. This will work a lot better- it uses the Fresnel instead, and a color ramp to clamp that transparency to just the edges

[aerocoptic]

Thanks @joseph. Almost what I need, except that I need ONLY the exterior contours to be rough. I was using the layer weight node previously and it creates the same problem in your last image, where Suzanne’s eyes and brow are transparent. Trying to avoid that and only roughen up the actual silhouette of the mesh.

[joseph]

Well… it’s possible, but you’re not going to like it Instead of using the Fresnel as the mix Factor, you use an image texture. That image texture you then paint by hand- anything white will be transparent, and anything black will be solid. You’ll have to very carefully paint around all the outlines.

EDIT: You can partially fake it with some geometry tricks, but not really

[aerocoptic]

Haha, thanks. That’s more or less what I figured. It’s video, so it would be tough to paint every frame. I might have to see if I can hire a developer to help me work this out.

[joseph]

@aerocoptic

I might have to see if I can hire a developer to help me work this out.

You’re certainly welcome to, but honestly, you’d be wasting your money.

I’ve devoted myself almost exclusively to NPR and node work in Blender, I’ve tried every different method of inner and outer lines extensively and even scripted a few of my own. I can suggest probably 20 different ways to get “close” to what you want, but a perfect version isn’t currently possible in Blender.

For example, you can easily have an outline that has variable/rough width, which looks painterly, but it doesn’t work with transparency. Or you can have like here, where you have a transparent outline that looks great in most contexts but not all.

You could randomize the transform of all vertices, which will give you a roughened edge, but it will also distort your object.

You could use a displacement or bump map, which would work great on the edges, but is going to mess with your shading in undesirable ways.

You can do X and Y tricks with Freestyle and the Compositor, and so forth, and so on… (for example, you can render each frame with a transparent background, and then roughen the edges with a Photoshop action, and then composite back into the background, which might be your best option here, although it still has problems.)

I’ve never found a way to perfectly create transparent edges with a painterly effect without doing that painting manually.

The good news is:

@aerocoptic

t’s video, so it would be tough to paint every frame.

Nope You only have to paint the object, just once. Granted, this method will also break from certain camera angles, but if you’re careful about that and have extremely good UV unwrapping, it might be possible to pull it off here.

EDIT: What I suggested above with a Photoshop action would technically be possible in Blender, but it would be easier in Photoshop. Render each frame as a flat black image with a transparent background. Make a noise map texture that’s 50% gray to white. Mix them together in photoshop- probably with the Overlay blend mode. Select all the colors greater than… probably 20% gray and delete them. Do this for every frame (you can automate this with a droplet or a batch script).

Then, render each frame as color, transparent background. Put that on top of the mask frame. Flatten the mask frame to black/transparent. Select all the black, delete it, use Content Aware fill to get the rendered colors. Do this for each frame.

Then, composite the image sequence onto the background

[aerocoptic]

Thanks so much. This actually gives me a bunch of ideas that are worth trying. Lemme see what I can do! Will update when I get a worthwhile result in case someone else is trying to accomplish something similar.

[jackmcveigh]

Hey guys thanks for the awesome thread, I’ve learned heaps from both of your examples!
Was wondering if either of you have tried combining this technique with lighting of any kind as opposed to just using the fresnel?

I’m currently trying to achieve something similar to create a 2-tone Sketch shader that would react to basic lighting in Eevee… at the moment I’m thinking the best way to do it would be to mix it with a ‘Shader to RGB’ node setup but I’m struggling getting the 2 to blend together nicely? any ideas?

This would basically be the intended outcome but with rough and painterly edges:

Thanks again!