H.264 and "round corners" (II): analysis of problem! - but now looking for workflow?

Support Compositing and Post Processing
#1

A couple of weeks ago I asked if anyone knew a solution for an H.264 encoding problem I had (and still have) and I got a first couple of helpful answers, but no final working solution yet. So I investigated further, asked in the ffmpegX forum and found a user who is very knowledgeable with codec stuff and who was able to analyse the problem for me, but now I still need to find out how-to implement what he tells me.

The short version:

“Encoding with 4:4:4 subsampling rather than 4:1:1 would eliminate both these problems. But I’m not sure how widespread the support for 4:4:4 h.264 is.”

The long version:

the thread in the ffmpegX forum (including a .tga sample image and a 1 sec. H.264 test) can be found here (forum.videohelp.com).

Since I am still learning about the finer details of chroma subsampling etc. I would need a simple to understand, practical solution for this, some kind of known workflow to achieve:

“Encoding with 4:4:4 subsampling rather than 4:1:1 would eliminate both these problems.”

Can this be done with blender…? Maybe with the Compositing Nodes? Which nodes would I need? Or does anyone know of another Mac software, preferably freely available, that does this?

The thing is that the H.264 encoding artefacts/problems I have are so bad that I can’t publish my soon to be finished short like that, it just looks too bad. Very frustrating. I would also use another codec, but it would need to be as widely used as H.264 (not just by one company) and of course bring good results (and: any MPEG based codec seems to have the same problems that I have in this case with H.264!).

Thanks for any tips…!

#2

You have to weigh up what your delivery method is. For online content, the H264 sample you posted on the link looked ok to me. 4:4:4 probably won’t be supported in many places:

Baseline Profile (BP): Primarily for lower-cost applications with limited computing resources, this profile is used widely in videoconferencing and mobile applications.
Main Profile (MP): Originally intended as the mainstream consumer profile for broadcast and storage applications, the importance of this profile faded when the High profile was developed for those applications.
Extended Profile (XP): Intended as the streaming video profile, this profile has relatively high compression capability and some extra tricks for robustness to data losses and server stream switching.
High Profile (HiP): The primary profile for broadcast and disc storage applications, particularly for high-definition television applications (this is the profile adopted into HD DVD and Blu-ray Disc, for example).
High 10 Profile (Hi10P): Going beyond today’s mainstream consumer product capabilities, this profile builds on top of the High Profile—adding support for up to 10 bits per sample of decoded picture precision.
High 4:2:2 Profile (Hi422P): Primarily targeting professional applications that use interlaced video, this profile builds on top of the High 10 Profile—adding support for the 4:2:2 chroma subsampling format while using up to 10 bits per sample of decoded picture precision.
High 4:4:4 Predictive Profile (Hi444PP): This profile builds on top of the High 4:2:2 Profile—supporting up to 4:4:4 chroma sampling, up to 14 bits per sample, and additionally supporting efficient lossless region coding and the coding of each picture as three separate color planes.

Quicktime encoding supports main, baseline and extended. I would just try using a higher bitrate and possibly multiple pass encoding.

If you absolutely have to post your content online without losing any quality, the animation codec would work. It’s lossless like PNG and doesn’t take up much space if you have a lot of solid colors in your work. If you keep the resolution low, playback speed should be ok.

Nope. You can do some manipulation on YUV but I doubt it will help the encoding. One technique for getting round issues with green screening DV is to blur the lower resolution UV channels but that’s for getting more information out of lower quality encoding not for making the delivery format better.

If possible, can you render a short PNG image sequence, open the sequence in quicktime. Save as a self contained movie and then upload that. I’ll check out some encoders and see if I get the same artifacts.

#3

Note: I can’t attach the PNG sample .mov you asked for and also the H.264 test from it that I prepared… (error/invalid file…) I could mail it to you via gmail or so…?

For some reason I can only select main and baseline, extended is grey/I can’t tick the checkbox. But I always use main and multi-pass… Attached a test I just did from the PNG sequence you asked for (I also attached that one) with main, multi-pass and a very high kbps rate of 16384 - the problem is that the artefacts don’t go away with higher bitrates… (The why is explained by that user in the ffmpegX forum thread.)

From what you write it looks like I would need the High 4:4:4 Predictive Profile (Hi444PP) profile. Do you know which software encodes with this profile and if QuickTime and the VLC could play back a movie that was encoded this way…? If not it would not make sense for me, since I currently publish all my works online only (as higher rate H.264/MPEGs plus the flash video option)!

I just rendered the same 25 frames long test from the PNG sequence once more using the Animation codec and it is almost 14 MB large… The whole movie is 4 min. long. That would be a very large file… (The same 25 frames .mov with the PNG codec is only a bit over 3 MB - which would also still be too large for this 4 min. video…) Looking at all my other online shorts I was hoping to get a 30 MB - 60 MB file (with H.264) for the whole animation. Even 100 MB or so would be OK if the quality is really satisfying.

It is strange that H.264 does not work as expected in this case, I am only starting to understand the reasons for it but still hope that there is a way to do this, possibly with the High 4:4:4 Predictive Profil that you mention…?!

I attached it!

And thanks for having a look at this!

#4

Since I can’t see how-to upload the test files, here just one PNG test render (I can’t convert it to a .mov sequence because of the upload limitations.) But maybe this is enough to have a look at the problem…?

The H.264 sample that is linked in the ffmpegX forum was encoded with 8192 kbits and x264 (it is a bit darker than the normal H.264, no idea why, but the artefacts are the same). The other H.264 test that I can’t upload here does not look much better even though I doubled the bit rate (and both were encoded with multi-pass). It really seems to be a problem with the colours as the user in the ffmpegX forum says…

Attachments


#5

I’ve PM’d you my email address.

Don’t use the option quicktime to mpeg-4 when you export mpeg-4 btw. You get better quality if you export movie to quicktime movie. Then choose the mpeg-4 codec in the list and choose automatic data rate and set the quality with the slider. (I think that changes to variable bit rate from constant).

You can then put it through the mpeg-4 exporter and just select pass-through for the video codec if you want to get an mp4.

I don’t know of any consumer software that would do that. From what I see, it’s mainly intended for production, not delivery.

Yeah the trouble with animation is that it depends on what you put in the movie. If there are a lot of gradients and things, it doesn’t compress nearly so well.

H264 can be a bit of a nuisance sometimes. Watch out for desaturation too - load the original and the compressed one side by side and check if the colors are washed out or darker.

#6

I’ve tried all sorts of codecs on that video and they all seem to produce some sort of edge problem. I even tried Apple ProRes, which has a 4:4:4 chroma sampling option and it still produces a bad edge.

Even blurring, contrast adjustment etc don’t seem to help.

I did get a fairly clean image out by doing this:

open the PNG sequence in quicktime
export and choose movie to quicktime movie
choose H264 (I meant that last time)
medium quality
automatic data rate

but there is a black edge in it.

This should take the size down to about 160k. This would give you a 4 minute movie of about 40MB assuming similar content.

It washes out the colors but you can adjust the brightness and contrast in the filter settings.

I’ll experiment with it some more and see if I can get anything better.

I find that most codecs are pretty bad at handling rendered content. I guess they were designed to work for film footage but you’d think they’d be able to do both properly.

#7

Thanks a lot for testing this - I am also still trying a couple of things with it. At least I know it’s not just me, but there really seems to be a problem with 3D footage and encoding in general. I wonder how companies like Pixar do this? I’ll have to find out.

I will post more of what I found out (or not…) on the weekend.

#8

I suspect it’s because rendering allows us to create much cleaner (less noise and blur), detailed (detail that we want in that is), sharper edges, wider dynamics (colour value ranges and combinations) intense (absolute colour saturation) signals. Guessing here, of course. It’s been over 20 years since I’ve touched an actual tv camera.

Also, I don’t know if it’s still true or not given recent changes in TV tech, but computer monitors have historically been able to create much more saturated colours. Common rule of thumb on Amiga video was not to push the computer-rendered content above 75%(?) of RGB brightness. Some of that was due to VHS limitations as well.

#9

• This is an interesting observation… And maybe part of the answer for this encoding problem: there are other parts in the animation that have toon edges - no problem there with round corners and H.264 encoding! So I tested this and rendered the part where I have problems once more but with toon edges: it would actually solve (or rather circumvent) the problem - but maybe does not quite work on the artistic side of things because the toon edges are visible as such and I’m not quite sure (yet?) if I want this in that particular part of the animation…

But maybe there are other ways to render this differently so that it works a bit like toon edges, solves the H.264 problem but you don’t really see it…? (One problem with the toon edge solution is that I might need at least two differently coloured ones that I’d need to be able to apply on different objects in the same shot…)

• Another thing I found is that if you view the H.264 encoded movie with half of its size the artefacts are gone, but for some reason you can’t just e.g. down encode from a 640x480 render to a 320x240 MPEG-4 and get the same benefiting effect - it still looks as bad as the larger one.

So I’m asking myself: is there a way to force an .mp4 1280x960 movie file to be played back by default as 640x480…? This would be a simple workaround, but it would have to work in all players - or at least in QuickTime and the VLC.

• I found that the TGA and the JPEG 2000 codecs might work, although with rather large files, but the main problem here is that the VLC (at least on the Mac) does not play back movies encoded with TGA and JPEG 2000…!

• So maybe changing something in the rendering (or directly on the borders of the 3D models) could be a solution - the thing is that the uploaded H.264 video will once more be converted - to Flash - after I’ve uploaded it to my blip.tv account! And this probably will then be what most people will watch…

So I did another test and found that Flash has similar looking artefacts, just for some reason they seem not as bad as with H.264, but that might be because of the overall quality of Flash and what you’d expect from it…?

To show the toon edge solution see the attached toon vs. no toon render - both converted to Flash in this case. (An H.264 conversion would get a similar benefiting effect from those toon edges.)

I now wonder how this will work out in the future when more and more 3D artists will just publish their works online: will we render differently to make it work for a particular codec or will there be other (codec related) solutions…?

Attachments

Flash toon and no toon.png812×379 208 KB

#10

Your problem here still hasn’t changed. Your colors, particularly the green, are over saturated. At least a 10% reduction in your green value should help here a little. However, the plain and simple fact of the matter is that compression does not typically play nice with high saturation or solid blocks of color. You will generally always have some form of artifacts happening at the edges. You can attempt to mitigate this by reducing saturation, or using edge detection and letting that define a blur mask for those transitions, or you can increase your color sampling and bitrate at the expense of file size.

The issue is not really with 3D or animation, it’s with color choice. Television producers can get really bent out of shape if the person on-camera is wearing a solid white or red shirt, especially on a vastly different-colored background. And in TV, you end up also having to worry about moire effects on tight patterns. You must learn to produce around these limitations. You were asking what other animation companies do? That’s it. They recognize limitations and produce around them… or on some rare cases where it may be possible, write programs to try to reduce the limitations.

As it stands, the only really effective ways of completely eliminating your artifacts are formats that have file sizes that are too large or formats that most players won’t play. You intend for these animations to be viewable to the public, correct? My suggestion would be to work with a combination of reducing your saturation, slightly blurring your edges, and choosing a codec and compression format that’s ‘good enough’. If this were a commercial project, I’d suggest simply delivering the raw stills and letting it get sorted in post.

#11

Well the artifacts don’t really come from too high saturation, they only become increasingly visible with higher color saturation, reducing it a few percent will not magically remove the problem.

What you need to avoid are borders with same brightness and vastly different chroma (as bright green and orange…). Reason has already been mentioned, chroma subsampling. The human eye is more sensitive for brightness changes than for chroma changes, hence all MPEG codecs (aswell as JPEG) by default encode chroma channels in a lower resolution than luminance (MPEG aswell as JPEG work in YCbCr color space).

The “intended” way to work around this especially with H.264 is probably increasing resolution…“HD” video/tv, the latest buzzword we all heard thousands of times…
Forcing chroma encoding at full resolution may not give you much compared to higher resolution, possibly not even signifficant CPU-usage savings on playback.