Blender Nodes need AI nodes

In my opinion Blender Nodes appeals to people at large by its results but it is a barrier due to its complexity. This barrier is not only to new people but also to people that touch nodes for a project but then is several weeks making other stuff. I remember using Softimage Ice nodes and there was always a time of adjustment when i spent a time not using it. It is even worse for people that are not artists full time and only touch Blender for certain projects or just a part of their work.

I think AI can propel Blender Nodes to next level of usability and attraction because it uses common language.
These nodes would need to be able to get acceleration by the AI engines that will be appearing embedded in CPU/GPU’s.

This project should also need to be compatible with any AI in Blender outside nodes and open to addons.

1 Like

Blender is not complex.
It was possible to make simple 3D models during decades with older release of Blender.
Geometry nodes are a way to simplify creation of complex scenes.
It is a complex way to create simple 3D models. But if it is your goal, you can simply ignore them.

That is demand of user that is complex.
Making something in 3 dimensions means defining what you want to see in 3 dimensions.
Making such definition by a text corresponds to a long wall of text.

When a writer do a description of a building, an object in a novel, that can correspond to a long paragraph of several pages or that can be few words.
That depends of its intentions, of what he absolutely wants to communicate to reader.
But almost always, there are enough blanks that can be filled by readers to let them, imagine what is described, according to their preferences.

An interpretation done by AI means that AI is filling the blanks. The result may differ a lot to what you expected if you are not the one who trained it.
If you are fine with a low expectation, you can work quickly with AI.
If you are not, you can pass hours to refine your request.

The reason why there is an interface, with a 3D viewport, is to have an instant feedback of adjustment done to what is displayed on screen.
The tool is designed to take, as first input, creation of simple 3D forms, and immediately offering ability to move, scale, rotate them, combine them or complexify their surface, their mesh.
Since Ngons, a face of this mesh can correspond to any polygonal shape.
You can make a face that corresponds to an L, a T or silhouette of a mickey mouse head.

Is it simpler to cut shape you want with knife tool, or to explain what object of mass culture reference is closer, to what it evokes to you, with a potential misunderstanding from AI ?

AI is not magic. To have a pertinent use of AI as nodes, you have to define precisely to what task you want it to be assigned.
And it has to be more variable than a nodegroup or a versatile rig, that can be stored as an asset and immediately reused.
To train this AI, you need to have thousands, millions of examples of expected results ready.
The language, AI will understand, will correspond to how those examples were tagged.
That is not a common language.

It is possible to imagine an AI picking 3D models from libraries, creating a GN nodetree to combine them, creating a correct light set-up, placing camera, creating a rig, and producing movements, according to sentences written by a user.
That would not change the fact that : if user wants to tweak resulting scene manually ; he would have to understand what he is tweaking, when he is pulling a gizmo in 3D Viewport.

The same way there is no “Make A Beauty Render” in UI.
Writing that in an AI prompt will not be a guarantee to obtain one.

6 Likes

Geometry nodes are a way to simplify creation of complex scenes.

A precision Zeauro note I am not talking only about current Geometry Nodes but the future “Everything” Nodes.

I think it is probably best for the discussion trying to perceive where the AI advantages can benefit Blender. At the moment in my thinking i would say this:

Use AI to do your initial setup.
You can have nodes for background creation, cloud creation <— how to split a scene description would be important
Variations AI
Themed AI - Christmas, Beach etc
Animation AI: say: birds flying, people walk 10 sec turn right and then starts running
Crash/destruction results.

The AI should be able to take existing inputs from the scene, so if you have a car model in the scene from a 3D database, you give its name to the AI- do not needs to be complete - for exemple if you have a Ferrari in the scene. You just say at frame x start change speed the Ferrari to frame y until it reaches 200kph and follow road “object scene name” . Now it is open ended if the Ferrari is static in frame x or already moving.

Of course in any of this the Blender user should be able to manually change afterwards - and ideally this should not break the AI connection - This is where a AI nodes would be advantageous, you can change the AI node after making modifications and the modifications you did if they do not break the pipeline would change accordingly.
Inversely you should be able to take a specific AI creation from the AI pipeline to not be affected by AI anymore or only when you choose.

I also hope that AI can be used to name automatically objects we build in 3D and that the AI creates.

I know it is growing an improving all the time, but AI reminds me of those life hacks for peeling garlic:

“just put the cloves in a vessel and shake them around, the skins will just fall right off!”

This is true, but it’s only true for the cloves that were already easy to peel. The cloves that are more challenging to peel you still have to do manually.

AI is really good at easy stuff. I’m sure you could set up an AI system to mask texture x with texture y, or offset the texture coordinates by texture z.

But that stuff really isn’t that hard. When you want to do something that is more complicated, it will just kinda fail.

I’ve used chatgpt to generate BPY scripts, and it’s pretty good at simple things:

image

But once you start getting into complex things, it quickly gets confused. I use it mainly as a template generator. blender has a bunch of templates built in, and these AI generated scripts can serve a similar purpose.

I know it will continue to improve, and it is a fantastic resource right now, but I don’t think it will be able to grasp nodes well without a ton of human generated data to train it on.

11 Likes

Yes that is another approach, AI learning how to use Blender nodes. In my initial post it was AI added side by side with nodes; use AI within nodes when convenient, use other nodes itself when it is more convenient.

Is AI really writing scripts, or is it just stealing templates from its training data by evaluating the original description? I once read a post somewhere where the AI even with several user hints wasn’t even able to correctly indent the code - in Python, where indentation is everything. Obviously the AI didn’t really “understand” what it was doing.
I’d need to see the results for anything more complex than a demo script…

2 Likes

From what I understand at the moment AI basically collects information and examples of things that have already been done by traditional methods and produces variations refining the results.

Somebody has to provide the original building blocks (nodes made by the devs), and somebody has to produce examples for an AI system to gather and refine (US we that use blender).

This is how the python script writer works it uses the python language (which was written by python devs) and many example scripts (written by users) to produce its (sometimes dubious) results.

No it copies us!

To me this sounds as if you are looking for a solution that can generate everything you are looking for, somehow.

There is just a tiny issue, how do you train such a thing? Where are you taking the data from to train it? How are users supposed to use it within Blender, how should users interact with it?

1 Like

I think @zeauro made good points, you can also try to play with this : https://withpoly.com/browse/textures
When you have very specific things in mind it’s not that easy to get something appropriate.

In the end, I’d better rely on already made assets. There are a lot of site for textures, 3D models , materials that you can pick from.

But as you probably know, if you don’t know how to model / texture / render yourself your work is still going to feel cheap.

It’s an interesting point, I could say, sculpting looks super fun but what is blocking people is that you need some training to make something good.
Just like guitar, what is blocking people is that it generally take a lot of time to play something correctly.

isn’t that’s just how thing works ? Mastering a craft is great but the greatest is what you learn about yourself while learning it.

And in the end it’s just like with python scripts made by chat GPT, sometime they work, sometime they don’t, but if you don’t know how to write code yourself how are you going to fix it ?

going back to that site : https://withpoly.com/browse/textures
the most impressive feature is the fact that you enter a prompt and get a new texture, but it’s probably the most useless outside of toying.

But other feature like upscale, make seamless, or export pbr from an image are probably much more useful , but yeah , they sound much less attractive for sure !
I believe much more in these kind of tool to assist artist rather than replace them…

3 Likes

Yes, but not all learning has equal appealing and not all learning have the same difficulty. There are degrees and also there learning with breaking points/critical mass or a linear learning that can be achieved.

For example look at appeal the Mograph had in C4D when it appeared. I think systems that are more successful with wider appeal tend to be more linear leaning than based on critical mass. You learn and you keep getting rewarded and you have energy to learn more. That is also one of the reasons i think AI can be helpful.

1 Like

My bet is that more and more we’ll see some uber node groups that non-technical artist can use.
In fact, the problem is that all these nodes things are difficult to learn, and they are coming to blender.

We don’t want blender to become as complex as houdini to operate , so how do we do ?

But to solve that AI isn’t the only solution, or at least that’s not the one that I’d bet on !

But having presets, better interface to operate the tools is what I believe in !
And I’m also quite confident on the blender foundation to find the good balance and keep blender be accessible to many.

There will always be a learning curve, but it can be evened out by ressources , addon …

If you look at shaders it’s basically the same problem, you need to know how PBR works, doing a bit of procedural texturing, it’s quite challenging to make good materials inside blender.
So what do people do ? they either use substance, buy some shader pack, or manage with PBR textures that we can find online.

It’s so easy that very few people I work with really know how to build a material from scratch , even after years working with blender professionally :smiley:

1 Like

Needless to say that people jump on every business opportunity, if it’s possible to help people to waist their money to be “more creative” rather than spending their time learning uninteresting vector maths or rule-based modeling, we don’t want to make them suffer at all :smiley:

It’s already happening : https://blendermarket.com/products/curvify
Stuff like that, frankly most of them are in fact good learning exercises that people should try to recreate in order to learn, and it’s unclear to me how useful they could be, but seems like people are happy to buy them anyway … and if an AI is able to generate such result it’s quite impressive but I’m having big doubt … And it’s probably on these kind of setup that it’s going to be trained so …

Nodes are not a very efficient way of work for all types of scenes or for all parts of creative process. For example we have scatters where the artist is just changing the “seed” and painting to get variations but what if we have something like a Nvidia Canvas 3D, in that instead of current 2D landscape it would build a 3D landscape scene by “painting with colors”, in that even specific brushes could be created by the artist.

1 Like

Don’t need AI for this- user defined brushes are already in Blender, and there’s a planned overhaul allowing for even more control soon

1 Like

Sure ! Just to be clear, nodes are very versatile but when BF talk about “Everything Nodes” which is more a concept, it’s not about having only nodes everywhere but more the ability to switch to using nodes , just like it’s possible to use nearly every functionality with python, or keyframe / drive every value.

But I’m pretty sure they don’t mean it’s mandatory to use them. Sorry if we are on the same page, but I didn’t completely understand what you mean.

And about the Nvidia Canvas in blender, well that’s great, and having something like that will probably comes from an addon. Just because the way I see it it’s too specific to be tackled by BF.

Most of the time they work on big , very generic features rather than specific functionalities.
So they make sure what they add is likely to be useful to many.

At least that’s my understanding of their strategy !

1 Like

So, you’re saying Mograph had a lot of appeal (and it did, it’s a great feature)… which helped people learn more (agreed). Then you add:

And all of that Mograph success happened, without AI coming to the party at all.

So…

2 Likes

The brushes i just meant in context of a Blender Canvas 3D. In Nvidia 2D app you can’t edit the brushes not feed them data so you are stuck with system that while is great it is not flexible enough for an artist.

flexibility comes at a great expense, particularly with AI. Given the limitations of a 2d canvas, and the resources of the biggest player in the AI industrial complex, that flexibility is what they came up with.

Check out some of the Geo scatter demos, their biome system works a lot like the handful of brush options you have in canvas, but they are fully dynamic. You can make your own and populate them with whatever objects you want. With a brush stroke, you can have a full 3d forest appear. With another, you can carve a path and cover it in bark chips. Using those custom curve geometry node assets posted above, you can draw a fence line.

We don’t need AI when we have real human intelligence spending a lot of time and effort building tools so that the rest of us can build complex, believable scenes with very little effort.

I’m sure at some point in the future, AI tools will be even more powerful and we can consider integrating that tech into blender at that point, but for now it’s just speculative. Would it be useful once the kinks are worked out? Probably, but blender doesn’t need AI nodes right now.

3 Likes

I will never understand why the most advanced feature of human beings needs to be replaced…

If someone uses any tool for anything… then only practice, knowledge and experience leads to mastery… – specialization was the main reason for building up whole civilzations… but now everybody wants to do everything in a blink of an eye… for free…

5 Likes

Consider a word processing application.

If the program makes it difficult to create a new paragraph, to select text, to make a sentence bold, defined incorrectly spelled words, or to print a page… We need to determine why. Perhaps the program cannot do this very well. Perhaps it does it very well, but the user interface is so poorly designed that no one can find the command. So to correct these issues, we can make the program easier to use, or correct the code so that the features work correctly.

If the program is difficult for many people to use in general, once again we determine why, and change how the program works to make it easier in ways that are reasonable.

If the program generally works well, but someone says that they are not a very good writer, and have difficulty coming up with good sentences that communicate their thoughts, then the problem is not the application.

In this case, the problem is the user. Perhaps they should take some classes on writing, or maybe read more books and study writing techniques. And in some cases, this person will never be great at writing at all.

This is not mean that our word processing application now needs to add an AI feature.

2 Likes