I was thinking the other day about blender and AI and how useful it would be to have optional AI utilities implemented in blender as nodes, here’s a few examples
a node for up scaling images (render or textures)
imagine being able to just plug a node in the compositor and upscale your render result from 1080p to 4k, or have that node in the shader editor for up scaling textures
a node for creating seamless textures, there’s a lot of ai algorithms out there that can do this so it would be amazing to have a node in the shader editor that can do that
And last, a general universal stable diffusion node where it has a text box you can write prompts in and use the result in the shader editor or anywhere that takes an image input, so imagine you want to create a wood material, you plug the stable diffusion node in the color input with the prompt “wood texture” and then an upscale node in between
all of the things I’ve mentioned today are possible and available but require jumping around between software to software, having it all in blender would be so powerful and convenient, but yeah it’s definitely easier said than done, but it’s an interesting discussion
Pretty simple for someone that cares to do it- Blender python is an incredibly simple API. You should try some of these ideas out and let us know how it goes
It has already been demonstrated in Blender (through OIDN) that AI can in fact do a good job in making certain tasks a lot easier and faster if it is at least guided by statistical data.
For your Stable Diffusion example, I do not know if any of the solutions are yet considered good enough for the creation of seamless wood materials for Cycles or Eevee, but perhaps it could be done if you used the existing wave node or a pre-existing wood texture to provide a skeleton for the AI to work on top of.
I wonder how long it will take until an AI could technically learn a whole piece of software like blender, and then be able to create using text prompts.
Yes I never really enjoyed using stable diffusion or dalle to create art, but I find them extremely useful for removing some tedious aspects of the creation process, like hunting down good textures and not having to wait for long render times, ai can be a great assistant
Yeah currently stable diffusion can only do 512x512 images without eating up all your VRAM, but I could see resolutions and hardware requirements going down extremely fast in the coming months, and switching the ai around to a better newer version would be trivial if we have a universal ai node already in place, the cool thing about stable diffusion is you can train it to be more specific for your needs, for example someone could create a version where it’s specialized in creating seamless textures only with the normal and roughness maps separated
I don’t have any programming experience unfortunately, people are probably working on add-ons already so we’ll see in the upcoming few months, i was just wondering if the blender foundation is interested in going down the ai route for useful utilities
Ton has expressed some interest on Twitter but he’s not really spearheading development anymore, and I haven’t seen any indications the official developers plan on doing anything with it. I wouldn’t count on it, at least not until there’s more definitive IP laws in place for all this stuff. The Blender Foundation has been very clear that they won’t explore anything new if there’s even the slightest possibility of legal trouble with it
In this case, it will keep the devs. away from an entire genre of shiny things that would distract from shoring up and polishing the many tools and features that could use it right now. Even worse, it would get in the way of fully realizing projects such as Geometry Nodes (ie. expanding it to physics and particles). I am not saying that the devs. would absolutely go full bore in padding the feature set with this, but it will make it more difficult for the community to push for everyday things that help like improving performance.
The last thing the BF wants to do is fall into the same trap Unity Technologies got caught in, which is letting the emphasis on AI and machine learning act as a replacement of more manual, but very effective tools (because why expand on dusty old things like navmeshes and behavioral trees when you can make a neural network, which in turn requires devs. to spend a huge amount of time playing the AI objects to train them and evolve them for every possible layout and corner case). See some of the machine learning Youtube videos, you can go to 100 iterations and still have a statistically significant chance of treating the player to a complete turkey of an object.
I agree- Blender is nowhere near polished enough with existing features to have a massive new feature implementation, and it seems like the devs agree as well
Yeah it’s probably going to be an area for add-on developers to explore, especially since there’s a heated discussion around the ethics of ai trained on artists work without consent, that can of worms is probably something the blender foundation wouldn’t want to touch
but there’s super easy wins like ai upscale that are in the same realm as ai denoise so it wouldn’t be a huge step in a new direction
