Blender Repository Client script framework

Coding Python Support
21

the diagram can be openned with Dia, just type dia on google and choose the 2nd & 3rd results. if you have linux, dia is likely available from your package manager.
yes it will support textures etc… scripts and whatever we have plugins for. Download of files which are not Blender related support will be possible :
imagine someone has written some plugin to browse/download from/upload to, a repository which provides some scientific data that in a special format you like, you’re still to download those and put them where you want on your disks or in the Blender scene, if you have some plugin to load the file in Blender.
The latter aspect does not appear on the diagram or wiki yet, just in my plans.

We’ll have plugins to access websites…
and I think plugins for a and for b : a. doing work on the downloaded file before installing, b. installing

some example plugins (which do not appear on the diagram):

  • one for converting (if the user asks it) some data prior to passing it to other installation plugins) For example, download a .3ds or collada file from some non-blender specific repository, pass it through the related import/export plugins for conversion and expose the return the resulting data to the framework which decides what to do with it.
  • one for installing data in a current running scene if the data is a blender data block
  • one installing data on the hard drive (there should be only one plugin for that, which would know where the user wants/is used to putting files (ie scripts will go in .blender/scripts by default, but images could go in “My Docs/textures”) for the different mime types… or the favorite/last visited folders…)
22

Ok I found dia but I think I won’t install it. Could you provide a standardized format like SVG, PDF or an OpenDocument Format to make it easier for others to see it without having to install an extra application?

23

hey !!!
it’s here : )
http://blender-repo-client-framework.googlecode.com/svn/trunk/overallModulesDiagram.png
so as you see there’s no MIME (or file type specific) filter/install interface yet…
This diagram misses a lot of things.
The comment at the bottom right concerns the getPluginSpecs() function. getPluginInfo gives rather non-development-specific general compared to what getPluginSpecs()'s returns (a dictionary or some other object) which should be exploitable in an automated way by the core (with this, for example it could know if it’s useful to try and authententify with the server… etc)when it loads a module.
The things I’ve written for the UserDB is just thoughts of mine, but I guess you PapaSmurf have better or different ideas : )
Hey those explanations about the diagram should (and they will soon I hope) be on some wiki/txt file.

on the right you see the RepoClientInterface interface (name in italic) which is implemented (dashed arrow) by the various client plugins.
this needs updating (and some code !) I’m still wondering how the GUI will work along with the Core (as I want other apps to be able to use the client without modifying it : just add/setup specific plugins and write their own GUI). Maybe the GUI will work in pipe with the Core, or import the Core and the Core will run autonomously as a daemon and receive signals (instead of threaded calls) from the GUI…
the file is in the trunk : https://projects.blender.org/plugins/scmsvn/viewcvs.php/?root=repoclientfmw

if anyone wants to criticize or comment feel free. And if you want to participate by coding its cool.
http://code.google.com/p/blender-repo-client-framework/source/browse/trunk/overallModulesDiagram.png

24

so far a nice design i think, especially because it’s suited for more types of repos, and has both-directions api. That’s just from user’s point of view.

25

Do you want to discuss stuff here or do you have a dedicated mailinglist or forum for this? I’d be really interested in helping out but my time is limited.

26

Hi,
we’re in the process of having a projects.blender.org project space application approved. If our project is granted a space there, we’ll also get a mailing list so this could be the place for more technical talking.

Personally, I find this could be nice to update this thread regularly with development news.

27

the diagram above has been update (do ctrl+f5 to refresh your browser cache if you don’t see any change) (dia and png committed to the server).
Here’s the commit message (I’m lazy) :

updated the diagram with :

  • a running blender instance
  • different schemes implementing the RepoSchemeHandlerInterface
    interface
  • different download file handlers

I need to write some code now and set some deadlines.
Our project has been accepted a projects.blender.org space, so we’ll be hosted there and I’ll take down the google code repository we had.

on my todo list:
with no gui for now, and out of blender :

  • create access to a local hard drive repository (browse,download,upload)
  • reimplement blendermaterials.org’s API as a plug in.
    I have many exams these weeks, so I set the deadline as end of june (I know that it feels I’m taking my time… but I’ll have to take a work experience right after the exams).
28

i have to get my paper published as well, so my time is limited this month. sorry.

29

we’ve been approved a projects.blender.org space
our project page is now : https://projects.blender.org/projects/repoclientfmw/
the google code host space was deleted
to browse the repository online use :
https://projects.blender.org/plugins/scmsvn/viewcvs.php/?root=repoclientfmw
I updated the previous posts with the new link
I’ll move the diagram to the new repo.

the wiki page is now moved out of the sandbox to here :
http://wiki.blender.org/index.php/BlenderDev/RepoClientFramework
and a link to that page has been added on Blender’s Current Projects wiki page, in the Scripting section :
http://wiki.blender.org/index.php/BlenderDev/Current_Projects

30

Hey,

I am glad to see this project showing up here. I’ve been browsing through the code trunk, the wiki pages, and the suggestions posted here, and want to mention another parallel project that I’ve been working, it’s actually what led me into #blender on freenode and then from there to blenderartists.org.

What I am working on is, specifically, a ‘metarepository’ project – the idea is to have semantic metadata files and all sorts of goodies (schematics, .blend, CAD/CAM files, etc.) easily accessible from within the database, which has been dubbed SKDB. And alsoOSCOMAK Semantic Community on Manufacturing Artifacts Know-how.

At the moment we’re doing some examples of the system – simple 3D models that can be printed out by an inkjet printer and then folded back up into a 3D object. The point here is to illustrate the client (agx-get (kind of like apt-get)) and showing an example of how it’s supposed to all work.

Here’s a pepakura example. A ridiculously dense explanation of what’s going on can be found here. An email that I sent to ISDC2008 (the spacetech conference) can be found here – it’s probably more accessible than the ‘ridiculously dense’ explanation.

I’m interested in collaboration, maybe we can help each other out since these are basically the same projects. The SKDB plan basically calls for git repositories plus ikiwiki as a repo for all of the metadata files and “dot skdb” files. I see that blendermodrepo is using cvs, but that’s not a big issue. Using Joey Hess’ ikiwiki, however, allows it all to be flatfile. Kind of like the debian repository architecture.

And the scripts in the db are python + YAML for object serialization and a human readable format (think of it as RDF, slash XML, without the problems that XML displays) – which might mesh well with the blender serialization exporters/importers.

So, I can be contacted on most of the IM protocols as ‘kanzure’, I’m on freenode, and there’s a phone number up on my site. There’s also a a mailing list (#hplusroadmap) or two.

Thoughts?

  • Bryan

BTW, this post has links, but the CSS theme seems to make them not jump out. What’s going on ?

31

hi
ok
please let me some time to read all the links you’ve put in your post (not today, tomorrow maybe).
your thing seems to engineering oriented, but I must read more.
the repository for the blender repo client framework project is actually an svn one although cvs is mentionned somewhere.

in the “another thing going on that’s kind of similar” topic, there’s another project by Macouno going on for Blender : http://www.projectpetunia.org/docs/project_workflow
here’s a small summary from what I’ve been able to understand and read from it on IRC :

how project petunia will use DBBlender so that many people can download/update a scene, animate some character and reupload it. Here’s a schematic of their workflow : http://www.projectpetunia.org/docs/project_workflowbasically they’ll have an online portal + online api & a gui in blender to interact with the website & local dbblender part. they will need versionning, which is not the first aim of our own project I’d say.

32

Hi
I’ve looked more in depth at some of your pages and tried to understand what your project is about
basically, you’d want to create a sort of apt system to download engineering knowledge and build things locally.
For example, apt-get plans for some machine, and build it locally. And chances are that these plans have dependencies on techniques or other plans for your own machines to be able to build the machine… so the latter also have to be downloaded. I understand how big this would be for humanity and that it’s not useless at all at first sight for people to endeavour into such a big project. For I’ve learnt that the researches which accompanied the US race to moon brought many advances for many industries over many years after and ended up creating indirectly much more economical assets than what it costed.

this is indeed similar to the aim of the repo client project because :
in the latter, we download blender-compatible content and either download it to some place on a storage device, or add it to the running scene, using a special filter (for example if the downloaded data is not blender native (ie x3d, collada, etc…)). Those filters to convert downloaded files prior to adding them inside blender, can also be downloaded.
So : you can download data, and if it has dependencies for it to be fully usable, you may also download those depencies and install them first.

and now for politics :

  • the blender repo client project was intended to be for more or less graphical/artistic contents. So this is not really the case for the projects you’ve mentionned, or… your project has a wider spectrum encompassing ours.
  • I think that there would be too much dependencies if a project with as large as yours is to be tied to blender.
  • the apt system is nice to keep track of package depencies (and a required package may also require other packages…). This could be the case for 3d scenes where imported and linked objects rely on other files (for example you have a house, you link its windows objects from an external windows.blend file, and further this windows.blend file’s glass material is a link from a glassMaterial.blend file). This linking thing is too complicated to deal with for now to me and I don’t want to work on it. If a file uses external data, this data must be embedded or provided along with the file. The goal of our project is not at first to let people be able to download a full project (lots of blend files and other data, with possible interpendencies between blend files), but to be able to download single pieces of contents (models, materials…).
  • I want to let people be able to download small things or pack of similar things (ex : a pack of materials), that do not require more than one level of dependencies.
  • some deb packages ship with install scripts to be run… I wouldn’t want this for the repo client fmw project because that’s not our aim.
  • the sources.list files design and remote gz files for describing the contents of repo being accessed is nice. But many repositories out there currently don’t support it/don’t have a similar design, and why not, just have a web interface.

Those are my personal views.

Now there’s something that PapaSmurf made me think about on IRC. Do we want a framework on the user side where the users stores all the scripts to interact with different repositories, or do we want a central server with which the user interacts with a single script, and which interfaces all the remote repositories to the user?

If the latter solution ends up being implemented, then, we could provide an interface/protocol support for your system so that it can query in its own language the central server for content and download those.

33

Actually, I don’t know what to do.
But as to APT, it doesn’t seem graphical contents oriented though it could be (for example, for each item, add an url telling where get a preview image).
There also are things that seem useless to me such as apt-get update or upgrade except for scripts (there’s a blender scripts repository out there).
I’m afraid that it would be too big to put it alongside blender, rather than a single python script + a full python install (the python install is considered not to be a loss because other apps can benefit from it as well).
for unices users having a debian based distribution, the sources.list for graphical file would should have to put to a different place than the system ones. the fact of getting remote content should not require a root password, while the apt requires it always… apt would have to be ported to windows if it’s not already done.
And just rewriting another apt system which seems to be what you want to do, is not at all what I’d be ready to work on.
python (server or client side)or php scripting (server side)is fun, and this will bring other people to contribute if this is how our project is going to be implemented.

34

This repository client is very interesting! Especially since it is for many kinds of objects. My Spyder project (http://blenderartists.org/forum/showthread.php?t=119282 for an old post) is to validate/serialize parametric objects in Python. A parametric object behaves syntactically as primitive objects: a tree can behave as a mesh or a list of meshes, and a link to an object can behave as the object itself. I have serialized meshes and materials, and I would be glad to collaborate if your approach would be along the same lines.

35

Hi Sjoerd
I don’t understand how your project works very much
If your objects are serializable/easily put a single file / blender datablock then it should be ok to share them. If they rely on some scripts to work, then those info could be written as metadata using blender python’s objects ID properties.
If the objects rely on other objects it would nice to be able to know where to get those objects (imagine the main obj is sent online on a repository but the referenced objects aren’t or… the references to those are absolute hard drive references…)

To Kanzure : maybe I may have sounded a bit tough when writing my two last posts. I am sorry for that. I agree that the model of apt is ok. If you planned to have the project ship as a binary distribution, that’s not something I’d want. If you think you could convince me on working along with you, don’t hesitate posting.

Frank Karlitschek from the kde-looks, kde-apps, opendesktop.org (and many others…) portals has opened a portal for blender called blenderstuff at http://www.blenderstuff.org .
He’s also opened a thread here :
http://blenderartists.org/forum/showthread.php?t=125972

I had told him over e-mail that I’d support his GHNS API (a REST api) through the client framework. Feel free to contact him and volunteer to help him manage the blenderstuff website !
I know “well” the GHNS REST api because I’ve written a script to fetch data from it and parse the xml. Note that there’s no way to upload data over this API ! So if you want to contribute to add this feature, go and ask Frank here on the thread he’s opened : http://blenderartists.org/forum/showthread.php?t=125972 .
Here’s an “old” script : http://www.ece.fr/~schroder/SoC2008/files/restApiMappings.py which runs outside blender and requires a usual full python install. You have to edit goodLogin and goodPassword with your blenderstuff logins and passwords. This script doesn’t download any content apart from the xml pages.
This script is not very modular, not necessarily very clean… Also it is not up to date because Frank has removed the 404 error today (thanks !!) (the results from the online API calls would be retrieved by reading the xml accompanying 404 errors). I’ll edit it and repost it for you.

I’m going to make some inkscape mockups for the UI for a blender client.
If someone is willing to script this and/or review the UI first with me, this could be nice.

We have decided to try and move on with a single protocol between client and server and plan to have a central server which aggregates frequently links and metadata on objects from all kinds of repositories and puts them into cached files. The blender client would then ask the server to search/browse through these cache files, using : keyword search, tags and categories browsing.
If people want the central server to fetch their own repositories, they can write a python script which meets some API specification (a class inheriting the API class for example) and this script after being checked by admins can be plugged into the server and run periodically.

The server will have a REST API to let the blender client fetch metadata and links to object downloads easily. If possible this API will make an interface for people to upload to remote repositoires.

In a comparison with the apt way of doing, there are online feeds but the blender client doesn’t download them, the central server does searches in place of the blender client. While for apt, the client downloads caches/feeds from the servers, and the makes its searches offline.

Please let me know about you think of this.

36

I am sorry if I misunderstood… I thought that you wanted to add many different kinds of repositories, not only those with .blend files but also those with image files or non-Blender models. IMO, the server should just describe the exact format of the content and provide a link, and the conversion to Blender should be done at the client side. One possibility is to describe each content format as a Python class and define converters between them. You can wrap a categories and queries in Python classes too, and write converters that retrieve a listing or execute the query through the web. In these cases, Spyder can help. If you want to do these things at the server side, you will have no need of it.

37

I am sorry if I misunderstood… I thought that you wanted to add many different kinds of repositories, not only those with .blend files but also those with image files or non-Blender models. IMO, the server should just describe the exact format of the content and provide a link, and the conversion to Blender should be done at the client side. One possibility is to describe each content format as a Python class and define converters between them. You can wrap a categories and queries in Python classes too, and write converters that retrieve a listing or execute the query through the web. In these cases, Spyder can help. If you want to do these things at the server side, you will have no need of it.
hi
I agree with all that you’ve said.
Conversions could happen on the client side but I so no reason as well why it couldn’t happen on the server side, but for the fact that if it’s done on the client side, the user has more control/a view on what’s happening.
Instead of thinking about classes first, it would be nice to have a sort of database or parsable script annotation which tells what file formats blender can import (and we can consider that a user will never want to convert a downloaded file rightaway from any format to a format different from .blend / blender datablocks).

I’ve read 2 pages of your msword Spyder technical explanation file and gave up I apologize… it did not get the gist of Spyder with this file.
I’ve downloaded the first link of your Spyder tool post which gives something the Objects> menu and Misc> menu. I haven’t undestand well the Misc> think but the Objects> script is nice. I don’t really understand how Spyder can convert formats except for the fact that it enables to load sort of scripts to create parametric objects, and to save those new objects into a blender format.

For what I’ve been able to understand that would be relevant for Spyder in relation to the repo client project is that :

  • the Spyder project introduces a new kind of content format named .web
  • this format is not Blender native and requires that the Spyder “framework” (a “script”) be installed alongside Blender first to be able to import .web files into Blender.

Here’s in a list disposition, put neatly what would be on the server side and what on client side for the repo client framework :
central server :

  • periodically fetch feeds from remote repositories and store the aggregated results nicely
  • search in those feeds when requested by clients (search by tag, categories, search strings)
  • when responding client search requests : send lists of metadata + previews + direct download links (those links are direct to one or another content repository)
    blender client :
  • search things on central server
  • download result info
  • download files directly thanks to links it got from the results
  • save downloaded files on hard drive, and/or convert them and/or load them into the blender scene.

you said :

In these cases, Spyder can help.
… finally what did you mean by “these cases” ?

I’m sorry if I don’t understand everything quickly.

Jonathan

38

here’s a mockup start for the client
please feel free to review :
http://svn.blender.org/svnroot/repoclientfmw/clientBlenderUI.png
and why not edit the svg file which is here.
Hey
there are typo mistakes here and there … I’ll change that
I’ll provide explanation as well for each of the views.

39

Hi Jonathan,

I am the one to apologize… my specification is… difficult to read :wink:
I will try to explain it (excuse me for the long post)

Web files are not really a format, they are just files containing text like:


Material (
  name = "red",
  color = (1,0,0),
  filename = "red.jpg",
  ...
)

You could see it as an XML-like file that validates against a Material schema. Or as a Python object of the Material class. Doesn’t matter. It is easy to make a Python script to feed this kind of objects to Blender using Blender.Material. If you are dealing with .blend materials, you would have a different class/scheme, for example:


BlendFile_Material (
  name = "red",
  blendfile = "red.blend",
)

and you would use Blender.LibData/Library to deal with that.

To deal with remote materials, you could define a new class BlendFile_Material_Link.


BlendFile_Material_Link (
  name = "red",
  filelink = "http://www.blender-materials.org/download.php?mat=244",
)

And you could put this file on your central server, a bit like torrent files. The client then needs to know how to convert BlendFile_Material_Link to BlendFile_Material, which is trivial ( urllib.urlretrieve(…) ).

Or you could choose an different format:


Resource_Material (
  name = "red",
  repository = "blender-materials"
  id = 244,
)

which would rely on a somewhat smarter converter on the client side (one that knows how www.blender-materials.org works) to convert it into a BlendFile_Material_Link. You could also make files that contain categories or queries.

The diagram that you presented looks very good to me (except that IMO the filesystem should be treated no differently than any other repository) but it is centered on the code, while the data flows in between. I would suggest to define the repository in terms of data formats plus converters between those formats (i.e. the code flowing between the data blocks). You could use XML as your storage format or Spyder .web files or your own text format, doesn’t matter. But you might be interested in Spyder because it handles conversions automatically. If only “Material” and “BlendFile_Material” have a method toblender(), it will not crash if you call it on for example on a Resource_Material object, but it will try and chain every converter that you defined in a path to reach “Material” or “BlendFile_Material” and then call toblender() on it.

I hope that you can see how the Spyder way works. I am not saying that this is the only right way to do it, there are many right ways I think. Just do what seems the best way to you.

Sjoerd