Blender Repository Client script framework

Hi,
since my SoC proposal’s been rejected (I have got no problem with that) and since I don’t mind developping this out of a summer of code…

I propose to you that a Blender GUI + client interface framework be developped for Blender to connect to any online repository for any kind of Blender compatible content (materials, models, textures, images, scripts, nodes setups, logic bricks setup… basically, a lot of what you can see here ).
I am speaking about making a client for repositories, not a blender-specific repository for now. Though… if you’re keen on the latter, someone at opendesktop.org can* probably deploy a template site like that used by kde-apps.org or kde-look.org for you and you could customize that a bit for Blender for example.

  • I’ve been proposed personnaly but declined. Ask me if you want to webmaster such a website or go to the ghns archive. See this.

Repositories generally are web pages which do not provide an API (such as SOAP and REST), but could and have a specific folder scheme to arrange contents. Hence the idea of an updater framework to which would be plugged small client scripts to wrap those different websites.

I know for now that there are repositories/websites up for : materials, models, and python scripts (at least here). And I know that there’s a Blender python client with a GUI for the former repository.

Would you want to take a look here for a more detailed design proposal and tell what you think of it ?
I should write a complementary document to list the classes’s various methods.

I’m ok to develop something on this way over time. But I’d be pleased to have someone accompanying me. Also there’s no project page and if you think I seem serious enough, there could be an entry on thisblender wiki page . It would just be nice to have a wiki… or maybe we could do the thing on http://code.google.com .

for a small to do list :

  1. make final decisions on a detailed (not like the one above) class diagram design :
  • decide on the model design (ex : will we choose to see repositories as folders + files side by side and so recursively ?.. does the model permit search capability, tagging, marking… when repositories support those?)
  • etc…
    maybe we could have input from developers of project such as APT, yum on the diagram/design we’d plan to use.
  1. write and test : the RepoCore class + one RepoAccessToSomeWebsite client which can interface to the former
  2. write the GUI for that.
  3. write other RepoAccessToSomeWebsite clients.
  4. test & distribute. Have a central place for the script to fetch for repository addresses & to download their relative client engine to use if those are not yet installed.

To help with step 1. it would be nice, first to list all the blender repositories up worth writing a small client engine fitting the blender repository client interface to fetch/upload things from/to those websites. The various interesting data types could be listed as well. Note that if importers exist for some file type (for instance COLLADA), such files could be downloaded into Blender if they’re filtered through the importer, or just downloaded onto the hard drive. This makes that “foreign” repositories not necessarily blender-related could be used.

To help with step 3. GUI research can be done, for example by looking at the different solutions used in software we know. I’ve started taking screenshots here but I’m lazy for now to go further, edit, highlight what’s noticeable…

I think that this project is feasible with time and patience…
I’d be ok to work on that, though could people confirm that they’re ready to help ?

Fantastic! I would love to work with you on this; I have some experience in systems architecture. To ease the effort, may I suggest using MySQL as a datastore, for example, the metadata- keeping track of what is where, trusted sites, login/access data, local preview picture, and so forth, and the actual object, so that the data (meta and object) is easily maintained and kept current (I can see a future google type engine for finding stuff distributed around the world), running in the background. It would also allow easier interfacing to an in-house digital asset management (DAM) system.

hey
JesterKing is working has made the concept of oneBlenderFile=oneSmallMysqlDB
(that’s the DBBlender project) work. He was interested in my SoC proposal so I guess he’s still interested in the repository client idea now.

and so what you say about mysql somehow already exists through what JesterKing’s done. There should more info on DBBlender here : http://wiki.blender.org/index.php/BlenderDev/DbBlender
Do you want to speak with him about your ideas ?

I’m a beginner so to say (still in a degree studying computer science & electronics )… so well if you want to design some graph or plans or text specs on the contents of your db (which you described briefly), that could be cool.

As to me, I’d like to focus on the client network standard calling interface design.

As far as db data is concerned on this, you mentionned it, it would be cool to be able to associate in a db a web adress (the domain adress) with a name and password.

If blend files become small mysql files, then a small blend file reserved only for personal settings/data could store the passwords & logins.
What do you think if those data were written in the .B.blend file or so ? or another such file.

I really know nothing about database designing so if you’re happy to work on that that’s cool.
There’s xml parsing capabilites in Python’s full install which I used for a REST API client for opendesktop.org (I made that when working on my SoC proposal, thinking about setting up a client & a repository on the opendesktop portal).
To me, having to fetch pages means using urllib or urllib2 (or httplib) which implies having a python standard install. I doubt there’s sql functionnality in the standard install.
Do you think that we should add one more dependency (to that of having to have a full python install) by using mysql ?

If such a repo client framework project ended up well, it could be shipped with Blender’s scripts. Maybe the script should tell what to install when run if those are not detected, and do nothing till requirements are not met.
Anyway, if something becomes very useful as has been the case for ffmpeg, the Bf could include the necessary libraries in Blender.

jonathan

Hey
I’d like to start working on the kind of files we might encounter… and write some object specs like what kind of object can have what attributes.
This I could do with Dia… but a wiki would be fine for now… I would like the document to be open to anyone to read and centralized.

Is that possible to have a place on the blender wiki ?
I have rights to edit wiki right now… But I don’t dare to edit anything for this project for now.

Or I could just post things here. But things are likely to get spread between messages.

Jonathan

Hi
I filled in a form on projects.blender.org to create a project space (svn + project tracking tools etc…) for this.
I had to take a definitive decision on the project unix name, I think I took blendrepoclient or so.
The project proposal has to be approved (under 72 hours the page says) first.
I’ll tell you when I get update.

we can create wiki.blender.org/index.php/Sandbox/RepoClient as a start page. Let me know and i will start the page. and yes, I can design the database, no problem. an XML file is fed then to MySQL to create it. MySQL utilities make it easy to make most data changes (adding a new site, updating password changes, etc.) so you don’t get dragged down writing all that maintenance code.

Random thought: The BlenderLibrary script had a section of code that automatically created a preview picture; i thought that was pretty neat and a good feature to have in the client browser (created with the model is first accessed).

  • ok for the wiki in the sandbox. I’ll put some dummy text to create the page. When I get feedback for the Blender project creating on projects.blender.org, I guess we could ask on bf-blender if we can do something in the dev wiki section.

  • as to the picture generation from the blender library

Random thought: The BlenderLibrary script had a section of code that automatically created a preview picture; i thought that was pretty neat and a good feature to have in the client browser (created with the model is first accessed).
this is good in the case of uploading to a server, this tool/part of code could be used to generate a preview for a new piece of content packed along with title, description etc…

  • concerning the conversion from xml to mysql… you mean maybe that you’d code some python that takes an xml formatted in some way and inserted it into the mysql db
    Well, rather here’s how I see it. The problem is that some repos online are just bare web pages with no online API to call (ie SOAP or REST) and which sends xml. So would be nice, would be that people just write html fetchers and parsers for the repositories, which could either :
    a) convert the fetched content to xml (with a defined structure of tags to be used, embed how, with what attributes etc… (that’s the principle of xml dtd or xml schema)
    b) convert the fetched content to python objects (ie create an instance of existing classes provided by the library, and pass the different details in the constructor, and set some attributes by hand/using helper functions).
    Here’s how I’d see things :
    step 1)websites can return the xml in the format of a) if they want it, or else, people have write wrappers/converters that return either something in the a) format or in the b) format.
    step 2)anyways, what comes from step 1) will end up formatted into format b). So things formatted in a), will reparsed to fit in format b). And what’s in b) will remain so.
    step 3) it’s easy to convert the objects from b) to mysql.

Maybe you know tools that feed mysql with xml easily (I don’t know mysql but can guess how it works)… why not use that so… to do step 3) from a) format. Though, as far as displaying in the GUI is concerned, it would be cool to have python objects at our disposal, because it will very annoying when coding to have to call the mysql db to get info on or or the other attribute of an object. Rather we’d have one call to mysqldb to get a python object equivalent representation of some requested object once, then read from that python object to represent things on screen.

So here’s a small text diagram (we should put that nicely sometime) I propose :
for the downloading part
internet =download=> html/xml =converter=> strictlyFormattedXml/PythonObjects =feeding=> MySQL DB
for the displaying in Blender part
MySQL DB =converter=> PythonObjects =GuiDisplay=> description text, images, size, tags etc…

hi,
i’ve written a few things on the temporary wiki to kickstart it.
we have to decide on how we want to decompose work over time
that question arised when I tried to write something for a sort of Roadmap section.
Here’s one thing I saw on some project tracker once :
1 roadmap = several milestones (that could be associated each with a version number).
1 milestone = several tickets (= 1 functionnality/fix…)
and each ticket is assigned to someone.
do you want to work like that ?

As for now, maybe we could just rummage around design ideas & write specifications drafts (I’d like those to be written most directly be on the wiki… how about you ?), without coding anything. And then write the roadmap.

There’s still research work that other people:eyebrowlift2: could handle along with us…
Please see the project’s current ToDo list if you think you can help (no need to know how to program for now !).
I copy paste down the research axes below, though they are described in details (I made an effort to formulate those as questions) on the wiki. :

  • UI design research (search how others have done, sketch one GUI & code it)
  • existing repositories locations (list them), provided contents(ie: such repository provides only materials) and ways of access (html only? some API/script already exists+direct page link to those info?)
  • defining a standard protocol/folder structure(?) for communicating with repositories
  • kinds of (blender)-compatible contents, attributes/meta-data to describe themJonathan

Hi,
I’ve added things to the wiki in the “what’s already been done” section with the scripts repository download scripts made by the Blender Python Consorting (BPyC)
and Inkscape Open Clip Art importer tool (which was packed with the 0.46 release).

I feel concerned with the issue of making something reusable for other apps.

And so I thought, that the Inkscape community could be interested since they have that Open Clip Art Library importer up (there should be an exporter(uploader) somewhere, but I don’t see it showing up in my File menu in Inkscape).
I was about to write an e-mail to the Inkscape devel list but… did not dare, because I did not know where to start from for writing.
So for now I just left a comment on a developer’s blog who had posted some weeks ago a screenshot of the importer/exporter tools for inkscape, I guess because this is what he was developping then.

I’ve just read this :


what I’d like to point out in this article is the transcript the person made of a talk given by two Novell employees. The latter said that there’s a lot of duplicated code between KDE and Gnome and that efforts could be made, to share things in common because there were obvious places where things could be made together/once only.

So I’m thinking about reaching out to the other communities, namely Inkscape for now because, the other app I know which uses repositories (apart from Blender) is Scribus, and I haven’t taken the time to discover their tool. I think that I’ve also read somewhere that Open Office has a tool to download graphics from the Open Clip Art library.
KDE also has a tool which is very simple, that provides you with a dialog to browse and download apps from certain repositories. The app name is khotnewstuff and if you have KDE, should have it normally. It uses a .knsrc settings file, provided by the calling program, to set itself up at run time. Amarok uses khotnewstuff to manage its scripts and the .knsrc file looks very simple.

I’ll try to make a new diagram for the framework with more details than the previous one (which was inthe google docs) showing more obviously where common parts and community/app-specific parts could relate to each other.

I’ll try to make your (PapaSmurf) MySQL preference/cache storing as a module on this graph, for I guess people may want to implement those same functionnalities with xml or other text files.

Jonathan

May I suggest something a little smaller with a smaller memory footprint instead of MySQL. I’d prefered it if it was possible to actually make the client framework DB agnostic. There should be plenty of abstraction layers provided by Python I assume. Other languages do have this (e.g. Java, Ruby, Perl, PHP) This way the database can be configured at runtime without having to maintain separate codebases.
For a default database I would propose SQLite (http://www.sqlite.org) since it can be embedded into the application and there is a python interface provided for it. (http://oss.itsystementwicklung.de/trac/pysqlite/) Since SQLite has a C API it should also be fairly easy to code in C if this becomes nessecary at one time.

I’d also recommend not to go down the road of writing an html parser. Instead provide a standardized REST interface. Since you allready have a xml structure this should be the easiest to adapt for the different repositories. REST is pretty common and should be fairly easy to implement in most script languages used on the net. Most of them allready have a framework for that.

One more thing that I find you should incorporate into the xml structure you are designing for the exchange of data. Provide some way of defining dependencies. So that if you have content which needs other content this can be easily expressed. Adding this later might proof impossible and be a big handicap for distributed content sharing.

For the client knowledge of what a web site contains, I was thinking that the User would have to pretty much manually go through an mark each blend file (and annotate what it contains - (materal, mesh, armature, ipo/actions, particle system, script, game). I don’t see how an HTML parser could possibly discern that, especially since many sites may generate the html dynamically via php server.

Agreed on SQLLite, I just chose MySql because I have used it and it’s OSS.

The simplest implementation of RepoClient is simply helping a user manage what is already on their hard disk…

<runs off to wiki>…

Yo this is up to PapaSmurf (if you’re still ok) to deal with that.

What I’ll do on my side is I’ll have an (class-)interface with functions like setCache(str someWebAddressToThePageToCache, str pageRawContent), storeLogin(str id,str psswd,str webDomainAddress) etc… and people that want to use such or such database system locally will just have to override/define those interface methods so that calls from the Core translate to sql calls, or xml read/writes etc…

I know that there are standardized REST interfaces, such as one at api.opendesktop.org, and one for the Open Clip Art Library or the ccHost platform maybe. I also know that some websites have a much simpler API that’s not REST at all.
For now, we’re just working on the client and thinking a design over to make it more very modular, so that you connect it to any communication interface, whether REST, SOAP, RSS, or simple html fetching.

I must admit though, that “we” are planning to reuse or extend an online API, document it and create a generic client side implementation (and why not server side), so that : anyone wanting to set up its own online repository, can try and match this online API and have its repository accessible by the client framework’s “generic repository access plug in” right away. If someone’s website’s API doesn’t match an existing plugin for the framework, either he has to write a new plugin, or he may modify his online API to match an existing plugin/API.

So, for example, there’s in the same way as for the database interface, a web interface for the client which would have functions such as : CategoriesListObject getCategoriesList(), CategoryObject getCategory(str categoryId), ContentObject getContent(str contentId).
And so, if someone has a repository named R at address A, which exposes an Online API OA, all he has to do, in order to enable the client script to access his website is:
to reuse/implement the client’s web interface, that is, to override/define the previous functions I’ve named so that, when the client app’s Core calls those functions for repository R’s registered plugin/interface/wrapper, nice code is run behind the scenes to connect to repository R at address and talk to the API OA.

Anyone should basically be able to setup its own online API, and code its own plugin for the client framework that will allow anyone running the client framework to interact with the servers for which web plugins have been installed.

yo I dont like my writing style.
Thanks for your message.

Yo
the project space creation approval on blender.project takes time and I still don’t understand why I’m told that I don’t have wiki access when I have it.
I want to be able to submit code online to some place.

so, I created a google code project here :
http://code.google.com/p/blender-repo-client-framework/

I can still delete that if that was useless or if we manage to obtain a space on the blender projects site (they had ask that our projects contents shape up more).

@PapaSmurf, if you want to have write access to the project (I could set you as an admin like me), please give me a google adresse you have.

this message is obsolete

I dont really have a google address, and i am not quite sure what that is. But, I thought we started a wiki page somewhere already.

yo

there’s a plain DIA diagram file for now, and I’d like to add some python code later :
https://projects.blender.org/plugins/scmsvn/viewcvs.php/?root=repoclientfmw
I’ve saved it as plain so that it is versionning-friendly.
you can checkout the svn repo following this page info :
https://projects.blender.org/scm/?group_id=147

Sounds like a cool idea, especially for free textures, models, etc. However, do be careful with scripting and plugins. Unlike (for example) firefox browser extensions, blender scripts (and model files which can contain run-on-start or run-on-repaint scripts) are run in python with full operating system access.

Which means you should probably include in your design for this thing consideration for that. It needs to be secure, tamper-proof, and scripts (especially) would have to be handled with care.

What do the other big names do? I mean, 3dstudio max has the same problem, doesn’t it? They have a vast plugin library but those plugins (the non-maxscript ones) are just C++ programs which naturally have full access to the system.

Just don’t want to see the thing burn because some hacker decided to upload a malicious .blend file. It would not take more than 15 minutes of work to set up a phone-home work-stealing trojan within a scriptlink blender file.

Hi
it is very good you’re pointing this.
I hadn’t thought about it.
I think that client plugins (1 client plugin for access to 1 kind of repository) should be checked someway by knowledgeable reviewers and then recommended/certified to the users.

For example, a client plugin script file could be md5summed before startup and the obtained md5 hash could be compared with hash present on a central server. If the hash isn’t in the database, the GUI could raise a popup saying that the plugin has not been certified as secure by the admins.
We could also use certificates, but I don’t know how those work. Or use quick code analysing techniques : no eval() in the plugins and no touching to something else than a python module (which exposes only useful and non dangerous functions) we’d give.
Maybe what I’m saying is a beginner answer, since I’ve never thought about this kind of possible issues.

If you or anyone know ways to ensure that plugin remain secure for the end-user, or would want to contribute in that aspect or another, that could be cool.

I think the client should just index and allow the user to browse something, and mirror a copy locally, and that’s it. If the user decides to use a texture, or run a script, that’s their call. The functionality of the Client should be only concerned with accessing libraries and mirroring copies if desired. The whole checksum process would be part of the mirroring a copy, to make sure an accurate copy was mirrored.

The client should also not be concerned with royalty and copyright issues. If the user can access the library and has the right to mirror a copy of the content, great. If they don’t, they should not invoke that function. It should not be responsibilty of the client browser software to somehow figure that out.

Hi
although I might seem as I can change opinions quickly,
I agree with PapaSmurf that we should make things simple for now.

There’s no written code for this project for now. So let’s focus on simple things first.

if plugins have a mean behaviour anyway, since they’ll be written in clear text, I guess they’ll be spotted quickly.
So far the repo client scripts I’ve seen don’t seem to care at all about security.
There’s just the GHNS repos REST API which require authentification for any small actions, to protect themselves from flooding and XSS attacks and that’s too specific.

Zos you are talking about some sort of diagram you created in order to show the basic architecture. Where is it? I’d like to take a look at it.

About security do you plan on simply copiing a blend file to the local computer or are you actually planning on directly linking/appending the stuff into the currently open blend file? And if I read this correctly you are also planning on supporting images for textures and the like?

EDITED: I found the diagramm in svn trunk what application is this for?