XML parsing

I have a project I’m thinking about starting, but first I want to know if there is any official system for handling xml in Blender. AFAIK there are no xml parsing modules included with Blender, which means a script writer has a limited number of options:

  • Require a full Python install
  • Hack your own purpose-built parser
  • Bundle an existing parserI’m leaning toward option one right now, except that it limits my scripts userbase to those who have or are willing to install Blender’s version of Python. Option two is out - I’m too lazy to write a parser for this project. Which leaves option three… Which raises an interesting question: Why doesn’t Blender have an xml module included by default? I’m sure there is a small, lite-weight, and fast pure Python XML parser licensed as GPL that could go in the bpyModules folder. If not though I’m sure we could come up with one!

I don’t know, maybe I’m wrong and there is a well defined route for xml handling?

Hi Kitsu,
What are you looking to parse?

try xmlproc, its pure python and could be distributed with blender. its also tried and true. :wink:

I have my own xml parser which I only tested on 2 files its pure python and can par an RSS feed and a collada file…
My goal is to keep it under 100 lines.

Its quite fast, only loops on the data twice, once to find the tag limits <> and again to put them into a hierarchy.

Dont worry about the class stuff- its realy only datastorage-
could just as well be a dict or a typle.

Feel free to rip it apart and have your way with it.

My main concern is that supporting 90% of xml files isnt enough, and this script may need to be 3 times as big to support the 10% of weirdo xml files.


# GPL XML Parser by Campbell Barton
class xml_block(object):
    __slots__ = 'name', 'children', 'options', 'data'
    def __init__(self):
        self.name = ''
        self.data = None
        self.children = []
        self.options = {}
    def __repr__(self):
        return '
name:%s
data:%s
options:%s
children:%s
' % (self.name, self.data, self.options, self.children)
    def dump(self):
        return '
name:%s
data:%s
options:%s
children count:%s' % (self.name, self.data, self.options, len(self.children))
    
def xml2list(xml_text):
    tag_bounds = []
    xml_list = []
    # Mark all start and ends for the &lt;&gt; and &lt;/&gt;
    START, END, SINGLE = 0,1,2 # is it a start tag, end tag or a single tag.
    i=0
    while i &lt; len(xml_text):
        # tag all &lt; and &gt; 
        if xml_text[i] == '&lt;':
            i = ii = i+1 # increase i so we can use slicing
            while xml_text[ii] != '&gt;': ii += 1
            
            if xml_text[i] == '!': pass # comment, assume &lt;!--
            elif xml_text[i] == '?': pass # assume &lt;?xml
            elif xml_text[i] == '/': # Ending
                tag_bounds.append((i,ii, END))
            else: # Starting
                if xml_text[ii-1] == '/':
                    tag_bounds.append((i,ii-1, SINGLE)) # dont include the /
                else:
                    tag_bounds.append((i,ii, START))
            i= ii
        i+=1
        
    def build_xml(tag_idx, children):
        '''
        gets all the data between here and the next index
        '''
        tag = tag_bounds[tag_idx] # must be the starter - tag[2] == True
        
        if tag[2] == END:
            print xml_text[tag[0]:tag[1]]
            print xml_list
            raise "Error"
        
        xml_blk = xml_block()
        children.append(xml_blk)
        
        name_and_opts = xml_text[tag[0] : tag[1]].split()
        xml_blk.name = name_and_opts[0]
        
        if len(name_and_opts) &gt; 1: # Some options were set
            i =1
            while i &lt; len(name_and_opts):
                # print name_and_opts
                key,val = name_and_opts[i].split('=')
                if val[0]=='"' and val[-1] == '"': val = val[1:-1] # strip ""
                xml_blk.options[key] = val
                i+=1
        
        if tag[2] == SINGLE: # this tag has no matching end tag, return the next index.
            return tag_idx+1
        
        tag_next = tag_bounds[tag_idx+1]
        xml_blk.data = xml_text[tag[1]+1:tag_next[0]-1].strip() # Text between now and the next tag is data
        
        tag_idx += 1
        while 1:
            tag_next = tag_bounds[tag_idx]
            # This ends the current tag
            if tag_next[2] == END:
                name = xml_text[tag_next[0]+1:tag_next[1]]
                if name == xml_blk.name :
                    return tag_idx + 1
                # Should only be ending the current tag
            else:
                tag_idx= build_xml(tag_idx, xml_blk.children)
                if tag_idx &gt;= len(tag_bounds):
                    return tag_idx # will finish
    
    build_xml(0, xml_list)
    return xml_list

pyXML at http://pyxml.sourceforge.net/ might be an alternative to Ideasman42 code.

The CrystalSpace project uses it to provide an XML translation layer between the Blender2crystal export plugin and the CS/CEL engines. It installs in your Python directory and the library works quite well. Cheers!

pyXML is dead AFAIK: http://sourceforge.net/project/showfiles.php?group_id=6473
I was actually looking at elementTree yesterday. It is mature, comes in c or python flavors, and is bundled with Python 2.5. I’ve never used it though because DOM/minidom/SAX all come as standard modules :rolleyes:

Cambo: Your code looks pretty interesting, I think I’ll tinker with it a little and see what I can make :wink:

Could you show how it can be used to parse this text (very simple xml file)
:


xmltext="""
&lt;svg 
     xmlns:x="&ns_extend;" xmlns:i="&ns_ai;" xmlns:graph="&ns_graphs;" i:viewOrigin="245 469.6484" i:rulerOrigin="0 0" i:pageBounds="0 841.8896 595.2754 0"
     xmlns="&ns_svg;" xmlns:xlink="&ns_xlink;" xmlns:a="http://ns.adobe.com/AdobeSVGViewerExtensions/3.0/"
     width="109" height="98.722" viewBox="0 0 109 98.722" overflow="visible" enable-background="new 0 0 109 98.722"
     xml:space="preserve"&gt;
    &lt;metadata&gt;
        &lt;variableSets  xmlns="&ns_vars;"&gt;
            &lt;variableSet  varSetName="binding1" locked="none"&gt;
                &lt;variables&gt;&lt;/variables&gt;
                &lt;v:sampleDataSets  xmlns="&ns_custom;" xmlns:v="&ns_vars;"&gt;&lt;/v:sampleDataSets&gt;
            &lt;/variableSet&gt;
        &lt;/variableSets&gt;
        &lt;sfw  xmlns="&ns_sfw;"&gt;
            &lt;slices&gt;&lt;/slices&gt;
            &lt;sliceSourceBounds  y="370.927" x="245" width="109" height="98.722" bottomLeftOrigin="true"&gt;&lt;/sliceSourceBounds&gt;
        &lt;/sfw&gt;
    &lt;/metadata&gt;
    &lt;switch&gt;
        &lt;foreignObject requiredExtensions="&ns_ai;" x="0" y="0" width="1" height="1"&gt;
            &lt;i:pgfRef  xlink:href="#adobe_illustrator_pgf"&gt;
            &lt;/i:pgfRef&gt;
        &lt;/foreignObject&gt;
        &lt;g i:extraneous="self"&gt;
            &lt;g id="Calque_1" i:layer="yes" i:dimmedPercent="50" i:rgbTrio="#4F008000FFFF"&gt;
                &lt;g&gt;
                    &lt;g&gt;
                        &lt;path i:knockout="Off" fill="#F35B13" d="M78,83.648c15,2.5,26-4,31-40c0,0-13,5-19,16s3,23-12,22s-36-10-26-46
                            s-17.5-37-26.5-35.5C13.082,2.218-2,6.648,0,10.648s14.5,2,25,1c9.178-0.874,17,3,11,25s-4,57,15,62
                            C59.68,100.932,68,85.148,78,83.648z"/&gt;
                    &lt;/g&gt;
                &lt;/g&gt;
            &lt;/g&gt;
        &lt;/g&gt;
    &lt;/switch&gt;
&lt;/svg&gt;
"""

To run a test, try this.

import xml_tiny_parser
reload(xml_tiny_parser)
print xml_tiny_parser.xml2list(xmltext)

  • I found some that it has a problem with
    viewBox=“0 0 109 98.722”
    Because it dosent account for their being spaces in the value.

So it will need some fixing, Ill have a go at fixing if nobody else wants to :wink:
If it proves usefull we can add into blender… that bug wont be too hard to fix.

hmm i made a xml parser/writer for a project i was doing it works for what i need it to the parser part is about 180 lines including the comments and blank lines

adn the writer is like 44 lines… it could probably be way better but it works for me

# Crappy XML Parser #
# By: scabootssca   #
# On:2/12/07        #

#Each split is a node.
class parserNode:
    def __init__(self,tag="",meta={},index=[None,None],data="",inside="",parent=None,children=[]):
        self.tag = tag
        self.meta = meta
        self.index = index
        self.data = data
        self.inside = inside
        self.parent = parent
        self.children = children[:]
        self.parsed = 0

    def getChild(self,**keys):
        candidates = self.children[:]
        for child in candidates:
            for key in keys:
                if key in child.meta:
                    if child.meta[key] != keys[key]:
                        candidates.remove(child)
                else:
                    candidates.remove(child)
        if candidates: return candidates[0]
        else: return 0

    def getChildByTag(self,tag,index=0):
        currentIndex = 0
        for child in self.children:
            if child.tag == tag:
                if currentIndex == index: return child
                else: currentIndex += 1
                    

class xmlParser:
    def __init__(self,xmlData):
        self.xmlData = xmlData
        self.meta = {}

    def readMeta(self):
        index = self.xmlData.find("&lt;?xml")
        meta = {}
        if not index == -1:
            endIndex = self.xmlData.find("?&gt;",index)
            data = self.xmlData[index+5:endIndex].split('"')
            for index in range(0,len(data),2):
                key = data[index].strip()
                if key:
                    meta[key[:-1]] = data[index+1]
            self.xmlData = self.xmlData[endIndex+2:]
        self.meta = meta
        return meta

    def removeComments(self):
        itera=0
        while "&lt;!--" in self.xmlData and itera&lt;20:
            itera+=1

            index = self.xmlData.find("&lt;!--")
            endIndex = self.xmlData.find("--&gt;",index)

            before = self.xmlData[:index]
            after = self.xmlData[endIndex+3:]
            self.xmlData = before+after

    def findEnd(self,node,data=None):
        nested = 0

        findTag = node.tag
        if not data:
            if node.data: data = node.data
            else: data = self.xmlData

        for index,line in enumerate(data.splitlines()):
            for tag in line.split("&lt;"):
                if not tag.startswith("?") and len(tag.strip()):
                    tag = tag.strip().strip("&lt;&gt;").split("&gt;")[0].split()[0]
                    if tag == findTag:
                        nested += 1

                    if tag == "/"+findTag:
                        nested -= 1

                        if nested == 0:
                            return index

    def getRootTagAndIndex(self):
        for index,line in enumerate(self.xmlData.splitlines()):
            for tag in line.split("&lt;"):
                if not tag.startswith("?") and len(tag.strip()):
                    return index,tag.strip().strip("&lt;&gt;").split("&gt;")[0]

    def getChildren(self,node):
        children = []
        data = "
".join(node.data.splitlines()[1:-1])
        split = data.split("&lt;")

        currentTag = None
        currentData = ""
        for tag in split:
            tag = tag.strip()

            if not currentTag:
                if len(tag):
                    currentTag = tag.split()[0].split("&gt;")[0]
            if currentTag and tag.strip("&lt;&gt;") == "/"+currentTag:
                children.append([currentTag,tag])
                currentTag = None

        searchOffset = 0
        for iteration,child in enumerate(children):
            #print "-"*50
            data = node.data[node.data.find("&lt;"+child[0],searchOffset):node.data.find(child[1],searchOffset)+len(child[1])]
            line = " ".join(data.split("&gt;")[0].strip("&lt;&gt;").split()[1:]).split('"')

            tag = child[0]

            
            meta = {}
            for split in range(0,len(line),2):
                if len(line[split]):
                    meta[line[split].strip("=").strip()] = line[split+1]

            childNode = parserNode(tag,meta)

            childIndex = [None,None]
            for index,line in enumerate(node.data[searchOffset:].strip().splitlines()):
                prelude = len(node.data[:searchOffset].splitlines())

                
                if "&lt;"+child[0] in line:
                    childIndex[0] = index+prelude
                if child[1] in line:
                    childIndex[1] = index+prelude

                if childIndex[0]!=None and childIndex[1]!=None:
                    break

            childNode.index = childIndex
            childNode.data = data
            childNode.inside = data.split("&gt;",1)[1:][0].rsplit("&lt;",1)[0]
            searchOffset = len(node.data[:node.data.find(child[1],searchOffset)+len(child[1])])

            childNode.parent = node
            node.children.append(childNode)
            #print childNode.children
        return node.children

    def getRoot(self):
        index,root = self.getRootTagAndIndex()

        tag = root.split()[0]
        
        meta = {}
        line = " ".join(root.split()[1:]).split('"')
        for split in range(0,len(line),2):
            if len(line[split]):
                meta[line[split].strip("=").strip()] = line[split+1]
        self.root = parserNode(tag,meta)

        endIndex = self.findEnd(self.root)

        self.root.index = [index,endIndex]
        self.root.data = "
".join(self.xmlData.splitlines()[index:endIndex+1])

    def depthFirstSearch(self,node):
        node.parsed = 1
        children = self.getChildren(node)
        for child in children:
            if not child.parsed:
                self.depthFirstSearch(child)

    def parse(self):
        self.readMeta()
        self.removeComments()
        self.getRoot()
        self.depthFirstSearch(self.root)
        return self.root

# Crappy XML Generator #
# By: scabootssca      #
# On:2/17/07           #

class node:
    def __init__(self,tag,header={},parent=None):
        self.tag = tag
        self.header = header
        self.parent = parent
        if parent: parent.addChild(self)
        self.children = []#children
        self.written = 0
        self.indent = ""
        self.inner = ""

    def addChild(self,child):
        if not self.inner:
            child.parent = 1
            self.children.append(child)
            return child

    def setInner(self,data):
        self.children = []
        self.inner = str(data)

def writeChildren(file,node,indent):
    node.written = 1
    node.indent = indent
    file.write("
"+node.indent+"&lt;"+node.tag)
    header = []
    indent += " "
    for data in node.header: header.append(' %s="%s"'%(data,str(node.header[data]).replace("'",'"')))
    file.write(",".join(header)+"&gt;")
    if node.inner: file.write(node.indent+indent+node.inner)

    for child in node.children:
        if not child.written:
            writeChildren(file,child,indent)

    file.write(node.indent+"&lt;/%s&gt;
"%node.tag)

def compile(path,root):
    f = file(path,"w")
    writeChildren(f,root,"")
    f.close()


#root = node("root")

#print root.children
#a = node("objects",parent=root)
#root.addChild("objects")
#b = root.addChild("groups")
#c = root.addChild("events")

#print root.children
#print a.children
##compile("saveTest.txt",root)

Well, I poked at Cambo’s for a while, then decided to roll my own. I got to the point where I had to write the parse though and I thought: this is stupid! There are better/cleaner/faster XML parsers existing than I could hope to make in any reasonable amount of time. I really wasn’t doing anything new either, if I had finished it probably would have looked like a less featureful version of scabootssca’s.

So instead I installed ElementTree and started reading the docs. It is really pretty nice. The only irritating thing is the way it attaches elements namespace to tags. I would definitely like to see etree included with Blender in the future. The only problem I see is that it requires pyexpat - a c module. AFAIK though expat is portable and supported on every modern Python platform (even WinCE ;)). It’s one more thing to compile, but it is also all that is needed to make Python’s DOM/minidom/SAX parsers work too. I think ElementTree would make the best ‘standard’ XML parser for Blender though.

Here’s a snipit to pull some random data from an SVG file:

from elementtree.ElementTree import ElementTree, iterparse

filename = "./atom.svg"

# The next part just strips the {namespace} from the tags
root = None
for event, elem in iterparse(filename, events=("start", "end")):
    if event == "start" and root is None:
        root = elem
    elif event == "end":
        #print event, elem.tag
        if elem.tag.startswith("{"):
            elem.tag = elem.tag.split("}")[1]

# If it weren't for all the namespace junk in SVG files you could do just:
# tree = ElementTree(file=filename)
# root = tree.getroot()

for elem in root.findall('path'):
    print "Tag: %s" %elem.tag
    print "ID: %s" %elem.get('id')
    print "Transform %s
" %elem.get('transform')

Output:

Tag: path
ID: path1042
Transform matrix(1.493507,0.000000,0.000000,1.493507,-185.7273,-103.7972)

Tag: path
ID: path1666
Transform matrix(0.688313,0.000000,0.000000,0.688313,31.49246,108.2279)

Tag: path
ID: path1667
Transform matrix(0.688313,0.000000,0.000000,0.688313,199.2376,106.7565)

Tag: path
ID: path1665
Transform matrix(0.688313,0.000000,0.000000,0.688313,118.3079,-27.88102)

Tag: path
ID: path1668
Transform None

Tag: path
ID: path1669
Transform None

Tag: path
ID: path1670
Transform None

And this is only a simple file. There is dozen of manners to write values.

I believe this one may be a bit more maintained: http://www.4suite.org/index.xhtml
and there is a Python 2.5 version listed…

It seems that there is already a xml parser in Blender bundle scripts (not in my script, i use a light and faster method, xml parsers just end to build their trees when my script has already ended to import all the needed data).

Don’t forget that svg is exported by a few applications : Illustrator, inkscape, coreldraw suite, openoffice… each of them uses a different format.

Yeah, non-standard standards suck. I don’t really care about special cases though - I just want something in Blender that simplifies/standardizes XML handling for simple scripts. Most XML formats are not widespread enough to be as fractured as SVG, and most script writers aren’t interested in supporting every invalid permutation of a format that could exist. If you want to deal with weird stuff you are probably stuck writing your own parser anyway. At least with ElementTree you aren’t stuck doing that from the start.

Any news on this matter? Did anyone try this? http://codespeak.net/lxml/index.html
Kitsu, what did you chose in the end?

I hope this is no double post, but the first one didnt show up.
So my question was: are there any news on this matter?
Did anyone try this http://codespeak.net/lxml/index.html ?
And Kitsu, what did you chose in the end?