FBX Importer v0.001

since fbx is a fairly structured format I wondered how hard it would be to make a parser in python.
Heres the first cut.

To make it useful I also added mesh import - only verts and faces.

Id really like if others could help coding this - the code is below, FBX isnt that hard to read and you can test with FBX exporter too and see how it works. :wink:


def parse_fbx(path, fbx_data):
    DEBUG = False
    
    f = open(path, 'rU')
    lines = f.readlines()

    # remove comments and 

    lines = [l[:-1].rstrip() for l in lines if not l.lstrip().startswith(';')]

    # remove whitespace
    lines = [l for l in lines if l]

    # remove multiline float lists
    def unmultiline(i):
        lines[i-1] = lines[i-1] + lines.pop(i).lstrip()

    # Multiline floats, used for verts and matricies, this is harderto read so detect and make into 1 line.
    i = 0
    while i < len(lines):
        l = lines[i].strip()
        if l.startswith(','):
            unmultiline(i)
            i-=1
        try:
            float(l.split(',', 1)[0])
            unmultiline(i)
            i-=1
        except:
            pass
            
        i += 1

    CONTAINER = [None]
    def isfloat(val):
        try:
            CONTAINER[0] = float(val)
            return True
        except:
            return False

    def isfloatlist(val):
        try:
            CONTAINER[0] = tuple([float(v) for v in val.split(',')])
            return True
        except:
            return False


    def read_fbx(i, ls):
        if DEBUG:
            print "LINE:", lines[i]
        
        tag, val = lines[i].split(':', 1)
        tag = tag.lstrip()
        val = val.strip()
        
        
        if val == '':
            ls.append((tag, None, None))
        if val.endswith('{'):
            name = val[:-1].strip() # remove the trailing {
            if name == '': name = None        
            
            sub_ls = []
            ls.append((tag, name, sub_ls))
            
            i+=1
            while lines[i].strip() != '}':
                i = read_fbx(i, sub_ls)
            
        elif val.startswith('"') and val.endswith('"'):
            ls.append((tag, None, val[1:-1])) # remove quotes
        elif isfloat(val):
            ls.append((tag, None, CONTAINER[0]))
        elif isfloatlist(val):
            ls.append((tag,  None, CONTAINER[0]))
        
        
        #name = .lstrip()[0]
        if DEBUG:
            print 'TAG:', tag
            print 'VAL:', val
        return i+1

    i=0
    while i < len(lines):
        i = read_fbx(i, fbx_data)



# Blender code starts here:
import bpy
def import_fbx(path):
    fbx_data = []
    parse_fbx(path, fbx_data)
    # Now lets get in the mesh data for fun.
    
    sce = bpy.data.scenes.active
    sce.objects.selected = []
    
    for tag1, name1, value1 in fbx_data:
        if tag1 == 'Objects':
            for tag2, name2, value2 in value1:
                if tag2 == 'Model':
                    print tag2, name2
                    # we dont parse this part properly
                    # the name2 can be somtrhing like
                    # Model "Model::kimiko", "Mesh"
                    if name2.endswith(' "Mesh"'):
                        verts = None
                        faces = None
                        # We have a mesh
                        for tag3, name3, value3 in value2:
                            # print 'FOO ', tag3, name3
                            if tag3 == 'Vertices':
                                #print value3
                                verts = value3
                            elif tag3 == 'PolygonVertexIndex':
                                #print value3
                                faces = value3
                            
                        # convert odd fbx verts and faces to a blender mesh.
                        if verts and faces:
                            me = bpy.data.meshes.new()
                            # get a list of floats as triples
                            me.verts.extend( [verts[i-3:i] for i in xrange(3, len(verts)+3, 3)] )
                                
                            # get weirdo face indicies
                            face = []
                            blen_faces = [face]
                            for f in faces:
                                if f<0:
                                    face.append(int(-f)-1)
                                    face = []
                                    blen_faces.append(face)
                                else:
                                    face.append(int(f))
                            
                            if not blen_faces[-1]:
                                del blen_faces[-1]
                            
                            me.faces.extend(blen_faces)
                            
                            sce.objects.new(me)

#import_fbx('/test.fbx')

import Blender
if __name__ == '__main__': 
    Blender.Window.FileSelector(import_fbx, 'ASCII FBX 6.1', '*.fbx')


Hey Ideasman nice work mate. Been having a look at it and playing ith lots of different things.
I got to the point of making a simple 3ds file of a few primitives scattered around, converting it from 3ds to fbx with the autodesk converter then importing into blender.
script runs choice. i noticed the following:
all meshes imported at origin 0,0,0
no cameras imported [either in native fbx or 3ds converted to fbx] though they are noted in the ascii text.
neither cam nor primitive animation.
though i don’t think the script was reading for that from what i can tell. bit green at the old scripting thing.
anyway, power to you ideasman.
j

I’d be willing to help with the parsing. I’m good with regex and python. But be aware that FBX is closed source and this is and only will be reverse engineering prone to breakage. Consider:
http://discussion.autodesk.com/thread.jspa?threadID=492418
Reply From: trevoradams
Date: Apr/05/07 - 22:04 (GMT) Re: FBX file format specification

Hello everyone.

Unfortunately there are no published specifications for the FBX ASCII file format. There are still no plans for Autodesk to have these specs published/written, either.

Considering what I’ve read in what’s been posted about this, I wish I could say otherwise, but that’s pretty much the bottom line for now.

Thanks for your understanding and any further feedback is definitely welcome,

Trevor Adams

FBX Technologies Group
Autodesk media and Entertainment
============================== also, from the same thread =============
Reply From: gdgi
Date: Oct/23/06 - 04:27 (GMT) Re: FBX file format specification

The lack of a true specification makes supporting FBX for game engines (which need to be cross-platform by their very nature) almost impossible.

We have been supporting FBX in our engine via the SDK for the past year and the inconsistencies between the various export plugins has been a major nightmare.

We had hoped that it would be a ‘standard’ solution for getting content into our engine from max, maya, lightwave, softimage etc but we have had very inconsistent results.

the latest round of plugins in particular break every program - animations exported no longer work with application that use previous versions of the SDK.

We should not end up being forced to update our SDK implementation whenever Autodesk decides to put out an update.

It’s a good thread to read if you’re thinking about doing a plugin. But it’s also clear that people really want it. :yes:

Its not as bad as all that, though this is correct.

First of all, its not to Autodesks advantage to change the format so old SDK cant load the files at all. - sure this might happen with specific features, but for general mesh/bones/lamps etc… they would be shooting themselves in the foot to do so.
They could try be like Microsoft with the DOC format, and use it to force people to use new versions however they have not done this so far and I dont expect them to.

regarding the parser, I dont think that it would be prone to breakage, the importer is more prone, but with a nice working parser it should be easy to adjust to new settings for files.

We can make sure the importer works all the time I was thinking it could be used with the FBXConverter that would always output a compatible ASCII file.

If they change their ascii format then we can change the importer,- its just a python script :wink:

I can understand the problems these guys are having quote - “We had hoped that it would be a ‘standard’ solution for getting content into our engine from max, maya, lightwave, softimage etc but we have had very inconsistent results.”

The problem is that a standard either supports a limited range of functions and everyone can interoperate - (See OBJ), or it supports a wider range and some programs wont be able to read or write some settings. FBX suffers for the latter. FBX Supports LOADS of weirdo features that are really specific to MotionBuilder, so most other apps just read/write FBX’s with a subset of these.

If we write an importer people will have to accept that some stuff is not supported, ( like rotation-pivot-offset, optional order of rotation ZYX, XZY etc…)
If users bitch and moan about this then Id rather not bother. However since most other apps do exactly this, It seems that it could be worth writing and just have good support for meshes, armatures, UV’s materials etc.

The parser I wrote is almost complete, Id prefer not to use regex is possible since It adds a full python install as a dependency (unless it has a big advantage over the current way)

I would however like some help with this, I wrote the exporter because I see its important for blender to be able to be used for generating content for games/simulation etc. but an importer isnt needed for this. Id like others to contribute and I’ll help out if needed. Maybe 1:1 offer, For every community added feature, I’ll add one too :wink:

After sending my bummer post I read the export thread and realized you’d done it.
I then synched to the latest subversion release to see it, and it’s over 3000 lines.
It was late and my eyes glazed over, but it looks like you have a nice map from
blender to fbx. However, exporters are a bit easier because you don’t have to
parse, and parsing and compiling are best friends. A compiler/interpreter has two parts,
the token generator (aka regular expression matcher) and parser generator (compose
tokens into meta data). The regex library of python is super good for this. I notice
you use time and sys in your exporter. Why is re not available?
I’m thinking the large swaths triple quoted export config templates could be converted to a
scanner, but consuming whitespace, and picking out data is where regex shines.
Also, I tried importing and exporting 3ds files and running them though the autodesk
converter util to get fbx files, to see what was supported in 3ds. It looks like your wrote
that one too. :ba:

I see what you mean about using converter to fix the version and support only that one.
Consider that even Autodesk changed their api and probably broke tons of stuff so
they added the version output option to converter. Why else would they do it? :rolleyes:

Anyway, your exporter is the best way to understand the fbx <-> blender paths.
I’ll link it here for completeness. :slight_smile:
https://svn.blender.org/svnroot/bf-blender/trunk/blender/release/scripts/export_fbx.py

Sorry about the bummer post. I’m new. :o

Hey zettix, a lot of work went into the last FBX exporter update and I now understand the format fairly well but I really dont know anything about parsers and dont know how to use regular expressions :confused:
The FBX ascii format is not that hard to parse so I should really finish it soon!

I can probably get buy being naive and still get it working but if your offering to write a parser I wont say no.

The FBX exporter should be able to run without python installed, time and os modules are only used if they are available, sys is from Blender.sys - I mostly try to write scripts with these constraints but using regex could be an exception.

re: 3ds, I maintain import and export rewriting sections but didnt do them originally.

Pretty rough but gives you all the number lists, name-value pairs.


#!/usr/bin/python

COMMENT = 0
BLANK_LINE = 0

debug = True

class SkipLine(Exception):
  pass

class DoneLine(Exception):
  pass

class FbxParser:
  """This class implements an FBX file format parser.
  In general, an FBX file is assumed to have name: value
  pairs.  This is stored in a dictionary:
   {name : value, name : value, ...} where value can
  be another dictionary or a list.  name can be a tuple
  or a string.
  A mesh's vertices for a model is accessed, for example, with:
  dictname['Objects'][('Model', '"Model::Suzanne", "Mesh" ')]['Vertices'] -&gt;
    a list of floats.

  The treatment of keys for Model and Property probably need work.
  """
  def __init__(self, linelist):
    self.lineno = 0
    self.lines = linelist
    self.total_lines = len(linelist)
    self.number_list_state = None

  def parse_lines(self):
    result = {}
    while self.lineno &lt; self.total_lines:
      try:
        l = self.lines[self.lineno].strip()
        if debug: print "Parsing[%d]: %s" % (self.lineno, l)
        #skip blank lines:
        if not l:
          raise SkipLine, BLANK_LINE
        #skip comments:
        if l[0] == ';':
          raise SkipLine, COMMENT
        # a closing brace ends the sub group
        if l[0] == '}':
          self.number_list_state = None
          return result
        #check if we're in a number list
        if self.number_list_state is not None:
          if l[0] != ',': # the number list has finished
            self.number_list_state = None
            return None
          self.lineno += 1
          if self.number_list_state == 'floats':
            result = [float(v) for v in l.split(',')[1:]] # skip leading comma
          else:
            result = [int(v) for v in l.split(',')[1:]] # skip leading comma
          ext_val = self.parse_lines()
          if ext_val is not None:
            result.extend(ext_val)
          return result
        #assume this is a name: value pair
        first_colon = l.find(':')
        if first_colon != -1:
          name = l[:first_colon]
          first_colon += 1
          # check for name: with no value
          if len(l) == first_colon:
            result[name] = None
            raise DoneLine
          value = l[first_colon:].strip()
          # check for open brace style lines:
          # an open brace starts the sub group
          if value[-1] == '{':
             # name may have extra strings, e.g. Model:
             extra_name = value[:-1]
             if extra_name:
               name = ( name , extra_name)
             self.lineno += 1
             value = self.parse_lines()
          # check for Vertices: style line:
          if name in ('Vertices', 'PolygonVertexIndex', 'Normals', 'UV', 'UVIndex', 'Materials', 'TextureId'):
            if name in ('Vertices', 'Normals', 'UV'):
              self.number_list_state = 'floats'
              value = [float(v) for v in value.split(',')[1:]] # skip leading comma
            else:
              value = [int(v) for v in value.split(',')[1:]] # skip leading comma
              self.number_list_state = 'ints'
            self.lineno += 1
            ext_value = self.parse_lines()
            if ext_value is not None:
              value.extend(ext_value)
            self.lineno -= 1
          # check for Property style line:
          if name == 'Property':
            name = (name, value.split(',')[0])
            value = value.split(',')[1:]
          result[name] = value
          raise DoneLine
        raise 'Cannot parse: %s' % l
      except (SkipLine, DoneLine):
        pass
      self.lineno += 1
    return result

def print_dict(somedict, indent=0):
  for k in somedict.keys():
    print '[%d]' % indent,
    print ' ' * indent,
    print '%s : ' % str(k),
    if type(somedict[k]) == type({}):
      print ' -&gt;'
      print_dict(somedict[k], indent=indent + 5)
    else:
      print  '-&gt; %s' % somedict[k]

def print_dict_keys(somedict, indent=0):
  for k in somedict.keys():
    print '[%d]' % indent,
    print ' ' * indent,
    print '%s : -- ' % str(k)
    if type(somedict[k]) == type({}):
      print_dict_keys(somedict[k], indent=indent + 5)


if __name__ == '__main__':
  f = open('monkey.fbx', 'rU')
  lines = f.readlines()[:]
  c = FbxParser(lines)
  result = c.parse_lines()
  print_dict_keys(result)

So you can now access a semi-tokenized version of the fbx file.
Everything is a string except for the number lists, so getting orientation or other
settings will require more glue, but the parsing is mostly done.

I made a simple suzanne scene and exported it as 3ds and then used converter to
get an fbx file:
http://www.zettix.com/blender/monkey.fbx
The new script converts all the data to the proper type in the dict, so strings are strings,
floats are floats, and ints are ints:
http://www.zettix.com/blender/fbx_dict.txt
That file is an address map of all the data in an fbx file (at least in the monkey.fbx file).
Accessing the vertex list is like so:
[Objects][(‘Model’, ‘Model::Suzanne’, ‘Mesh’)][Vertices] : [12.030659799575799, 9.6101728534698498, 11.4921074008942, 12.572036638259901, 10.5063197040558, 11.314858045577999, 11.7659097290039, 9.5484554481506407, 11.8038991737366, 12.3846243858337] …
And the faces list like so:
[Objects][(‘Model’, ‘Model::Suzanne’, ‘Mesh’)][PolygonVertexIndex] : [0, -3, 46, 2, -45, 3, 1, -48, 3, 47] …
Also the properties and normals and so on.
The latest version of the script is here:
http://www.zettix.com/blender/fbx_import_2007_09_05.py

If you need some special functionality, or really don’t like the tuple as a key, let me know. :slight_smile:
Thanks

what a cool guy!

Long live Ideasman :slight_smile:

Thank you for this!!

@zettix
A few comments on your parser -
The way your parsing lists of numbers is keyword spesific and wont hold up when files use keywords that arnt in the list.

  • if name in (‘Vertices’, ‘PolygonVertexIndex’, ‘Normals’, ‘UV’, ‘UVIndex’, ‘Materials’, ‘TextureId’):

Mine is not ideal either though, passing everything as a float and making the blender importer part convert to ints when it needs to.

The fragility of the number list handling was good feedback, thanks. Re thinking it made it much simpler. I’ve documented it more and tried to make it easier to read, and I also
convert all the data to a string, int, or float. It should be really easy to map dictionary
values to blender interfaces, even importing textures.

The new version is here:
http://www.zettix.com/blender/fbx_import_2007_09_05_2.py

I deleted about 20 lines of cruft with the number list change. Now it’s simple. :smiley:

Also buffers the input file, so you don’t have to read the whole thing in.

Zettix & Cambo - glad to see you guys are working on this. I’m working with an artist who has 3DS Max and it would be very nice to be able to import his .FBX and work on it in Blender… keep it up! :slight_smile:

zettix, skimmed through and this looks much nicer, a few notes.

    # this is a number list continuation line:
    if l[0] == ',':

Should test for the first value being a valid int or float also since line splitting isnt really defined in teh fbx format.

if name == ‘Property’:

Are you sure its necessary to check for property?, looks like its possible to parse it genericlly. I was aiming for no Keywords in the parser.

if value.find(’,’) != -1:
In these cases where the index is not used, its nicer to do this…

if ‘,’ in value:

replace…
if type(s) != type(’’):
if type(somedict[k]) == type({}):
if type(somedict[k]) == type([]):

… with
if type(s) != str:
if type(somedict[k]) == dict:
if type(somedict[k]) == list:

  • float and int are types too

This makes a new list for looping…
for k in somedict.keys():

This loops on the dicts keys without making a new list, so its a bit faster
for k in somedict.iterkeys():

Still, good work man, I hate when I come over so negative in these messages, I really dont mean it.
Maybe we could split the fbx importer into 2 parts, you write the parser, I write the blender importer part? :wink:

@Ideasman42

Should test for the first value being a valid int or float also since line splitting isnt really defined in teh fbx format.
You’re right, but I think the code does the right thing, in that something of the form:
name : value, value, value, value,
,value, value, value, …
Will be handled by the guess_type function so it will convert to int, float, or string automagically.

Are you sure its necessary to check for property?, looks like its possible to parse it genericlly. I was aiming for no Keywords in the parser.

Property lines are special, because the 'name: value" model breaks down. Consider:


            Property: "Use2DMagnifierZoom", "bool", "",0
            Property: "2D Magnifier Zoom", "Real", "A+",100

The first line will set the dictionary to […][‘Property60’][‘Property’] : [“Used2D…”, "bool,…
The second line will clobber the data from the first line.
So I toss the ‘Property’ keyword b/c the ‘Property60’ section only contains Property lines.
I’m open to alternatives, but again, the current scheme breaks on Property lines, and
I’m unaware of other lines that have this issue. One nice check would be to check to see if a key is already in the dictionary and complain if something gets clobbered.
I’ll add that later.

if value.find(’,’) != -1:
In these cases where the index is not used, its nicer to do this…

if ‘,’ in value:

replace…
if type(s) != type(’’):

Fixed.

This makes a new list for looping…
for k in somedict.keys():

Fixed.

Still, good work man, I hate when I come over so negative in these messages, I really dont mean it.
Maybe we could split the fbx importer into 2 parts, you write the parser, I write the blender importer part? :wink:

Not negative at all. This is like a code review and good ones are hard to come by. :wink:
I’m kind of busy so I don’t have lots of time, but the parser is mostly done. I can add
the blender callback for the file selector, but the dictionary-to-blender stuff requires two
things I don’t know:

  1. FBX - are the PolyLists quads? tris? What’s the material/layering stuff about?
  2. Blender’s API: I’m still kind of a noob at blender and don’t know a bone from an ika.
    I can give you convenience functions, like providing lists of mesh verts, normals, etc.
    Names and stuff, navigating the dict.
    If I have some time I’ll see if I can frankenstein your original import and my import to make a mesh in blender.

Here’s the most recent:
http://www.zettix.com/blender/fbx_import_2007_09_06.py
Cheers.

http://www.zettix.com/blender/fbx_import_2007_09_06_2.py
There’s a little helper function to get mesh dicts. It’s fairly straightforward to add other things like normals, UV lists and so on, but I don’t know the format of these:


[Objects][('Model', 'Model::Suzanne', 'Mesh')][('LayerElementUV', 0)][UV] :   [0, 0, 0, 0, 0, 0, 0, 0, 0, 0] ...
[Objects][('Model', 'Model::Suzanne', 'Mesh')][('LayerElementUV', 0)][UVIndex] :   [46, 0, 2, 46, 2, 44, 3, 1, 47, 3] ...

It might be a good idea to gpl it.

@Ideasman42, this Key: syntax is very weird… there is no comma between ‘n’ and the int. Is your exporter making these odd commas? Or is it a feature of fbx?


                                                Key: ,
                                                        0,0.921003520488739,C,n
                                                        1847446320,0.921003103256226,C,n,

Anyway, I’d like to import the above fbx and am scratching my head over this one. :spin:

Actually, c’est accuse… export_fbx.py, line 2558:


                                                                        file.write('
						Key: ')
                                                                        frame = act_start
                                                                        while frame &lt;= act_end:
                                                                                if frame!=act_start:
                                                                                        file.write(',')

                                                                                # Curve types are
                                                                                # C,n is for bezier? - linear is best for now so we can do simple keyframe removal
                                                                                file.write('
							%i,%.15f,C,n'  % (fbx_time(frame-1), context_bone_anim_vecs[frame-act_start][i] ))
                                                                                #file.write('
							%i,%.15f,L'  % (fbx_time(frame-1), context_bone_anim_vecs[frame-act_start][i] ))
                                                                                frame+=1


Also at line 2587. It looks like using write() with prepended newlines is the trouble.
You can treat the first data line as a special case, or append newlines. But right now
I’m kind of surprised the keys were imported by other programs. Cheers.

Ah! thanks for pointing this out! - I didnt double check because the animation could be read by the sdk still… will fix.

PS, fixed in SVN trunk and stable, was a 1 liner… writing a parser seems a useful way to validate your own exports.
PPS, updated http://members.optusnet.com.au/cjbarton/fbxExample.zip too

Thinking about property being a special case or not…

cant they just be parsed as a list of values? - a bit like float list or int list but allowing for mixed types.
If this works well it could be used for parsing Relations: and Connections: too

Connect: “OO”, “Material::Material.003”, “Model::Hands.003”

Model: “Model::A.L”, “Limb” {
}

Hello,
I grabbed the latest zip and it’s identical to the first one, and I looked at the web subversion export code and
it still looks the same. Here is the md5sum:
eb6b64249ac08246502a97033949d31b fbxExample.zip

I was able to get a working model by running it through the Autodesk converter, fbx to fbx.
Here is a screenshot:
http://www.zettix.com/blender/fbx_kngcalvn_handsonly.jpg
All the meshes got made into one big mesh called ‘Model::Hands’ :spin:

Properties are not the only special case…
You mentioned Connect:
That is similar to Property: in that multiple connections clobber previous ones.
This is how the parser sees it:


[Connections][Connect] :   ['OO', 'Model::f2_02.R', 'Model::f2_01.R']

This is how fbx does it:


    Connect: "OO", "Model::f1_02.R", "Model::f1_01.R"
    Connect: "OO", "Model::th_IK.R", "Model::th_str.R"
    Connect: "OO", "Model::f1_IK.R", "Model::f1_str.R"

So what should [Connections][Connect] hold?
One approach would be to make lists of lists…


got {key: newvalue}, e.g. {'Connect' : ['OO', 'Model::f2_02.R', 'Model::f2_01.R'] }
If key in dict.keys():
  Assume newvalue is a list.  If it isn't, make it one.
  value = dict[key]
  if type(value) == list: #assume yes.
    if type(value[0]) == list: # already a list of lists
      dict[key].extend(newvalue)
    else:
      dict[key] = [dict[key], newvalue] # make it a list of lists
  else:
      dict[key] = [dict[key], newvalue]

Something like that. (The above code is buggy):stuck_out_tongue:

But later code will have to distinguish between a single Connect: and multiple ones.
One thing to keep in mind is that while the parser may be general and not tied to the fbx keywords,
the glue to get it into blender will be all about keywords, so while a special case looks bad in the parser,
the dict to blender code will be made of special cases and key words.

So maybe it’s better to treat Connect: special, like Property: is special.
Or whatever else has this issue. The code around Connect: to get the connections into blender will be many
times bigger than the code to parse it. Something to consider.

However, you’re going to be the primary consumer of this dictionary, so tell me how you want it.:evilgrin:

will post on other topics in a tic. just double checked http://members.optusnet.com.au/cjbarton/fbxExample.zip and it is looks correct to me.

Im not 100% clear on what your doing so Ill just try explain the way I think it should work as simply as I can…

connections should be stores as a list of pairs, each pair has a keyword, in this case - ‘Connect’ and a value - [‘OO’, ‘Model::Lamp’, ‘Model::blend_root’]

Using dictionaries is not ideal IMHO because it means you cant have 2 items with teh same name, or if you do, you have to store in a list of values under the key name {‘Connect’:[…all connect values]}.