Read data from file opened as Binary

Coding Python Support
#1

I would like to know how i can read the data when a file is opened in Binary mode “rb”. I got this file which is used for a render engine which contains a tuhmnail preview, which is BMP i believe. The addon already uses a piece of code to take out the data of that thmbnail image. But im also intrested to take the rest of the data out.

The beginning of the code is like this;

matFile = open(filename, "rb")
matFile.seek(0)
matData = matFile.read()

If i print matData is see the values i believe as HEX str, not sure. Here’s a small piece of that code.


 b'\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00

At the end of the file i see regular chracters mixed with HEX code. looks like this;


Focus Distance\x02 \x00\x00\x00\x00\[email protected]\x06\x00\x00\x00\x11\x00\x00\x00Radius Multiplier\x02 \x00\x00\x00\x00\x80?\x06\x00\x00\x00\x06\x00\x00\x00Radius\x02 \x00\x00 \x9b\xa4D\x06\x00\x00\x00\x08\x00\x00\x00Min Rays\x01 \x00\x00\x08\x00\x00\x00\x06\x00\x00\x00\x08\x00\x00\x00Max Rays\x01 \x00\x00d\x00\x00\x00\x06\x00\x00\x00\x05\x00\x00\x00Layer\x01 \x00\x00\x00\x00\x00\x00\x06\x00\x00\x00\x12\x00\x00\x00Light Buffer Index\x01 \x00\x00\x00\x00\x00\x00\x06\x00\x00\x00\x14\x00\x00\x00Interface Appearance\x03 \x00\x00\x00\x00\x00\x00\x05\x00\x00\x00\x04\x00\x00\x00\x0f\x00\x00\x00Global Settings\x0f\x00\x00\x00Global Settings\x00\x00\x00\x00\x0f\x00\x00\x00Global Settings\x06\x00\x00\x00\x10\x00\x00\x00Background Color\x07 \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06\x00\x00\x00\x0c\x00\x00\x00Illumination\x03 \x00\x00\x14\x00\x00\x00Image Based Lighting\x06\x00\x00\x00\x08\x00\x00\x00Sky Type\x03 \x00\x00\x07\x00\x00\x00Sun+Sky\x06\x00\x00\x00\x08\x00\x00\x00IBL Type\x03 \x00\x00\x07\x00\x00\x00Sun+IBL\x06\x00\x00\x00	\x00\x00\x00Sky Frame\x06 \x00\x00\xd8\xc4\x14?\xf7UP?\x00\x00\x00\x00\xf7UP\xbf\xd8\xc4\x14?\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80?\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x06\x00\x00\x00\r\x00\x00\x00Sun Direction\x03 \x00\x00\x1a\x00\x00\x000.514046 0.722707 0.462008\x06\x00\x00\x00	\x00\x00\x00Turbidity\x02 \x00\x00\x00\x00 @\x06

How can reach that part after the thumbnail image and get the data?

Iva attached the file which is used to load the data from

Attachments

Goedap_8k_270deg_Toned-v1.zip (20.7 KB)

#2

Reading a binary file is next to impossible unless you have the header format scheme. For example,
they may have a bit set at x0E, which signifies that the image is color, but how would someone who
does not have access to the header format know that? They wouldn’t, and, it would take a lot of
research to determine what each part of the header signified. Most headers have things in them
like file length, read only, etc. But as I said, getting at that information if extremely difficult unless
you have the key . . . the format of the information inside the file. Once you have that, it would
be a breeze, because then you know how to interpret what you see. You might be able to get lucky
by doing a search for that info, maybe, maybe not, it could be proprietary. If it is BMP, it is likely that
somewhere you could indeed find that info, if you are half-way decent at doing a Google search.
Then all you need to do is access it and use that info to interpret what you see.

#3

Well the header starts with \x03 and the a big chink of the first part is indeed BMP. Near the end i see regular characters followed by the same code format.

I can give a txt file with all the data printed to the console, is that useful?

#4

No because that will make for a very long print. Instead experiment.

I disagree with skywola that you cannot read the file , of course you can its nothing more than a collection of bytes. But then so is an executable.

Skywola is correct that having the format scheme will make things a ton easier and I agree with anything else he points in his post.

BMP format if memory serves me correctly is extremely simple and fortunately it does not use any form of compression (which would complicate things for you) BMP also does not alpha so the byte format should be the typical RGB, Because usually the values are 0 to 255 you can safely assume that it will use one byte for each color.

As Skywola said , finding the blueprint of the format should be your no 1 priority. Dont understimate how helpful people that worked on similar project can be.

If that is proven impossible, take a look at these bytes and load a blank BMP. The file will then have a large part of zeros in it , that part will most likely will be the one assigned for BMP data. You can also try diffirent colors to make sure you understand exactly the boundaries of BMP data inside the file.

#5

Well i some part i understand from the current script, yet i cant find all the info i need. Here’s is the complete script which loads a file which has a thumbnail in it and also other data. These are not files used by blender but by Thea Render, its a different render engine.

EDIT
i cant add the code, the stupid captcha check gives errors each time???

#6

Man i hate this new method with the captcha code, cant add the code now, keeps giving 404 error after i check the captcha

Attachments

preview.py.zip (2.02 KB)

#7

Lets take this from the start , which format you try to reverse engineer ?
which render engine ?

#8

Its for Thea Render, ive added a file in the beginning. In my last message i also added the complete py code to read a bitmap which is in such a file.

#9

Ok bare with me because I know nothing about Thea

Is the preview.py an actual importer or code you came up ?

Also are you aware there is an addon for Blender to export to Thea ?

ALL Blender addons are under obligation of the GPL license to release their code and addons are nothing more than python source anyway so technically speaking if that addon is a true exporter , which I assume it is, it will contain the format scheme you need to reverse the code and turn it from an exporter into an importer

All that assuming there is not already a Thea file importer . If there is then obviously you just copy paste code from it.

#10

That py code is just a small part of a very large py file. Im thinking its impossible todo.

Yes, im helping the guy with this plugin. Ive been using this engine since 2009, so yes i know this addon. Ive been adding options and extra to the plugin to make it a more versatile addon.

There is no importer for this part. It is something i would like to be able to. The guy which started this plugin also just told me, something about the GPL. We would probably need SDK part of Thea and that will then violatte with those GPL rules if im correct.

#11

Exporter and importer is doing the same thing the only diffirence is that one write and the other reads. The real meat is the spec of the format. If you have the one then you just copy paste to the other.

If there is an exporter I see no reason why you dont know how to make the importer. The exported will tell you exactly what to read and how from the file.

GPL code can use closed source code, afterall GPL depends on tons of OS libraries that are all closed. Including Blender. Its not possible to forbit such scenario because no code out there can function without OS libraries. What GPL forbids is being used by closed source.

So if Thea SDK used a Blender addon then yes it would have to become GPL which Thea devs may not want to do. But in this case you talk about using the SDK from inside your addon which would not cause any violation.

By the way you dont need an SDK to read a file if the exporter has the full specs. Unless the exporter depends also on the SDK because Thea wants to keep its format closed. I doubt it.

In any case you can contact the devs of Thea and ask them for specific clarifications, I am sure they will be very helpful at least to a degree.

#12

Well im trying to explain that the file is a preset file. These are normally loaded into the studio application itself. But GRZYBU made code so he reads out the thumnail image in the file which is saved in the file aswell.

There is a eporter for code to make a XML file which is used to import into studio or the render app. But these are preset files. Again look at the first post.

Problem is Thea devs hardly or never answer, is freaking terrible man!

Thea for blender addon is free to download, but you need a darkroom app which renders. Also you will need a license to render the images without watermarks.

#13

Ok I see now you are in deep trouble, bare in mind reverse engineering is illegal in some national laws so making an importer without their permission may violate their license and sue you. So you need to contact them anyway.

Just to be on the safe side. GPL is the least of your worries.

#14

hmmm… weird. THe files only contain some settings need to for the engine, there is no special or secret data in there.

But thanks for the mention, i think i will do so. Perhaps i get a bit further there. Thanks again for the help