Proposal for Particle System

Hi everyone.

I’m a newcomer to this forum, and first it looks like a right time to introduce my self. I’m a postgraduate student studying traffic and transportation engineering, more specifically traffic simulation in China. I got to know blender quite early in 2003/2004 when I’m undergraduate but didn’t use it heavily until recently.

Unlike most people in this forum, I’m a sort of tech guy more than a artist(I am even not sure whether I’m a technique guy :o ). So my main use of blender is visualize data set of various kind.

Current I’ve finished a mesoscopic traffic simulation package and with it I can simulate the vehicle traffic in a large road network in Shanghai, China(100000+ vehicles in an area of 16km*16km). The result is huge vehicle trajectory data set and I’m looking for a way to visualize it. Although the simulation package has a GUI that visualize the moving of vehicles using OpenGL but the result is not impressive at all (remember I’m a tech guy:eyebrowlift2:).

But I need to impress my advisor and more importantly the bureaucrats who control our research budgets. Here blender comes as a rescure. The basic idea is import vehicle trajectory data in blender and then have my friend(he is a good artist as you guys:yes:) add some fancy things like shining car model, bright sky, trees, grass, buildings, roads… This way I’m sure I can give them an impressive presentation.

Motivation

Enough introduction, here comes the real tech problem.

Currently I’ve done some preliminary work use blender 2.4x and python script:
http://blenderartists.org/forum/showthread.php?t=161403

the basic idea of the script is that for every vehicle in the simulation output an object is add in blender with ipo curves of location, rotation, layer. The location, rotation ipo controls the moving and turing of vehicle object; the layer ipo controls the emerge and disappear of vehicle object. That is, for a specific vehicle object, the layer ipo puts it on layer1 when it’s traveling in the road network and layer2 before the time entering and after the time leaveing the road network .

I repost the screen shot here, the above description should make more sense with it.
http://i32.tinypic.com/6zuc1c.jpg
full size version:http://i29.tinypic.com/mwqs91.jpg

This method is working in my experiment but has two fundamental flaws.

First,the above mention object+ipo method needs a object in blender for every vehicle in my data set even its not entering or already leaving the road network. It’s not a problem in my experimental scene with 1000+ vehicles and no fancy things like buildings, roads… I’m not quite sure how it will work with the real scene with 100000+ vehicles and all kinds of fancy things.

Second, and more importantly, it’s very hard for my friend(still remember him? the art guy:eyebrowlift2:) to manipulate all vehicles as a whole. For example, some times he wants to scale and translate all the vehicles to match the underlying road meshes. Now he has to choose every vehicle objects in the scene and then do the manipulation, this process is not only tedious but also error prone . It’s quite hard for him to choose exactly all vehicles with no missing vehicles and no extra objects. Even worse it’s practically impossible for him to adjust the time factors of vehicle moving(like for a fast forward effect).

I’m now considering to use particles system as the solution. Indeed, the particle system should be an idea solution for my problem. Although I’m not familiar with the blender particle system, but my experience with particle system in real time graphical engine like Ogre tell me that particle system will only generate a particle when it comes to life and will “recycle” it after its death. That’s exactly the solution to the first problem above mention.

As for the second problem, we can solve it elegantly by put all particles(vehicles) in the local coordinate system of the object which the particle system stick to. Then we can manipulate all vehicles as a whole through manipulate the object that has the particle system. The time factor and time translation could be represented as parameters to the particle system.

Proposals

If I understand right, with current particle system I cannot do what I want to do, because there is no way I can specify the precise emerging time, dead time, emerging position, moving trajectory for a specific particle. All I can do is specify the number of particles I want, the spacial and temporal distribution of particle generation, particle moving rules(physics like newtonian, boids, …) then let blender particle system to “simulate”(or bake) the particle emerging time, dead time, emerging position, moving trajectory into the cache.

If you think vehicles as a particle my simulation package did the same thing as blender particle system, so basicly all I need is a method to put my simulation result directly in the cache. I think this wouldn’t need a major change in the blender particle code, just a new physics type(maybe call it keyframed physics:yes:) or even better some kind of python api deaper into particle system will do. I think if this can be done, all my needs in my current project will be satisfied.

I fully appreciate that all blender Dev guys are busy coding 2.5 (I’m eagerly looking forward to its final form, thank you guys), and I also want to take my part in by adding this small piece. I‘m a decent c++ and python programmer(the dev language for my traffic simulation system) but has no previous experience in blender source code could anyone give me some direction as where should I started(doc is good, code is better), thank you.

Any comments are welcome, not only about the idea its self but also about the way I carry it out, even comments on my grammar error is helpful.(As you can see, I’m chinese, and trying to translate my paper in chinese into english for international conference of course for traffic not for blender:p.)

p.s. Thank you for you patient for such a long and verbose post. In fact there is some further thought on particle system with animated object, not quite related to my current project but interesting. Although I’m not sure if it’s original, but I’ll try to explain if someone is interested.

I would recommend that you check out the Jahka Particles blog. This is basically the homepage for Blender’s particles development.

http://jahkaparticles.blogspot.com/

yupp that’s the guy doing the particles ;D

cool project btw, and nice if you can get back to us when you are done !! definitly something for blendernartion news IMO

cheers from the other side of the earth

i’m pretty sure that 100000+ vehicles is too much for blender. it’s just too much data. too many objects, too much geometry, even if they’re all only instances from the same car mesh and of course too many IPOs/curves (4 for each object i’m seeing in your image) with too many keys. everything needs memory!

check how much ram blender consumes with 1000 vehicles and then multiply it by 100 or more. do you have enough memory to handle this?

however, i could think of hardwiring your simulation into blender working as an own kind of particle system but that would probably be a lot of work. on the other hand you could indeed consult Jahka.

I think it could be done if it’s done in passes. There shouldn’t be a need to render this as a single render, or even have it inside of one blend. Hopefully, you can parse your data and develop the final animation one aspect at a time, and composite the results.

Easier said than done, though.

First off–very cool. Please post the video when you get it finished.
Second–your grammar is very good. I didn’t know you weren’t a native speaker until you mentioned it.

As far as particles go, getting in touch with Jahka is the way to go.

If particles don’t work, here are a couple ideas that might help…
You could parent all the cars to an empty and then scale the empty.
You could just scale and move the environment to fit the cars.

And like kernond said, you might need to do it in multiple passes.

Hi kernond

I would recommend that you check out the Jahka Particles blog

Thanks, I’ll take a look at it. btw are you the guy behind blendernewbies?It’s really helpful, because I’v been newbie for a long time since 2003:).

Hopefully, you can parse your data and develop the final animation one aspect at a time, and composite the results.

It’s possible. But I’m trying to avoid this. There will be many tedious maintenance work involved especially when you are dealing with constantly changing requirements

Hi aermartin

cool project btw, and nice if you can get back to us when you are done !!

Thanks. The project will not be done for a while, but I’ll try to release the animation of smaller experimental scene soon.

Hi Kai Kostack
Thank you for your advice.

(4 for each object i’m seeing in your image)

It’s 5. There is an ipo curves for Layer. Sorry I forgot to check it in the screen shot:).

check how much ram blender consumes with 1000 vehicles and then multiply it by 100 or more. do you have enough memory to handle this?

It’s 150M for a 1000 vehicles scene. But I think multiplying 100 is not an appropriate estimator. For one thing I’ll generate the vehicle object on the fly when it’s entering road network; For another I’ll recycle vehicles that are out of networks. I estimate there will be on more than 5000 vehicles on the network at a given instance.

i could think of hardwiring your simulation into blender working as an own kind of particle system

My thought is write a particle system that not “simulate” particles but “key frame” particles. The simulation work is done by my simulation package and the key frames are already generated.

Hi RobCozzens

First off–very cool. Please post the video when you get it finished.
Second–your grammar is very good. I didn’t know you weren’t a native speaker until you mentioned it.

Thank you, for both points. The reason why I was not quite confident with my english is that my first english paper was rejected with a major reason of bad grammar…:frowning:

If particles don’t work, here are a couple ideas that might help…
You could parent all the cars to an empty and then scale the empty.
You could just scale and move the environment to fit the cars.

These tricks are evil:), as I explained in the main post. But if the deadline is due I’ll consider them.

p.s. It’s sad. In China the blogspot, youtube, twitter, wordpress even google will be blocked once in a while by the GFW. I’m visting Jahka Particles’s blog through proxy, but all flash content was filtered by the proxy…:frowning:

Once in a while? So when is the Golden Shield enforced more or less strongly? Just curious. Also, that must suck. But not so much that you can’t contact Jahka, right?

Not sure. No one really knows how the golden shield work except for the big brother. Fortunately email is working fine, maybe for the reason there’s not many netizen use it:)

Hi xiongyaohua,
this is an interesting project indeed (I did my bachelor’s thesis on traffic simulation with cellular automata)! It’s a shame you’re blocked from viewing most of my blog content, but I’ll pm you my mail address so you can contact me that way. I have some plans on making particles capable of reading their motion etc. data from external files and have received requests about this from other too, so I think we can very well work something nice out.

If you like, I can host a few of the videos on my server in a zip file. That would be blendernewbies.com, so maybe there wouldn’t be a problem accessing them.

Hi jahka
I see your pm, and a mail is sent. Looking forward to your reply.

Hi kernond
It’s very nice of you to offer such help, thank you. Fortunately I figure out a better way to solve this problem—
The hotspot shield free vpn service. So you don’t need bother to do that.

p.s. hotspot shield VS golden shield, the name is interesting:)

Hi xiongyaohua

I had a very similar request (being able to define the initial state of the particle cache/cloud), ideally through Python. I think the amount of requests looking for a way to handle particles through scripts is increasing and am looking forward to maybe seeing something happening in 2.5 "soonish"™.

Cheers

Hamu

Hi everyone

This is the result of my final object+ipo effort to solve my visualization problem. I post it here for people that might interested. It’s a sub area(about 1/5) of my simulation zone, and 2000 vehicles in 3 minutes.


http://www.vimeo.com/5688025
(sorry for the dark background in video, because I think at night you can see the traffic pattern more clearly)

From now on I’ll turn my full effort in the particle thing.

Hi hamunaptra

looking forward to maybe seeing something happening in 2.5 "soonish"™

I hope so too:).

Hey, I really like those results! I’ve always found it interesting to see the flow of dense traffic.

That demo is amazing! Good luck with this project. Do you expect to be able to perform realtime representations of local groups (not whole data set)?

Shhhhhhhhhhh…you don’t want China blocking this site do you, there’s multiple Chinese users here, they probably already block the off-topic forum though:spin:

That is a very nice start, about the comment on the vimeo site, I don’t see cars teleporting or disappearing as too big of a problem right now, if you played SimCity at all you see a lot more of that than shown in the video.

Also, you’ll literally flip with joy after seeing the big commit by Jahka for 2.5’s particles which is the start of what he calls Boids v2. Check the 2.5 development thread.

Hi kernond

Hey, I really like those results! I’ve always found it interesting to see the flow of dense traffic.

The real fun part of do simulation is not “see” traffic flow but “play” traffic flow. I often block some road or change the signal in the simulation process and think if I do that in real city…:evilgrin:

Hi 3pointEdit

That demo is amazing! Good luck with this project. Do you expect to be able to perform realtime representations of local groups (not whole data set)?

Thank you. Yes, it is. There is a plan to show off a super complex control logic at a super big intersection of 5 main roads.

Hi Cyborg Dragon

Shhhhhhhhhhh…you don’t want China blocking this site do you, there’s multiple Chinese users here, they probably already block the off-topic forum though

It’s ok. The big brother watch big social site like blogspot and twitter more closely.

That is a very nice start, about the comment on the vimeo site, I don’t see cars teleporting or disappearing as too big of a problem right now, if you played SimCity at all you see a lot more of that than shown in the video.

I check the video again. There are some vanishing cars but it’s the feature of underlyine simulation package.

Also, you’ll literally flip with joy after seeing the big commit by Jahka for 2.5’s particles which is the start of what he calls Boids v2. Check the 2.5 development thread.

Yes, I’m flipping:spin: :slight_smile:

Boids 2?! I thought it was keyed particles that was the new thing. (they’re cool too, look what I did:

http://i614.photobucket.com/albums/tt222/ShnitzelKiller/Picture4-3.png

)

I’ll have a look at that too…

Commit by jhk :: r21810 /branches/blender2.5/blender/ (8 files in 5 dirs): (link)
External cache

Particle point cache can now be loaded from external files.
Activated by “external” checkbox in cache panel and giving proper folder path and file name identifier.
External cache panel has controls for particle emission start, end, lifetime and random lifetime. These should be set according to the external data for correct playback.
External files should be named “identifier_frame_index.bphys” or “identifier_frame.bphys” where: * “identifier” is a freely choseable name. * “frame” is the cached frame number. ** Six digits padded with zeros!, for example “000024”. * “index” can be used to tell caches with the same identifier apart. ** Two digits starting from zero. ** The index and the underscore before are optional. If no index is present the index number in ui should be set to -1.
Cache file format is pure floating point numbers (in binary, not text!) with each particle’s data one after the other with the following data members: * 3 floats: particle’s location vector * 3 floats: particle’s velocity vector (per second) * 4 floats: particle’s rotation quaternion * 3 floats: particle’s angular velocity vector (per second) * 1 float: frame of the actual data (this can be non-integer for particles that are born or die between two integer frames, but otherwise should be the same as the “frame” in the file name)
Cache files don’t have to exist for each frame. * Frames without actual data are interpolated from surrounding frames that have data (extrapolation is not supported).
Cache file formats with extended (or reduced even) data members are in future plans for easier usage.
Current code only does particles, don’t yet know if it’s applicable to cloth or sb.
Known issue: endianness can’t yet be handled in any way. Other changes: New hard limits for many particle parameters. Some examples:
Maximum amount of particles: 10M particles :slight_smile: And before you all go and crash your Blender trying this out remember that this limit is only for those freaks who really have the machine power to handle it. 10M particles alone take around 2.2 Gb of memory / disk space in saved file and each cached frame takes around 0.5 Gb of memory / disk space depending on cache mode. * Known issue: To actually use this many particles they most likely need to be allocated in parts as taking hold of a 2.2Gb chunk of memory at once is probably not ok with any operating system.
Maximum amount of children: 100k children/particle (1T childparticles here we come :D)
Kink frequency: -100k to 100k half-rotations (really strange the previous limit was only from zero upwards)
Path draw steps: 10 (power of 2 remember)
Path render steps: 20 (power of 2 also!! If over 1M segments doesn’t get you smooth paths then I think nothing will!)

Finally, it’s here. The particle system will have even greater potential and it’s exactly what I need. Thank you jahka for you awesome work.:).

I’m compiling the newest svn code, I can’t wait to see how it works.