VR / 360 Degree Movies

I’ve been kicking around the idea of 360 degree movies for the past year now, and I’m at the point now where I’m starting to experiment. When I say 360 degree movie, what I mean is a video that can be viewed with a VR device like the Oculus Rift or whatever else is out there. It’s not an interactive game, it’s not in stereoscopic 3D (at least not yet), it’s just like being in the center of a spherical screen- you can look in any direction and see what’s around you. It might sound too simple for some, but I think this is a good starting point.

Now, the content I’m planning would all be generated inside Blender. It’s basically like rendering any CG movie, except I’m using the panoramic camera with the “Equirectangular” setting. This way the seams are perfect (or nearly perfect), and I don’t have to worry about any funky and expensive camera rigs.

After editing and sound and all the usual workflow, you have a funny looking movie in a normal video format, like this:

And with the Oculus Rift, for example, the same video can be played back like this:

So it works. It might be a bit hacky or inconvenient, but it shows that it’s possible for me to upload a film in a normal video format onto my YouTube channel, to then be downloaded and played back on a VR device and experienced in 3D.

Now that I know it’s possible, I wanted to know if anyone on here actually has an Oculus Rift? It will be a while before I choose to purchase one, but I’m going to be testing in the meantime. Would anyone like to volunteer to try out the short video animations, and give feedback on them? Any feedback at all, like “it was blurry” or “I could see seams” or “the camera movement made me puke”.

I think 360 degree movies are something that will gain popularity in the near future! Let me know your thoughts!

Is that even possible? You’d need two cameras for stereo, but with each camera having a 360 view, it would be able to see the other camera. Your left eye would see the right camera, and your right eye would see the left camera.

Steve S

You’re right, Steve; for physical cameras that are recording a scene, it would take a clever camera rig to get around this issue. Since you would have to use multiple cameras anyway to shoot enough coverage of the scene for 360, you could arrange them in pairs, and then match points in the footage to generate depth maps. This is similar to the process used with the Spheron HDR camera on “The Smurfs” film:

In fact, it may be possible to generate depth maps without using multiple cameras- I know the new Lytro lightfield cameras generate depth maps. Although you would still be limited by either not moving the camera at all, or coming up with clever (or not so clever) ways to move it- on a car, held by an actor, or simply having a camera operator in part of the shot.

Fortunately, if the content is generated in CG, you don’t have to worry about this. You can also still use compositing to incorporate live action elements into your scene, so it needn’t be purely CG. As for achieving separate images for right and left eyes, I would expect someone to come up with a solution in time- a new video format that is specifically for 3D playback would be the simplest answer. Or does such a format already exist?

I rendered an image using one of Stonemason’s models. It’s just a test so it’s not perfect looking, but it should be the type of image that will work.

Does anyone here have a VR device? If so, is there a program that allows you to load single images and view them? I’d be very interested to hear how the image looks.

I think the biggest issue would be resolution. Given a 1080p display divided into 2 eyes, and an approx 90° hfov, the full panoramic image would have to be in the 4k neighborhood. Not too inconcievable, given the pace of hardware, but you are looking at some pretty huge filesizes and a demand for a very powerful system.

If you’re willing to sacrifice some resolution, I bet you can make a pretty functional 360° vr viewer. as to solving parallax issues, I have no solutions for that, short of rendering off a set of panoramas offset by an eyes distance apart and doing some very clever and possibly nausiating swapping based on head rotation.

As a side note, I use a similar concept for pre-rendering off hdr backdrops for a lot of my scenes. I have an 8k equirect pano of the inside of a convention hall that I use as a backdrop for a lot of my renders (I work for a company that builds tradeshow exhibits). It works pretty well, but If I change my eye elevation, it throws a lot of things off. it works ok in a pinch, but it can cause some headaches when the angles aren’t right. I’ve considered having different panoramas for different elevations, but I don’t want to have to manually swap them out. I could set up some sort of environment node group to swap them out dynamically, but loading multiple 8k images into memory will really put some strain on my videoram. Lately, I’ve been rendering with plain white backgrounds and nobody has complained yet. The part of me that wants to make the most believable CGI I can really doens’t like it, but the clients and my rendertime sure appreciate it.

That’s interesting to hear about your use of environment maps for your work, SterlingRoth. I seem to remember someone talking about trade booths on one of Anthony’s threads; was that you?

You’re right about the resolution required, it would have to be very big. It looks like the current Oculus Rift development kit has a resolution of 960x1080. The final consumer version will supposedly be higher resolution, and I’m sure future versions will be even higher. I think by normal standards, video file sizes would have to be quite large in order to play sharply. But seeing as how YouTube is already starting to make 4K videos streamable on-line, I think the large resolution will be more common in the near future, and something that many people will be able to play on their computers.

As for the issue of how to make such videos sterescopic, I’m not too concerned about how to do that at the moment. If it is possible, and without too much work required, I’d give it a shot for a video. But for the time being, my focus would be on getting the 2D 360 movie format figured out, before moving on to something more advanced. Honestly, I’d be more interested in getting the short film into a game engine format, where all the lighting is baked into the scene and you can move around in true 3D space as the scene progresses. This would be interesting to try out for a short action film.

I do believe you can’t beat a good, old fashioned 2D movie in a lot of situations. But I think it’s important to try these new types of formats, because we won’t know what it’s like until we do. Maybe a 360 degree movie will end up being more fun for certain types of stories!

Yeah, that was me, talking business whenever I can.

on a doubly related note, I was at E3 back in 2005, and the Xbox booth had a massive (100’+) circular projection screen, about 16’ tall. They projected panoramic content on it in a very similar way to what you are describing. there was a mafia sequence where someone in front of you gets shot and you turn around and the shooter is behind you, or racing game footage with a car drifting around a big turn, all caught on the 360° screen. It was pretty cool, especially with good surround sound.

I think the idea of doing a pre-baked scene that you can explore and view from whatever angle you please. Almost like a cutscene in halflife, but less interactive.

Very cool idea! :slight_smile: I don’t think this would be accepted for film, though. Most non-interactive film is an art of framing shots and directing the viewers’ attention. This would oppose that goal.
Now, architectural visualizations would love this!

That circular projection system sounds really cool, and massive! Also, incredible to hear you’ve been to E3!

Because I’d like to also start incorporating scanned models into my films, I think the same data could be useful in this type of content.

I have two ideas for a good “easy” 360 film- One would be a cinematic tour of a post-apocalyptic city, with ruined buildings and remnants of the last days of humanity. Maybe show nature reclaiming the land. A bit depressing, but it would be a good way to see how much the format can immerse the viewer and make them feel like they’re there. Plus, there are plenty of people that enjoy depressing stuff, me included!

My second idea is to use the scanned data, and create one of those “frozen time” sequences, where little or no time passes during a big action sequence. The War Thunder Heroes trailer did something like this. This would be an easy way to create a video with people in it, without having to build a camera rig to do so.

So those are the two ideas I’m working towards for now, with normal versions of each so people without VR devices can still enjoy them normally.

You’re right, Austin! It’s not exactly a film, but even if I’m able to create a visual story where you can move around in true 3D space, it still wouldn’t be a game, because you’re not really interacting with it; you’re just a spectator. I don’t know exactly what to call it, but for now I’ll just stick with 360 movie I suppose.

Certainly, I think traditional film making is something that will always be relevant, and a true art form, so I don’t foresee this type of thing as replacing it ever. And you’re absolutely right, architectural firms would love this! Someone should get started on that, they could make a fortune!

With the 360 Video Ball application, you got a poor man’s VR device, that can read 360 videos.

The Google Camera application allows to look at 360 panoramas interactively too, although I only tested it on “photo spheres” that I took before.

Might be interesting to look at how it renders on Android phones.

VR player should work, though i would suspect that this would not be decent enough resolution for the rift (you are looking at approx 90FOV, at 640x720 per eye, that would be approx 2560 across that you will need atleast for the rift 1, probably 4k for the rift 2.

Depends on their client… if they are trying to fake how much room they actually have, it will be very evident in VR… whereas in a normal flythrough / visualization they can get away with it.

haha, Very true.

Thanks for those links, SolarLiner! I had never heard of them until now, I’ll try them out sometime soon!

doublebishop - You’re right, I think the demo in the video from my original post actually used VR Player (could be mistaken). If it’s freely available I should probably download it and see if I can preview the interface just using my computer monitor and mouse movement. If the video resolution needs to be 4k, I think that is achievable. That would mean the height would be 2k, which would actually be a little less than a normal 16:9 4k video, so if a viewer can play that on their computer, they should be able to play this. Hopefully before the end of this year I’ll have a demo or proof of concept that I can test out on my own VR headset!

I forgot to post an update! I downloaded VR Player (http://vrplayer.codeplex.com/), loaded up my image, set Projection > Sphere and Distortion > None. Since I’m only viewing with my PC monitor, I also set Layout and Format both to Mono. If you try it out, let me know if it works for you! I’ll be rendering my first proof of concept video this month!