Near-perfect virtual reality of recent times and tourism


Recently I tried Facebook/Oculus Rift Crescent Bay prototype. It has more resolution (I will guess 1280 x 1600 per eye or similar) and runs at 90 frames/second. It also has better head tracking, so you can walk around a small space with some realism -- but only a very small space. Still, it was much more impressive than the DK2 and a sign of where things are going. I could still see a faint screen door, they were annoyed that I could see it.

We still have a lot of resolution gain left to go. The human eye sees about a minute of arc, which means about 5,000 pixels for a 90 degree field of view. Since we have some ability for sub-pixel resolution, it might be suggested that 10,000 pixels of width is needed to reproduce the world. But that's not that many Moore's law generations from where we are today. The graphics rendering problem is harder, though with high frame rates, if you can track the eyes, you need only render full resolution where the fovea of the eye is. This actually gives a boost to onto-the-eye systems like a contact lens projector or the rumoured Magic Leap technology which may project with lasers onto the retina, as they need actually render far fewer pixels. (Get really clever, and realize the optic nerve only has about 600,000 neurons, and in theory you can get full real-world resolution with half a megapixel if you do it right.)

Walking around Rome, I realized something else -- we are now digitizing our world, at least the popular outdoor spaces, at a very high resolution. That's because millions of tourists are taking billions of pictures every day of everything from every angle, in every lighting. Software of the future will be able to produce very accurate 3D representations of all these spaces, both with real data and reasonably interpolated data. They will use our photographs today and the better photographs tomorrow to produce a highly accurate version of our world today.

This means that anybody in the future will be able to take a highly realistic walk around the early 21st century version of almost everything. Even many interiors will be captured in smaller numbers of photos. Only things that are normally covered or hidden will not be recorded, but in most cases it should be possible to figure out what was there. This will be trivial for fairly permanent things, like the ruins in Rome, but even possible for things that changed from day to day in our highly photographed world. A bit of AI will be able to turn the people in photos into 3-D animated models that can move within these VRs.

It will also be possible to extend this VR back into the past. The 20th century, before the advent of the digital camera, was not nearly so photographed, but it was still photographed quite a lot. For persistent things, the combination of modern (and future) recordings with older, less frequent and lower resolution recordings should still allow the creation of a fairly accurate model. The further back in time we go, the more interpolation and eventually artistic interpretation you will need, but very realistic seeming experiences will be possible. Even some of the 19th century should be doable, at least in some areas.

This is a good thing, because as I have written, the world's tourist destinations are unable to bear the brunt of the rising middle class. As the Chinese, Indians and other nations get richer and begin to tour the world, their greater numbers will overcrowd those destinations even more than the waves of Americans, Germans and Japanese that already mobbed them in the 20th century. Indeed, with walking chairs (successors of the BigDog Robot) every spot will be accessible to everybody of any level of physical ability.

VR offers one answer to this. In VR, people will visit such places and get the views and the sounds -- and perhaps even the smells. They will get a view captured at the perfect time in the perfect light, perhaps while the location is closed for digitization and thus empty of crowds. It might be, in many ways, a superior experience. That experience might satisfy people, though some might find themselves more driven to visit the real thing.

In the future, everybody will have had a chance to visit all the world's great sites in VR while they are young. In fact, doing so might take no more than a few weekends, changing the nature of tourism greatly. This doesn't alter the demand for the other half of tourism -- true experience of the culture, eating the food, interacting with the locals and making friends. But so much commercial tourism -- people being herded in tour groups to major sites and museums, then eating at tour-group restaurants -- can be replaced.

I expect VR to reproduce the sights and sounds and a few other things. Special rooms could also reproduce winds and even some movement (for example, the feeling of being on a ship.) Right now, walking is harder to reproduce. With the OR Crescent Bay you could only walk 2-3 feet, but one could imagine warehouse size spaces or even outdoor stadia where large amounts of real walking might be possible if the simulated surface is also flat. Simulating walking over rough surfaces and stairs offers real challenges. I have tried systems where you walk inside a sphere but they don't yet quite do it for me. I've also seen a system where you are held in place and move your feet in slippery socks on a smooth surface. Fun, but not quite there. Your body knows when it is staying in one place, at least for now. Touching other things in a realistic way would require a very involved robotic system -- not impossible, but quite difficult.

Also interesting will be immersive augmented reality. There are a few ways I know of that people are developing

  • With a VR headset, bring in the real world with cameras, modify it and present that view to the screens, so they are seeing the world through the headset. This provides a complete image, but the real world is reduced significantly in quality, at least for now, and latency must be extremely low.
  • With a semi-transparent screen, show the augmentation with the real world behind it. This is very difficult outdoors, and you can't really stop bright items from the background mixing with your augmentation. Focus depth is an issue here (and is with most other systems.) In some plans, the screens have LCDs that can go opaque to block the background where an augmentation is being placed.
  • CastAR has you place retroreflective cloth in your environment, and it can present objects on that cloth. They do not blend with the existing reality, but replace it where the cloth is.
  • Projecting into the eye with lasers from glasses, or on a contact lens can be brighter than the outside world, but again you can't really paint over the bright objects in your environment.

Getting back to Rome, my goal would be to create an augmented reality that let you walk around ancient Rome, seeing the buildings as they were. The people around you would be converted to Romans, and the modern roads and buildings would be turned into areas you can't enter (since we don't want to see the cars, and turning them into fast chariots would look silly.) There have been attempts to create a virtual walk through ancient Rome, but being able to do it in the real location would be very cool.


Just one comment (other than "I agree with all of this, and have thought about quite a bit"); It will not be required to have a sterile capture environment with no crowds in order to reproduce things correctly. It's pretty trivial with more highly photographed sites right now to remove people from frames just by comparative elmination. I'm guessing you've seen the Microsoft demos with fully modeled and re-rendered environments based on a single 2d FPV GoPro camera? Being able to multiplex multiple stills and video will allow easy creation of scenes with no interference. My question is this: what is the universal language of storage for these scene data? The combination of 2d, 2.5d, and 3d data will need to be combined in some way and preferably stored in one single syntax to allow for rapid retreival and presentation. Who is doing this type of work now?

Yes, you can remove the crowd from a 3-D model. I apologize for talking at cross purposes a bit, because when it comes to capture of the world's great sites, I have been interested in doing it even before we can build that accurate 3D model and re-render it, and so I have also been interested in the question of just top-quality panoramic video, with no ability for you to walk around and change your viewpoint. For that, it's a little harder to remove the people in the scene, though not entirely impossible. Aside from software tricks, you can also just put the camera gear on a tower, above everybody, in many locations to get a view not even available if you visit in person.

For viewing, say, Yosemite Falls from the viewing platform, it is not necessary to be able to walk around the small viewing area, and a view from a tower above it would be great. There are many locations like this.

Are you aware of anyone working on a consistent, structured format for VR scene data that is "modern" and open? Preferably one that can digest both 2d, 3d, and 3d synthetic...

If you're interested in VR, I had an idea you might like. When people tell me that they won't ever accept autonomous vehicles because they "like to drive", I point out that I'll be playing car racing games during my commute. It seems possible that the ultimate combination of those technologies would be a VR system where you were in a simulated race but the course was cleverly contrived to make your progression through it provide the same kinds of physical g-forces your actual car ride will experience. Since the autonomous cars can plan so well ahead what kinds of turns and stops will be coming, it should be possible to match this to a racing experience. If you get out of sync through your own agency, then in the game, you'd just lose or crash, etc.
Anyway, since I follow your autonomous car posts I thought you might find this amusing.

I have thought of having somebody play a racing game as a cute and ironic demo, but I am not sure you could really make the g-forces come close to matching. A real drive wants to minimize those forces -- I have even contemplated vehicles that tilt to minimize lateral forces of all kinds. A race is all about those forces.

Racing games on the couch are pretty fun. Even the tiniest feedback would make a huge impression. So even cars operating on algorithms to minimize jostling would have potential. But if you're controlling for physics forces, you could easily keep them turned up. Tilting vehicles on gimbals or something could both minimize passenger forces or, intensify them. If the technology will be there for the 99% of people who want a smooth ride, it will probably be there for the 1% who want F1 physics. :-)

That's not the future photogrametry exists to today. Photos from same place can already be automatically be put together to form models almost as accurate as lidar with no artist required. Which do you think will become cheap enough for consumer robocars first, lidars or the processing power required for real time photogrametry?. Some cameras at each corner of the car should provide much more information than the human head with only two eyes a couple of inches apart. Is Lidars problem with rain solvable?

It is technologies like that which inspired the post, and while they have been around for a few years they are not quite up to the level I am talking about, which requires a little bit of AI to fill in the small numbers of missing places.

No, today cameras are not suitable for robocars. I have a long article on the issues. Humans drive with sight, but stereo plays a very small part -- as you will know when you realize that everybody can drive with one eye closed. Motion parallax probably plays a bigger role than stereo, actually. When with cameras 5' apart stereo isn't enough for arterial road speeds.

The company has some pretty impressive robovision running off cell phones and their cameras.

Oh man robovision sounds awesome. Back when I first learned about 3d printers, even that blew my mind on so many levels.

Add new comment