Augmenting a webcam
posted by Robert Hodgin 
on March 13, 2009 at 02:36 PM
filed under: San Francisco, Processing, and Art
Robert Hodgin
Co-founder, Partner Emeritus : San Francisco
topics: Art Direction, Barbarian Labs, Visualizers, Barbarian History, and Design
filed under: San Francisco, Processing, and Art
A while back, I started experimenting with compositing content into a live webcam. Its a fairly simple process: you just need to intercept the webcam feed with something like Processing, add some content to the image, then send it on its way. The original execution involved setting buildings in San Francisco on fire.
To create the sense of depth, I cut the view into 6 masked layers so I could stick smoke graphics into the scene, not just on top of the scene. Eventually, this project moved to the back burner because I just wasn’t satisfied with the results.
A year later, I revisited the project intent on making it more entertaining. I went with a higher res still image instead of a webcam to see if I could do even more damage to this city I love.
Fun!, but ultimately a departure from what I wanted to accomplish because it didn’t rely on a webcam feed and the striking position of the meteorites was manually determined by clicking on the image (the click references a rudimentary depthmap to determine how far into the scene the meteor should strike).
And now we come to version 3. If you look out the back windows of the San Francisco office, you will see a rooftop with some skylights and a couple AC units. Its fairly dull back there so I decided it was the perfect candidate for augmentation.
First step, create an elevation map. This is a greyscale image that allows me to approximate how tall things are on the roof. The ‘floor’ of the roof would be a dark grey, the tops of the AC units would be light grey, etc. Since I don’t have access to their roof to make measurements, I decided to give Google Earth a go. I found the roof and zoomed in as much as I could and grabbed a screenshot. Sadly, it was highly obscured by shadows but it was enough for me to make a best guess.
Above, you see the original Google Earth view on the left, the elevation map in the middle, and the normal map which allows me to do some basic bounce angle tricks which I will discuss in a later post.
The next step, and so far the most difficult step, is to create a depth map. This is similar to the elevation map but instead of mapping the relative heights, I am estimating the distance from camera.
There are stereo cameras and other tricky ways to get an exact depth map but for my purposes, estimation should be good enough. It is definitely the most time consuming aspect of this project, but Id say I am about 85% satisfied with the depth map.
Now comes the fun part. For the purposes of this study, I decided to just drop a bunch of balls into the virtual space. After getting the in-applet camera to line up with the real world positioning of the webcam, I start dropping balls. Each ball checks the elevation map to see how far it can fall before encountering a surface. It also compares its own depth to the depthmap to see if it needs to disappear to give the illusion it is going behind a closer object.
The effect is fairly mesmerizing. By including a normal map, you can make the objects bounce off corners and angled surfaces. All you then need to do is add a fullscreen webcam view in the background and voila!! Instant occlusion!
It is still fairly messy and there a few major hurdles I am not sure just yet how to get past, but I think it shows a great deal of potential. If I could get position of the sun, weather conditions, and wind-speed involved, it would help sit the virtual objects in the space more realistically.
Next step would be to refine the physics so the objects cannot disappear into the sides of objects taller than the object’s current elevation. Id also like to allow viewers to shoot objects into that space. Perhaps some sort of rudimentary virtual skeeball. Or maybe just grow some grass on the roof and transform the space a la PS3’s “Flower”.
8 comments
It looks like there is a graphic used in the video, rather than a live webcam feed? I really like the idea of compositing over live things, too...
Looks like SketchUp might be the way to go. Looks like tweaking the 'fog' settings can get you a depthmap so all I would have to do is build this scene by hand. Shouldn't be too hard though the floor plane sags in the middle left.
Anyway, thanks.