Kinect

Human Body Recognition and Tracking:
How the Kinect Works

Kinect RGB-D Camera
• Microsoft Kinect (Nov. 2010)
– Color video camera + laser-projected IR dot pattern + IR camera

• $120 (April 2012)
• Kinect 1.5 due 5/2012
IR laser projector

color camera

T2
IR camera

Many slides by D. Hoiem
640 x 480, 30 fps

What the Kinect Does

How Kinect Works: Overview

Get Depth Image

IR Projector
IR Sensor

Projected Light Pattern

Stereo
Algorithm
Segmentation,
Part Prediction

Application (e.g., game)
Estimate body parts and joint poses

Depth Image

Body parts and joint positions

Part 1: Stereo from Projected Dots
IR Projector
IR Sensor

Part 1: Stereo from Projected Dots
1. Overview of depth from stereo

Projected Light Pattern

Stereo
Algorithm

2. How it works for a projector/sensor pair
3. Stereo algorithm used by PrimeSense
(Kinect)

Segmentation,
Part Prediction

Body parts and joint positions

Depth Image

Depth from Stereo Images image 1

image 2

Depth from Stereo Images
• Goal: recover depth by finding image coordinate x’ that corresponds to x
X
X

Dense depth map

z

x x’ x x' f
C

Some of following slides adapted from Steve Seitz and Lana Lazebnik

f
Baseline
B

C’

Basic Stereo Matching Algorithm

Depth from Disparity
X

x  x f 
 z
O O

z x’ x

f

• For each pixel in the first image

O

– Find corresponding epipolar line in the right image
– Examine all pixels on the epipolar line and pick the best match – Triangulate the matches to get depth information

f
Baseline
B

disparity  x  x 

O’

B f z Disparity is inversely proportional to depth, z

Basic Stereo Matching Algorithm

Correspondence Search
Left

Right

scanline

Matching cost

• If necessary, rectify the two stereo images to transform epipolar lines into scanlines
• For each pixel x in the first image
– Find corresponding epipolar