Experimenting with Kinect using opencv, python and open kinect (libfreenect)

The kinect is an amazing and intelligent piece of hardware. It has a RGB camera, an IR laser projector, an IR CMOS sensor, a servo to adjust the tilt of the device and a microphone array.  The RGB camera is like any other camera such as a webcam but it is the depth sensor that the Kinect is known for as it enables the Kinect to perceive the world around it in 3D !

Here are the steps to get started with using the kinect :-

Note :- This tutorial assumes that you have Linux(Ubuntu or Ubuntu based Linux distro) with opencv installed on your system.

1)  Open a terminal and run the following commands

sudo apt-get update
sudo apt-get upgrade

2) Install the necessary dependencies

sudo apt-get install git-core cmake freeglut3-dev pkg-config build-essential libxmu-dev libxi-dev libusb-1.0-0-dev

3) Clone the libfreenect repository to your system

git clone git://github.com/OpenKinect/libfreenect.git

4) Install libfreenect

cd libfreenect
mkdir build
cd build
cmake -L ..
sudo make install
sudo ldconfig /usr/local/lib64/

5) To use Kinect as a non-root user do the following

sudo adduser $USER video
sudo adduser $USER plugdev

6) Also make a file with rules for the Linux device manager

sudo nano /etc/udev/rules.d/51-kinect.rules

Then paste the following and save

# ATTR{product}=="Xbox NUI Motor"
SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02b0", MODE="0666"
# ATTR{product}=="Xbox NUI Audio"
SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02ad", MODE="0666"
# ATTR{product}=="Xbox NUI Camera"
SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02ae", MODE="0666"
# ATTR{product}=="Xbox NUI Motor"
SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02c2", MODE="0666"
# ATTR{product}=="Xbox NUI Motor"
SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02be", MODE="0666"
# ATTR{product}=="Xbox NUI Motor"
SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02bf", MODE="0666"

7) Log out and back in. Run the following command in a terminal to test if libfreenect is correctly installed


This should cause a window to pop up showing the depth and RGB images. Pressing ‘w’ on the keyboard causes the kinect to tilt up and pressing ‘x’ causes the kinect to tilt down. There are several other control options that are listed in the terminal when “freenect-glview” is run


8) In order to use the Kinect with opencv and python, the python wrappers for libfreenct need to be installed. Before doing that, install the necessary dependencies

sudo apt-get install cython
sudo apt-get install python-dev
sudo apt-get install python-numpy

9) Go to the directory ……./libfreenect/wrappers/python and run the following command

sudo python setup.py install

10) Save the code given below as a (.py) file say (kinect_test.py)

#import the necessary modules
import freenect
import cv2
import numpy as np

#function to get RGB image from kinect
def get_video():
    array,_ = freenect.sync_get_video()
    array = cv2.cvtColor(array,cv2.COLOR_RGB2BGR)
    return array

#function to get depth image from kinect
def get_depth():
    array,_ = freenect.sync_get_depth()
    array = array.astype(np.uint8)
    return array

if __name__ == "__main__":
    while 1:
        #get a frame from RGB camera
        frame = get_video()
        #get a frame from depth sensor
        depth = get_depth()
        #display RGB image
        cv2.imshow('RGB image',frame)
        #display depth image
        cv2.imshow('Depth image',depth)

        # quit program when 'esc' key is pressed
        k = cv2.waitKey(5) & 0xFF
        if k == 27:

11) Run the above program

python kinect_test.py


Hit the ‘esc’ key to quit the program

For further information, check
1) Open kinect page :- http://openkinect.org/wiki/Main_Page
2) libfreenect github page :- https://github.com/OpenKinect/libfreenect


Happy hacking 🙂