Experimenting with Kinect using opencv, python and open kinect (libfreenect)

The kinect is an amazing and intelligent piece of hardware. It has a RGB camera, an IR laser projector, an IR CMOS sensor, a servo to adjust the tilt of the device and a microphone array.  The RGB camera is like any other camera such as a webcam but it is the depth sensor that the Kinect is known for as it enables the Kinect to perceive the world around it in 3D !

Here are the steps to get started with using the kinect :-

Note :- This tutorial assumes that you have Linux(Ubuntu or Ubuntu based Linux distro) with opencv installed on your system.

1)  Open a terminal and run the following commands

sudo apt-get update
sudo apt-get upgrade

2) Install the necessary dependencies

sudo apt-get install git-core cmake freeglut3-dev pkg-config build-essential libxmu-dev libxi-dev libusb-1.0-0-dev

3) Clone the libfreenect repository to your system

git clone git://github.com/OpenKinect/libfreenect.git

4) Install libfreenect

cd libfreenect
mkdir build
cd build
cmake -L ..
make
sudo make install
sudo ldconfig /usr/local/lib64/

5) To use Kinect as a non-root user do the following

sudo adduser $USER video
sudo adduser $USER plugdev

6) Also make a file with rules for the Linux device manager

sudo nano /etc/udev/rules.d/51-kinect.rules

Then paste the following and save

# ATTR{product}=="Xbox NUI Motor"
SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02b0", MODE="0666"
# ATTR{product}=="Xbox NUI Audio"
SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02ad", MODE="0666"
# ATTR{product}=="Xbox NUI Camera"
SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02ae", MODE="0666"
# ATTR{product}=="Xbox NUI Motor"
SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02c2", MODE="0666"
# ATTR{product}=="Xbox NUI Motor"
SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02be", MODE="0666"
# ATTR{product}=="Xbox NUI Motor"
SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02bf", MODE="0666"

7) Log out and back in. Run the following command in a terminal to test if libfreenect is correctly installed

freenect-glview

This should cause a window to pop up showing the depth and RGB images. Pressing ‘w’ on the keyboard causes the kinect to tilt up and pressing ‘x’ causes the kinect to tilt down. There are several other control options that are listed in the terminal when “freenect-glview” is run

libfreenect_demo

8) In order to use the Kinect with opencv and python, the python wrappers for libfreenct need to be installed. Before doing that, install the necessary dependencies

sudo apt-get install cython
sudo apt-get install python-dev
sudo apt-get install python-numpy

9) Go to the directory ……./libfreenect/wrappers/python and run the following command

sudo python setup.py install

10) Save the code given below as a (.py) file say (kinect_test.py)

#import the necessary modules
import freenect
import cv2
import numpy as np

#function to get RGB image from kinect
def get_video():
    array,_ = freenect.sync_get_video()
    array = cv2.cvtColor(array,cv2.COLOR_RGB2BGR)
    return array

#function to get depth image from kinect
def get_depth():
    array,_ = freenect.sync_get_depth()
    array = array.astype(np.uint8)
    return array

if __name__ == "__main__":
    while 1:
        #get a frame from RGB camera
        frame = get_video()
        #get a frame from depth sensor
        depth = get_depth()
        #display RGB image
        cv2.imshow('RGB image',frame)
        #display depth image
        cv2.imshow('Depth image',depth)

        # quit program when 'esc' key is pressed
        k = cv2.waitKey(5) & 0xFF
        if k == 27:
            break
    cv2.destroyAllWindows()

11) Run the above program

python kinect_test.py

kinect_test

Hit the ‘esc’ key to quit the program

For further information, check
1) Open kinect page :- http://openkinect.org/wiki/Main_Page
2) libfreenect github page :- https://github.com/OpenKinect/libfreenect

 

Happy hacking 🙂

Bare Bones Breadboard Arduino

Ever felt you wanted a new Arduino for a project that you have been planning to do for a while, but reluctant to buy a new one or pull out an old one from another project ?

In this tutorial I will show you how to build a minimalist Arduino on a breadboard !

Here are the parts you will need :-

  1. Bootloaded AVR Atmega 328 microcontroller
  2. Crystal Oscillator (16Mhz)
  3. push button switch
  4. Resistors (10k, 330 ohms)
  5. two ceramic capacitors (22pF)
  6. USB to TTL converter
  7. led
  8. Breadboard
  9. connecting wires

So, let’s get started !

1) First of all, slot the micro controller and the push button switch onto the breadboard as shown below. Connect up the power and ground rails on either sides of the breadboard.

BBBarduino_bb1

Note:- You can buy an Atmega 328 micro controller that has already been bootloaded or you can bootload it yourself using another Arduino. http://arduino.cc/en/Tutorial/ArduinoToBreadboard                                                                 or, you could choose to use an AVR ISP to bootload the chip.

 

2) Build the clock circuit consisting of the crystal(16Mhz) and capacitors(22pF). The two leads of the crystal are connected to pins 9 and 10 of the micro controller. Add the pull up resistor(10k) to pin 1 of the micro controller. Also connect one end of the switch to pin 1 and the other end to ground. This acts as the Reset switch.

BBBarduino_bb2

 

3) Next, connect pins 7, 20, 21 to VCC and  8, 22 to Ground.

BBBarduino_bb3

 

4) Finally, connect the TTL converter to the micro controller and also add an led to digital pin 13 (pin 19 on the avr microcontroller) as shown below.

BBBarduino_bb4

 

 

A TTL converter looks something like this :-

20140407_232412

 

5) Install the drivers for the TTL converter, then try uploading the Blinky sketch. You would have to click the Reset button every time you upload any code to the chip. Some TTL converters come with a auto reset pin. You could get one of those if you like.

That’s it !

Happy hacking ! 🙂