Kinect Page

From CSclasswiki
Jump to: navigation, search

--Thiebaut 12:28, 12 April 2011 (UTC)


"Kinect is to multitouch user interfaces what the mouse was to DOS."



James McQuivey




Welcome to the Kinect Page, a repository of ideas, algorithms, code, and anything Kinect-related. Feel free to add your discoveries as well as contribute code to this page!


Contents

Kinect2.jpg



News

Repository of Kinect-Related Material

KinectHacksDotNet.png

Meet Microsoft's Kinect in Action

This section contains several movies that highlight some of the interesting applications people have found for the Kinect. It is in no mean comprehensive, and does not mean to be. YouTube has a great collection of movies on the subject.



Johannes Kreidler's Kinect Studies #1

Kreidler uses a Kinect to detect how motion on the screen interacts with surfaces and lines created on the video display. Thinking out of many boxes!




Johannes Kreidler's Kinect Studies #2

As with his Studies #1, Kreidler uses a Kinect to detect how the motion of a human makes the skeleton frame interact, intersect, various surfaces, lines, directions on the video display.





Facial Recognition with Kinect

Interesting video from truebones.com showing how the software extracts facial elements using Kinect and uses them to animate a smooth rendered human face. The software is called FaceShift.


KinEtre: Animating real objects with human motion

KinetreAnimatedObject.png

From microsoft research: "Imagine you are asked to produce a 3D animation of a demonic armchair terrorizing an innocent desk lamp. You may think about model rigging, skeleton deformation, and keyframing..."
Watch the video for an illustration of what researchers at Microsoft have implemented.



Choreography by Inky

Inky's own creation, using cloud points and splicing different raw 3D movies into a long sequence. This was done in the Department of Computer Science, at Smith College, Nov. 2011. (This is also available on Vimeo: http://vimeo.com/32746935 )


Hand Detection

From YouTube: This is a graphical interface inspired by the movie "Minority Report". It uses the Kinect sensor from Microsoft, and the recently released libfreenect driver for interfacing with the Kinect in linux. The graphical interface and the hand detection software were written at MIT to interface with the open source robotics package 'ROS', developed by Willow Garage (willowgarage.com). The hand detection software showcases the abilities of the Point Cloud Library (PCL), a part of ROS that MIT has been helping to optimize. The hand detection software is able to distinguish hands and fingers in a cloud of more than 60,000 points at 30 frames per second, allowing natural, real time interaction. Code available at: http://www.ros.org/wiki/kinect http://www.ros.org/wiki/mit-ros-pkg

Work done by CSAIL's LIS Group (http://lis.csail.mit.edu/) and Robot Locomotion Group (http://groups.csail.mit.edu/locomotion/)


Dance in 3D

Cliff Biffle demos 3D point clouds with Kinect, filming his wife dancing. Cliff has also made his code available here: https://bitbucket.org/cbiffle/kinect-viewer


Kinect at Yaskawa Electric

The world's largest industrial robot manufacturer, Yaskawa Electric, wants to invade your grandmother's home... from engadget.com


Evoluce Kinect SDK

New SDK from Evoluce. Read more on engadget.com



NSquared and Integration of Mobile, Tablet and Kinect computing

NSquared Seamless Computing. Check out NSquared introduction on EnGadget; it integrates mobil applications, tablet computing, and Kinect.


Virtual Dressing Room

from engadget.com: "Don't be shy now: which of you doesn't love raiding your mother's closet and trying on her paisley dresses and velour tracksuits? That's more or less the idea behind Virtual Dressing Room, a Kinect program that taps into the clandestine thrill of sneaking into other people's boudoirs. Unlike some other shopping hacks we've seen, the app goes beyond just pilling on 2D pieces, but uses 3D models so that the items mold to your limbs, with the shadows and creases in the virtual fabric changing as you preen for the camera. That all comes courtesy of a special physics engine, while the app itself was written in C# along with Microsoft's XNA tools."


Minority-Report type of Interaction

from engadget.com: "In a brief video using the Kinect on Windows 7, he demos various ways of manipulating on-screen content with hand gestures and body tracking -- neglecting his mouse in the process. It's still a work in progress, but the results are already quite striking, so take that ancient input device and click past the break for the full demo."


Not Kinect, but related

from techcrunch.com, a very good video for inspiration on user interface and multi-displays.

Intelligent Shopping Cart

from Vimeo's page: wi-GO is a project that seeks to unite the technological knowledge and liability in the construction of a common path toward integration and improved quality of life. With the aim of building a barrier-free society, the project wi-GO is a robot system based on sensor technology Kinect that enables the disabled person, but also the elderly and pregnant women, among others, carry objects without difficulty, comfortably and safely. Scheduled to follow the people, promotes mobility, reduces the inability of people with physical disabilities, ensuring integration and quality of life.

3D Teleconferencing with Kinect

Kinect is used to track the position of the eyes of the person, and presents him/her a 3D view of his/her correspondent. See the whole article at engadget.com.

Just a game...

The original intent: Microsoft releases Kinect for the XBox 360...

3D holographic capture of a room

By combining the color and the depth image captured by the Microsoft Kinect, one can project the color image back out into space and create a "holographic" representation of the persons or objects that were captured.

Controlling Windows 7 with Kinect

Controlling Windows 7 using gestures... Even 2 people can interact at once.


The Eye of Sauron

The Lord of the Rings meets Kinect: The Eye of Sauron!

Kinect + Quadracopter

3D reconstruction

Autonomous flight and 3D terrain reconstruction.

Flight control

Controlling a Quadracopter with a Kinect.

Kinect + DaVinci = Fun Physics

Physics with Kinect: Razorfish ports DaVinci to Kinect.

Puppets and Artificial Life

Puppets/Artificial life with Kinect.

Kinect and 3D printing

Kinect3DPrinting.jpg

From http://www.engadget.com: As souvenirs go, a miniature replica of yourself -- striking a pose of your choosing on Barcelona's La Rambla street -- is a far sight more original than a bullfights-and-senoritas snow globe. This past January, the hilariously titled BlablabLAB enlisted three Kinects and a RepRap machine to snap passersby and render them into personalized tchotchkes, in a project called Be Your Own Souvenir.

Various Hacks

12BestKinectHacks.png

A collection of hacks (some of them not really in good taste), but interesting in possibilities... Taken from http://kinecthacks.net/.

Fight With a Shadow

From KinectHacks.net. Projecting your own shadow with a delay to create an illusion of a fight.

NewsBlobs

Colorful kinect-controlled visualisation of a RSS field.

Room Lights

Kinect turns on the lights in the room that are closest to your position.


Flight Controller

Using Kinect to simulate flying a plane over Manhattan.


Any Surface As A Multitouch Screen

from TechCrunch:"A new interface created jointly by Microsoft and the Carnegie Mellon Human Computer Interaction Institute allows for interfaces to be displayed on any surface, including notebooks, body parts, and tables. The UI is completely multitouch and the “shoulder-worn” system will locate the surface you’re working on in 3D space, ensuring the UI is always accessible. It uses a picoprojector and a 3D scanner similar to the Kinect."

(if you follow the link there is another (less technical) video with more examples of using different surfaces)


Controlling databases with Kinect gestures

from kinecthacks:"an experimental project that allows you to control SQL Server 2012 with gestures using Microsoft Kinect."


Kinect helping surgeons

Kinect allows surgeons to bring up and examine patient information without having to touch anything.


'Multiple Reality' Acro-Yoga

From the creator's website: "I created some software to merge multiple 3D images and video streams captured by the Kinect into a single 3D space. Objects from each video stream are superimposed as if they occupy the same physical space, with nearby objects from one video occluding more distant ones from another. Sometimes objects overlap, creating interesting mutant forms."


Kinect Used in a Talk by Polish Prime Minister

Polish Prime Minister uses the Kinect. kinecthacks.com



On the Artistic Side

Inky is on Fire

From CSC400-Kinect-Based Choreography: This is an example of current work in progress by In Kyung Lee and D. Thiebaut, in the CS Dept. at Smith College. The pixels from a still image of flames is superimposed on a Kinect point-cloud. More examples can be found here.

Moving Sculpture

From theCreatorsProject.com : By asking a dancer to physically interpret the song “Kreukeltape” by Machinenfabriek, and recording her movements on three Kinect cameras, the project creates a moving sculpture from the captured motion data. The three Kinect perspectives are then spliced together as point clouds to create the fluid abstraction in the video [below].

Ego Echo

Ego Echo is a responsive installation work. Inspired by classical trompe l'oeil perspective and audio samplers & loop pedals, it acts as a mirror, reflecting a room and the people in it and capturing and sequencing movement.


Mind the Dots

A live choreographic interactive AV performance from ExLex (under development)

Dancer: Anna Rubi
Visual concept: Matyas Kalman
Quartz Composer programming: Tamas Herceg, Matyas Kalman
Camera: Daniel Besnyo, Matyas Kalman
Editing: Daniel Besnyo

Mind the Dots 2

A live choreographic interactive AV performance from ExLex (under development)

A live choreographic interactive AV performance from ExLex (under development)

Dancer: Anna Rubi
Visual concept: Matyas Kalman
Quartz Composer programming: Tamas Herceg, Matyas Kalman
Camera: Daniel Besnyo, Matyas Kalman

Choreography by Inky

Inky's own creation, using cloud points and splicing different raw 3D movies into a long sequence. This was done in the Department of Computer Science, at Smith College, Nov. 2011. (This is also available on Vimeo: http://vimeo.com/32746935 )


Moulinex/Catalina

Moulinex/Catalina: Music video for Catalina, a track off the Chocolat EP (beatport.com/en-US/html/content/release/detail/317674/Chocolat%20EP) by Moullinex, released in January on Gomma Records.

Done with Kinect + Processing + Cinema 4D + After Effects.


Echo Lake -- Young Silence

Echo Lake -- Young Silence: The Echo Lake 12" 'Young Silence' is released on No Pain in Pop on February 14th 2011.


New Look/Nap on the Bow

New Look/Nap on the Bow: Making of cargocollective.com/timandjoe#2125444/Making-of-New-Look-Nap-On-The-Bow

Taken from the Album newlook-thealbum.com

Available on itunes: itunes.apple.com/gb/album/new-look-bonus-track-version/id465683098


Kinect Graffiti (TM)

Kinect Graffiti: Kinect Graffiti is a digital graffiti tool using "Microsoft Kinect" camera.

Idea behind this project is to use the kinect to track the motion behind graffiti. Visualizing the body and drawing trough different angles in realtime, Understanding surrounding space, pausing the time, etc...

Kinect Graffiti is a tool built in processing & openGL, SimpleOpenNI, openNI and primeSense libraries.

More stills are available at : flickr.com/photos/njc002/sets/72157626221800708/


Make the Line Dance

Kinect Graffiti: 2011 / Augmented dancing experimentation. While using a Kinect to track the human skeleton, we're mapping a video layer over a moving body.

Using : Kinect + QC + MadMapper + MaxForLive.


B-Seite festival 2011

"At the B-Seite festival 2011 we established advanced progressive workshops. Our task is to bring well skilled artists in different disciplines together. The workshops should lead to an installation or performance."

[...] Final performance realized with vvvv, Kinect & Triplehead2Go ...

THE KINECT DANCER @ KUBUS, Zeitraumexit, Mannheim.



F&N Kinect Singapore Dance Delight

F&N Kinect Singapore Dance Delight: Dancers are transformed into colourful bubbles which react and replicate the dancers' moves. As dancers groove to the music, sounds will be triggered. Depending on which screen region the dancer move to and trigger, different pitches will be produced, leading to a unique mish-mash of creative sounds and movements. A first of first in many ways, the result is an unforgettable experience for the dancers and the audience.



21385114

Versus - First Teaser

Versus - First Teaser: Stereoscopic Realtime Dance Performance by 1n0ut, with Nanina Kotlowski

http://1n0ut.com

Experimental performance with Kinect and Adaptive Learning Algorithms





Programming the Kinect

Skeleton Tracking

--Thiebaut 18:24, 27 March 2012 (EDT)

See article in PCWorld] on SkelTrack.






ProcessingLogo.jpg

Kinect and Processing

  • Clay Shirky]'s Blog is a good place to start.
  • It refers to Daniel Shiffman's good intro page on the subject.
  • A nice collection of Processing] programs to do all kinds of image manipulations with Kinect.
  • To generate movies with MovieMaker, this chart from processing.org/discourse generated by PhiLho shows good combinations of parameters:
    • Tested with Apple QuickTime 7 for Windows (XP SP2)
      • ANIMATION: perfect - 1,800KB
      • BASE: empty! - 0
      • BMP: perfect, enormous! - 121,500KB
      • CINEPAK: Lot of artifacts on background, trail - 1,600KB
      • COMPONENT: Little artifacts on background, big! - 81,000KB
      • CMYK: empty! - 0
      • GIF: empty! - 0
      • GRAPHICS: Dithering - 16,600KB
      • JPEG: Good quality, reasonable size - 2,500KB
      • MS_VIDEO: empty! - 0
      • MOTION_JPEG_A: Good quality, reasonable size - 3,200KB
      • MOTION_JPEG_B: Good quality, reasonable size - 3,200KB
      • RAW: perfect, enormous! - 162,000KB
      • SORENSON: Little artifacts on background, small - 780KB
      • VIDEO: Small palette (blocky gradient), trails - 470KB
      • H261: Lot of artifacts on background - 460KB
      • H263: Lot of artifacts on background - 330KB
      • H264: Sketch fails (exception quicktime.std.StdQTException[QTJava:7.4.5g],-8960=codecErr,QT.vers:7458000) after writing 120KB

Deciding on a API between the kinect and the PC

Before the Microsoft SDK was released on June 16 for developers, there were many open source APIs available as well the most popular one being Openni/NITE( as well as OpenCV and CLNUI) which is still popular for developing commercial apps with kinect which the official SDK is yet to offer. To use Openni/NITE with other appplications or developing new applications these steps posted on i-programmer would help one get started: [3] .

Some really cool tools that one may want to use along with it is FAAST( great to use with any virtual reality app): [4] and Brekel( which helps to track the user's skeletal motion and cast it into a motion builder):[5]. The only drawback lies in the fact that it needs pose tracking with a custom gesture before one can start using it.

The Microsoft SDK works only on Windows 7 computers and is available for download here:[ http://research.microsoft.com/en-us/um/redmond/projects/kinectsdk/]

Getting data streams out of the kinect: Video, Depth and Skeleton

Here let us basically concentrate on the Microsoft SDK which gives access to all these three kinds of data from the kinect sensor. One needs to download Microsoft Visual C# Studio, Coding4fun Kinect Toolkit as well as Silverlight. The basics of getting the data from the Kinect involves three simple steps :

  • Create a Runtime object for the NUI(Natural User Interface)
  • Sign up for the event to get the Skeletal/Depth/Video frame ready.
  • Open the stream so that the event will fire.

The article and video that help to get one started to access the video frame are the following:

  • Getting started with microsoft kinect sdk :[6]
  • Getting started with kinect windows 7: [7]

The problem with accessing the depth frame lies in understanding the fact that the depth data cannot be accessed in the same way as the image as the the image I am getting back is 16 bits and it needs to be converted to 32 bits. Here is a nice way of shifting bits to access the depth frame in the section accessing the depthframe: [8]

This is a good app( along with source code) to get one started with accessing the skeletal frame: [9]

  • Some of the simplest sample applications which uses the data streams to get one started:

Kinect Paint [10]: Kinect Paint is a skeleton tracking application that allows you to become the paint brush! This application uses the Kinect for Windows SDK and its skeletal tracking feature to create a painting experience. Use several different tools, a simple color palette, and create a masterpiece!

Kinect Earth [11]: KinectEarthMove, which use color image and skeleton from Kinect, needs Kinect for Windows SDK beta and was demonstrated on Kinect for Windows SDK beta launch at Channel 9 on June 17 2011.

Becoming the Incredible Hulk [12]: Replces three joints with interesting features!

Developing a new application

This is a student summer research page which talks about Kinect Background Removal and adding an External Video Backdrop mostly in the later posts : Bismita Sahu Summer 2011