Kinect

The NETLab Toolkit can take input from the Microsoft Kinect system, making it very easy to track a person’s position and gestures. For example, it’s possible to use the left hand’s position to control the volume of a sound as the person moves their hand up and down.

Special thanks go to Nathan Cornes in Brazil for helping to figure out how to do this, creating the OSCulator patch, and making suggestions for improvements to the toolkit!

More info after the video

What You Need

To work with the Kinect, here’s what you need for the Mac (see notes below for the PC):

  • Kinect Sensor – with an external power supply (third party versions can be found much cheaper on Amazon and eBay). You DO NOT need an XBox.
  • Synapse software – talks to the Kinect, converts its information into the OSC protocol, needs OSCulator to translate some things, requires calibration, free
    • OSCulator – which helps process the the OSC messages from Synapse, combines them, and sends them to the NETLab Toolkit as a set of x,y,z points for all 15 joints
    • sendskeleton.oscd patch for OSCulator – configures OSCulator to process the OSC from Synapse, and tells Synapse to send the joint positions
    • kinect_skeleton_demo_synapse_hub_2.fla (hub_1.x) – example Flash project that demonstrates how to work with Kinect via Synapse
  • NI MATE software– talks to the Kinect, communicates directly with the NETLab Toolkit, does not require calibration, beta free now, but will likely be paid
  • Widgets version 3.3.5 minimum – installed for your current version of Flash

Quick Start

  1. Download and install the above software
  2. Attach the Kinect
    1. Connect the Kinect power supply to an A/C outlet
    2. Connect the Kinect sensor to the power supply
    3. Connect the power supply USB cable to your computer
  • Synapse
    • Open the sendskeleton.oscd file in OSCulator
    • Start the Synapse application – you should see a blue representation of the Kinect’s camera view
    • Tell Kinect to recognize your body – To properly interpret your body motions, Kinect needs to be calibrated to you. To do this, stand in front of the Kinect, and put your arms out, with your forearms pointed up as seen below
    • Open the kinect_skeleton_demo_synapse.fla and test the movie in Flash – you should see the skeleton in Flash
  • NI MATE
    • Open NI MATE
    • Click on the OSC tab, then click on the Skeleton sub-tab. Next to the Format label, set it to OMNI, which will send all skeleton joints
    • Stand in front of the Kinect, no calibration needed
    • Open the kinect_skeleton_demo_nimate.fla and test the movie in Flash – you should see the skeleton in Flash

 

 

 

Detailed Notes

The combination of Synapse and the sendskeleton.oscd OSCulator patch, or NI MATE alone sends all of the 15 joint coordinates (x,y,z) in a single OSC message to the Toolkit. Set your AnalogIn or DigitalIn to these parameter settings:

  • controller: osc
  • controllerInputNum: 0-44 depending on which coordinate of which joint you want, i.e. 15 joints * 3 (see below for specifics)
  • controllerPort: 2202 for Synapse, 7000 for NI MATE
  • oscString: /skeleton
    • an alternative with NI-Mate is to use /Head, /Body, /Hand_Left, or /Hand_Left – other options are available, see the OSC tab in NI-Mate. Using these settings gives you data even if the Kinect doesn’t find a skeleton – so if you only need the person’s position, then use one of these, since the Kinect will acquire this data more easily and quickly.

There are two ways you can access the joint coordinates. The kinect_skeleton_demo.fla file shows both of these methods.

  1. Use controllerInputNum to get a single value – In this method, you choose a number that corresponds to the single joint and x,y or z value of that joint. That single joint value is then available to use in the normal way by any widget that is set to listen to the input widget. For example, if you wanted to use the X position of the person’s left hand, you would input 9 for Synapse, 15 for NI-Mate into the controllerInputNum parameter of the widget.
  2. Use code to access all 45 values for all the joints – If you need access to many of the joint values, it is better to use code to get at an array of values that can then be used in any way you choose. In the demo, the code creates 15 circles, and then sets their x,y position and size based on the x,y,z values respectively for each joint.
Kinect software documentation

Joint values from NI-Mate Omni Mode, Synapse with OSCulator patch sendskeleton.oscd

X,Y,Z NI-Mate Synapse/sendskeleton.oscd
0 – 0,1,2 head head
1 – 3,4,5 neck left elbow
2 – 6,7,8 body/torso left foot
3 – 9,10,11 left shoulder left hand
4 – 12,13,14 left elbow left hip
5 – 15,16,17 left hand left knee
6 – 18,19,20 right shoulder left shoulder
7 – 21,22,23 right elbow neck
8 – 24,25,26 right hand right elbow
9 – 27,28,29 left hip right foot
10 – 30,31,32 left knee right hand
11 – 33,34,35 left foot right hip
12 – 36,37,38 right hip right knee
13 – 39,40,41 right knee right shoulder
14 – 42,43,44 right foot torso

 

Generic Code to access multiple values from AnalogIn via OSC or hubFeed

//
// OSC can send multiple values to AnalogIn, and the message would look something like this
// /skeleton -0.13460922 0.06446591 0.6056481 
// In the case of Kinect, there are 45 numbers passed in, 3 each (X,Y,Z) for 15 joints

input0.multiInput = processInput; // process multiple values from AnalogIn with an instance name of input0

// function to process multiple values
function processInput(... args) { // args is an array that will contain all the values
   for(var i:int=0; i<args.length; i++) { // get each value in order
      trace(args[i]);
   }
}

SYNAPSE CODE

var list:Array = new Array();
var totaljoints:int = 15;

// Build all the joint graphics
for(var n:int=0; n<totaljoints; n++) {
   // create the graphic
   var newJoint:Sprite = new Sprite();
   newJoint.graphics.lineStyle(0);
   newJoint.graphics.beginFill(0x000000, 0.5);
   newJoint.graphics.drawCircle(0,0,10);
   newJoint.graphics.endFill();
   // add it to the screen
   addChild(newJoint);
   // put it in a list
   list.push(newJoint);
}

input0.multiInput = processInput; // process multiple values from AnalogIn or DigitalIn with instance name of input0

// function to process Kinect data
function processInput(... args) { // args is an array that will contain all the joint values
   var x:int;
   var y:int;
   var z:int;

   // each joint has three values - x,y,z
   // cycle through all joints to get their x,y,z position
   for(var i:int=0; i<totaljoints; i++) {
      // args contains x,y,x values for 15 joints
      // so each set of values starts at joint * 3 into the set of arguments
      x = i * 3;
      y = x+1;
      z = x+2;
      list[i].x = args[x];
      list[i].y = args[y];
      list[i].scaleX = list[i].scaleY = 3-(args[z]/1500);
   }
}

NI MATE CODE

// display joints from Kinect

var list:Array = new Array();
var totaljoints:int = 15;

// Build all the joint graphics
for(var n:int=0; n<totaljoints; n++) {
   // create the graphic
   var newJoint:Sprite = new Sprite();
   newJoint.graphics.lineStyle(0);
   newJoint.graphics.beginFill(0x000000, 0.5);
   newJoint.graphics.drawCircle(0,0,10);
   newJoint.graphics.endFill();
   // add it to the screen
   addChild(newJoint);
   // put it in a list
   list.push(newJoint);
}

input0.multiInput = processInput; // process multiple values from AnalogIn or DigitalIn with instance name of input0

// function to process Kinect data
function processInput(... args) { // args is an array that will contain all the joint values
   var x:int;
   var y:int;
   var z:int;
   var theScale:Number;

   // each joint has three values - x,y,z
   // cycle through all joints to get their x,y,z position
   for(var i:int=0; i<totaljoints; i++) {
      // args contains x,y,x values for 15 joints
      // so each set of values starts at joint * 3 into the set of arguments
      x = i * 3;
      y = x+1;
      z = x+2;
      list[i].x = ((args[x] + 1) * 400) + 275;
      list[i].y = 250 - ((args[y] + 1) * 400);
      theScale = 1 / (args[z] / 1.5);
      if (theScale > 3) theScale = 1;
      list[i].scaleX = list[i].scaleY = theScale;

   }
}

Last modified November 15th, 2012