If you want to be in a band, but don't have an instrument and your friends are 500 miles away, use Tokbox Ukulele. We allow users to play an "air instrument" with friends via webcam. The application tracks hand motions using a webcam to determine which chord is being played. Once the chord is determined, it's sent over the network to all participants, allowing for collaboration with your 'virtual band'.
We use the Tokbox API for video capture and streaming, then draw the video to an html5 canvas for processing. Applying the Lucas-Kanade optical flow method and some color normalization, we track finger and marker positions. Normalizing these finger positions, we can classify to the nearest chord and play it to all participants whenever we detect a 'strum'.
Put some distinctive markers on a ukulele or other object in approximately the pattern shown below. Then launch the app, click (in this order!) the marker on the 'head' of the 'instrument', the one on the 'body', then the index finger, middle finger, and ring finger. Finally, click the marker you are using to strum the 'instrument'. Then, strum away!
All users in the 'multiplayer' mode will join the same room, but 'singleplayer' mode gives some more diagnostics and details.
