Last few days I add a few more nice features to Gibbon. I spend hours trying to adjust and tweak the parameters for optical flow tracking to get a more consistent and stable result. I also tried four or five different types of diffusers. One big problem was that features where unstable and it would make it very difficult to discriminate for type of hand motions. I finally ended up applying minEigenValue filter to the image before feeding it into cv::goodFeaturesToTrack algorithm. This is a bit puzzling to me since the goodFeatureToTrack is using cv::cornerMinEigenVal() function internaly, but I am not going to worry about that right now since it works. Features are relatively stable now and the Lucas-Kanade method can track the features quiet reliably. Here is a frame capture that shows features and some new visualizations.
Enhanced feature tracking with depth and orientation
The hollow red circle is larger when feature is detected far away. As finger or detected feature gets closer to screen the size of circle decreases. When measure of closeness which is based on image sharpness passes a predefined threshold the feature is considered touching. Touch points are shown with filled red circles of fixed size. Becase the hole hand is also tracked, the approximate orientation of fingers can be obtained by computing a vector from the center of the hand to the position of features. These vectors are drawn with bright green lines from the center location of features.
Finger orientation and tilt
In the above figure user is touching the screen with one finger. They can now change the aim of direction of their finger by turning their hand around the touch point. Furthermore, the relative tilt of the finger (the angle between finger and the touch surface) is visualized using the length of the green line.
I think Gibbon is getting to the point where it actually might be usable :). I am going to try even harder from now on to documents changes to it. It is very exciting when you think about all the interaction design opportunities that become possible when we have access to hand tracking, finger trajectories, orientation and tilt!
