FaceTrackNoIR PointTracker Plugin

+ + +

About

+PointTracker is a plugin for the free head tracking software FaceTrackNoIR +which introduces the capability to track a (typically IR-) point model comprising 3 bright points to FaceTrackNoIR, +much like the popular free tracking software Freetrack does.
+It was created as a stable modular alternative to Freetrack, which has some stability issues with newer systems and seems to be no longer actively developped. +

+ + +

Settings

+This section desribes the various settings of the PointTracker plugin in detail. +

+ +

Show VideoWidget: Whether the video widget is updated or not. It may save some performance to turn this off when not needed
Sleep time: Time the tracking thread sleeps after each processed image. It's inverse should be below the framefrate you want to achieve. +(check the framerate in the status region when tracker is active, in case the sleep time is too high, the framerate will decrease). +Low values will result in more CPU-load.
Dynamic Pose Resolution: Whether the point correspondence and pose ambiquity is resolved using a more sophisticated dynamic algorithm (constant velocity prediction) or a simple static resolution. +Dynamic pose resolution can capture more extreme poses but may occasionally get stuck in a wrong pose estimates so that a reset of the internal state becomes neccessary.
Auto-reset time: If no valid tracking result can be found when using dynamic pose resolution, the tracker will automatically reset its internal state (used for resolving the pose ambiguity and point correspondence) +and return to a fail-safe initialization phase that assumes a neutral pose after this time. +Decrease this time, if you get stuck in a wrong pose too often.
Reset: Manually reset the trackers internal state used for dynamic pose resolution and return to a fail-safe initialization phase that assumes a neutral pose. +You may use this in case you get stuck in a wrong pose.
Enable Axis ...: Which axis to use for FTNoIR.

+ +

Device: The camera used for tracking.
Resolution: The desired capture resolution. If your camera does not support the entered resolution the true output resolution may be different or even invalid. +You may check the true capture resolution in the status area while the tracker is running. A higher resolution results in more accurate point positions and will increase the +stability of the tracking result, as long as the signal/noise ratio is sufficiently high.
FPS: The desired capture framerate. Again, if your camera does not support the entered framerate, the true caputre framerate may be different or invalid. +You may check the true processing framerate in the status area while the tracker is running.
F/W: The focal length of the camera divided by the sensor width (of course in the same units). +In case you don't have access to your camera's specifications, you can measure this yourself by placing a plane object of known width (for example a piece of cardboard) in front of the camera until it fills the whole image width. +Then measure the distance between the object and the camera and divide by the object width.
VideoWidget: Shows a resizable stand-alone video widget that shows the same content as the integrated video widget in FTNoIR. +Update rate is only 10 fps and may lag behind a bit. Mainly useful during calibration of the point extraction. Same as for the integrated wiget, to save resources, this widget should only be shown when needed.
Roll Pitch Yaw...: The orientation of the camera relative to the reference frame. +If these angles are setup properly, the direction of translations may not be correct. +Roll is treated in a special way since it is implemented as a frame rotation by +/- 90 deg that is transparent to the rest of the processing pipeline. +
Threshold: The threshold for point recognition. Areas above the threshold are shown in blue in the VideoWidget. +Since point accuracy is best if the points are as big as possible in pixels, the theshold should be chosen as low as possible (stop before the contour of the points becomes "noisy"). +If small reflections are being falsely classified as points, increasing the minimum point diameter (see below) may help.
Min Diameter: Minimum diameter of blobs to be classified as a pointmodel-point.
Max Diameter: Maximum diameter of blobs to be classified as a pointmodel-point.
Status: The tracker's status is shown in this area while the tracker is running. +The FPS shown here correspond to the framerate of the whole tracker processing chain and may be lower than what your camera is able to provide, when
+1. The processing gets not enough CPU time
+2. The sleep time of the tracking thread is set too high

+ +

Model Selection and Dimensions ...: +First select your model type (point, clip, custom), then enter the dimensions of your model in milimeters here.
+For the custom setting, the coordinates of the two remaining model points have to be entered (reference point M0 is at (0,0,0)) in a pose where the model roughly faces the camera. +For orientation, the coordinates for the standard Freetrack clip are (0,40,-30), (0,-70,-80) and the ones for the cap (40,-60,-100), (-40,-60,-100).
+When using a custom point-model configuration, the following restrictions should be observed:
+The plane in which the 3 points lie should never be parallel to the image plane, M0-M1 and M0-M2 should be roughly perpendicular.
Model Position: The vector from the model to the center of the head in the model frame. Can be calibrated automatically.
Calibrate: In order to automatically calibrate the model-head offset, do the following:
Press the Calibrate button, then look around while not moving your shoulder. (i.e. only rotation, no translation). +Do not stay in one pose for too long. The current translation estimate will be updated in real time. As soon as the values stabilized sufficiently, press the Calibrate button again to stop the calibration process.

+ + +

Filter Setup

+This section desribes how the FTNoIR filter work and what the recommended settings for PointTracker are. +

+Filtering is always a tradeoff between stability, accuracy and responsiveness. +

+The Smoothing filter in FTNoIR is just a simple average over the last n samples. +Since this filter produces input lag no matter how fast the head-movements are, it is recommended to turn it off by setting samples to 1. +

+In the filter tab, it is recommended to select Accela Filter Mk2. +Accela is a non-linear filter that works as follows:
+It looks at the difference between the new raw values new_val from the tracker and the last filtered value old_val +and maps this difference via the customizable response function f via:
+

+new_val = old_val + f(new_val - old_val) / reduction_factor +

+So by setting f(x) = reduction_factor * x, one will get no filtering at all.
+If you set lower values for small x, small deviations (usually noise) will get dampened. +This results in a dynamic dead-zone around the current position. +

+The last two points are used by accela to extrapolate for large deviations. +So in order to get a fast unfiltered response for large deviations, the line connecting the last two points should have a slope >= reduction_factor. +

+More aggressive accela settings than the default FTNoIR accela settings are recommended in order to decrease the filtering lag and fully use the potential of point tracking.
+My current settings are: +


+[Accela]
+Reduction=20
+
+[Curves-Accela-Scaling-Rotation]
+point-count=4
+point-0-x=0.1
+point-0-y=0
+point-1-x=1.43
+point-1-y=2.45
+point-2-x=2.0
+point-2-y=5.44
+point-3-x=2.06
+point-3-y=6
+

+The curve is not too different from the standard one (except that I like a small dynamic dead zone for steady aiming, that's why the curve has a slope of 0 at the beginning).
+However, the reduction factor is decreased to a value of 20 (compared to the standard value of 100). This implies that each value of the curve is effectively 5 times higher than in standard FTNoIR (see formula above), which means higher responsiveness but can also lead to jitter/shaking.
+Keep in mind that there are no best filter settings. Since filtering is always a compromise it's a matter of personal taste and +playing around with the filter settings is highly recommended. +

+ + +

Support

+For questions/feedback about the plugin, post to the FTNoIR-Forum.
+In case you like this plugin and would like to support the author, you may consider making a donation. +

+ +

+ + +

ChangeLog

1.1

Added camera yaw and roll correction (intended for vertically mounted cameras)
Improved point extraction algorithm, thanks to Michael Welter
UI improvements: Select camera by device name, different VideoWidget architecture
Bugfixes: Removed 99 FPS limitation

+ +

1.0

Added camera pitch correction
Better communication with FTNoIR: output axis configuration, status report

+ +

1.0 beta

Switchted to videoInput library for capture. Desired capture resolution and fps can now be customized
Introduced dynamic point-correspondence and POSIT-ambiguity resolution, which allows for the reconstruction of more extreme poses
More convenient freetrack-like model dimension GUI
Bugfixes: VideoWidget skipping frames, Timer resolution too low for accurate FPS measurement

+ + +

Build Instructions

+This section describes what you need to do in order to build PointTracker yourself.
+You can find the sources at the project site +or as part of the FTNoIR sources. +

The project was created with Visual Studio.

+ +

Dependencies

Qt 4.8.2 library
Qt plugin for Visual studio
OpenCV 2.4 prebuilt for Windows
Boost 1.47

+ +

Details

Common

setup environment variable "QTDIR" (example value "D:\Devel\Libs\Qt\4.8.2")
add "%QTDIR%\bin" to PATH
setup environment variable "BOOST_DIR" (example value "D:\Devel\Libs\boost_1_47_0")
setup environment variable "OPENCV_DIR" (example value "D:\Devel\Libs\opencv\build")

Debug

opencv linked dynamically:

add "%OPENCV_DIR%\x86\vc9\bin" to PATH

(in case of different Visual studio, change PATH and linker dependencies accordingly)

Release

opencv linked statically:

custom build a statically linked version of opencv with the buil-option BUILD_WITH_STATIC_CRT set to OFF!
copy resulting libaries to "%OPENCV_DIR%\x86\vc9\static_lib"

(in case of different Visual studio, change PATH and linker dependencies accordingly)

+ +