Tracky Mouse
Hands-free mouse control
Hands-free mouse control
Tracky Mouse lets you control your computer with just a webcam.
Precisely control the mouse cursor by moving your head.
Click by hovering over a control for a set amount of time. Cancel by moving away.
Regain control of the mouse by simply moving it. Head tracking will pause temporarily.
Adjust the sensitivity and smoothing of head tracking.
Tracky Mouse combines state-of-the-art face detection with tried-and-true optical flow tracking to precisely control mouse movement with your head.
Enable Viacam
uses optical flow to track the movement of points on your face,
but it has trouble finding your face in the first place.
Project Gameface
uses state-of-the art machine learning model FaceMesh to find your face,
but cursor movement is very jittery because it uses the face position information directly.
Tracky Mouse works like Enable Viacam, tracking points on your face precisely with optical flow, but it uses state-of-the-art face detection to place those points.
Since the face detection is so good, it can place the points not just on your face, but at locations on your face that are ideal for tracking, and it can prune away the points when they stray from those specific locations on your face.
The result is a system that's more accurate than either.
People who have limited mobility, due to motor neuron diseases like ALS, or other conditions that make it difficult to use a mouse, can greatly benefit from hands-free computer control.
I hope to make it good enough that people want to use it even for common repetitive strain injuries like carpal tunnel syndrome.
It's also just fun! It feels like controlling the computer with your mind.
Ever wanted to use your computer while playing guitar, or while eating a sandwich? Maybe Tracky Mouse is for you!
You can try out Tracky Mouse right here in your browser.
Just click to allow camera access, and then adjust the settings until it feels comfortable pointing to any part of the screen.
If you scroll down, there is an area where you can practice pointing at targets.
A demo of Tracky Mouse should appear here.
Install the desktop app to control your whole computer.
View Releases - none yet
Most of the functionality of Tracky Mouse is available as a JavaScript library.
You can use it to add head tracking and dwell clicking to your own web application.
Check out the API documentation for more information.
JS Paint uses Tracky Mouse for dwell clicking in its Eye Gaze Mode.
This is a head tracking system similar to eViacam, using LucasāKanade optical flow to track points for high accuracy, and face detection for understanding of where to place tracking points. It uses two different face detectors, a fast-to-load one (clmtrackr.js) and a slower to load but much more accurate one (FaceMesh), which it switches to automatically when available.
JSFeat is used for optical flow, and clmtrackr.js also uses JSFeat, so I patched clmtrackr.js to export JSFeat so it doesn't need to be loaded twice.
clmtrackr.js used eval heavily, but I created
a tool that eliminates the need for eval
by running the code and seeing what it tries to evaluate,
and creating functions that do the same thing.
This allows Tracky Mouse to run in a context secured with Content-Security-Policy.
Open source on GitHub. MIT-licensed.
All processing is done locally on your computer.
Tracky Mouse does not store any personally identifiable information.
Tracky Mouse may use a third-party service to report errors, including some context about how you've configured the software.
Please email me with any feedback or questions, or open an issue on GitHub.