← Home
Side quest · Computer vision

Maestro

Conduct a video with your hands. A webcam watches for one hand, and MediaPipe turns it into 21 tracked points; clenching a fist pauses playback, and flicking up or down rides the volume. No controller, no keyboard.

The interesting part is not detecting a hand: that is a solved problem you can import. It is deciding which movements were meant as commands. Hands drift, wrists rotate, and a hand returning to a resting position looks a lot like the opposite gesture. Most of the logic below exists to throw those away.

Try it

Needs a webcam and one hand free. The camera feed is processed frame-by-frame in your browser and never sent anywhere. There is no server behind this page. The hand-tracking model (~2 MB) downloads when you press start, not on page load.

No hand detected

Camera preview appears here once you start.

Volume
50%
Gesture sensitivity

Press start and allow camera access when your browser asks.

  • Clench a fist to pause
  • Open your palm to resume
  • Flick up or down for volume

This demo wants a desktop webcam and a free hand, since on a phone you would be holding the camera you are gesturing at. The write-up below works anywhere, and the source is on GitHub.

Making a hand mean something

Three problems stood between raw landmarks and controls that felt deliberate rather than twitchy.

A fist is a ratio, not a distance

The obvious way to detect a clench is to measure how far the fingertips sit from the wrist. That works until you lean back, and then every distance shrinks and your open palm reads as a fist. Openness here is the mean fingertip-to-wrist distancedivided by hand size, meaning the wrist to middle-knuckle span. Both scale together with distance from the lens, so the ratio holds whether you are close to the screen or across the room.

The pause and resume thresholds are deliberately far apart (0.85 and 1.40). A single threshold would flicker between states every time a finger twitched near the boundary; the gap means you have to genuinely open your hand to undo a genuine clench.

Volume rides on velocity, not height

Mapping volume to how high your hand is means your hand is always issuing a command, so there is nowhere to rest it and you run out of vertical space. Instead volume responds to movement: wrist velocity smoothed over six frames, and only past a threshold. Hold your hand still anywhere and nothing happens, so you can reposition freely between gestures.

Frames where the wrist jumps more than 10% of the frame height are discarded outright. That is not a hand moving fast; it is a hand leaving the frame and a different one entering, and averaging across that discontinuity produces a violent phantom flick.

The return sweep problem

Flick up to raise the volume and your hand has to come back down, which is the gesture for lowering it. Early builds cancelled out almost every adjustment this way.

The fix is direction locking: once a direction is established, reversing it requires a flick 2.5× stronger than starting one. A relaxed return sweep is not intentional enough to qualify. Holding still for eight frames releases the lock. On top of that, volume is suppressed entirely for 600 ms after any pause or resume, and for the first 20 frames after a hand appears. Both are moments when the hand is moving for reasons that have nothing to do with volume.