tui

Concepts

The whole workshop rests on one idea: three coordinate spaces, connected by two homographies.

            corner pin (lesson 02)                camera calibration (lesson 04)
 TABLE mm  ───────────────────────►  PROJECTOR px        CAMERA px  ───────────────────►  TABLE mm
 (where you                          (what the           (what the                       (back where
  draw)                               projector shows)    webcam sees)                    you draw)

1. Three spaces

Space	Units	Origin	Who lives here
Table	millimetres	table’s top-left corner	your sketch, touches
Projector	pixels (e.g. 1920×1080)	top-left of the projector image	the browser window
Camera	pixels (e.g. 1280×720)	top-left of the webcam frame	tracking

Your code only ever thinks in table space. The projector and camera are adapters at the edges.

2. Homography

When a pinhole device (projector or camera) looks at a flat surface, the mapping between surface coordinates and device pixels is a homography: a 3×3 matrix applied in homogeneous coordinates:

[x']   [h0 h1 h2] [x]
[y'] = [h3 h4 h5] [y]      →   (x'/w, y'/w)
[w ]   [h6 h7 h8] [1]

• The bottom row is what lets parallel lines converge: perspective.
• Scale doesn’t matter, so fix h8 = 1 → 8 unknowns.
• Each point pair gives 2 equations → 4 points pin it down exactly.

That’s why every calibration step is “four corners”. lib/homography.js builds the 8×8 system and solves it with Gaussian elimination.

A homography maps straight lines to straight lines. It cannot correct lens distortion (barrel/fisheye bending) or non-flat surfaces. Those need more advanced calibration (see Going further).

Why it works in the browser

CSS transform: matrix3d(...) is a 4×4 projective matrix. Our 3×3 homography slots into it (see toCSSMatrix3d), so the GPU warps the canvas for free, at full frame rate, with no pixel loops.

3. Why not the projector’s keystone setting?

Built-in keystone correction usually handles only vertical tilt (sometimes horizontal), not arbitrary 4-corner placement. It also doesn’t tell your software where the table is. Software corner-pinning gives you exact millimetres and a mapping you can invert.

4. Sensing: from pixels to touches

1. Shrink the camera frame (160 px wide): faster, and averaging removes noise.
2. Segment: decide per pixel “is this a thing?”

   • Background difference:

     frame − empty table

     > threshold

   • Colour key: hue within tolerance, saturated enough, bright enough
3. Blobs: group touching pixels (connected-component labelling / flood fill), drop tiny ones.
4. Locate: blob centroid (camera px) → camera→table homography → table mm.

5. The feedback problem

The camera sees the projector’s light. Anything you project changes the camera image, and the tracker can mistake it for an object. A projected ripple can trigger a new ripple, which triggers another…

Approach	Cost	How it helps
Dim, sparse graphics + higher threshold	free	less projected change
Colour tracking	free	track a colour you never project
Re-capture background when scene is static	free	projected scene becomes “background”
Infrared camera + IR-pass filter + IR illuminator	~¥3–10k	camera can’t see projector light at all
Depth camera	~¥30–60k	measures height, not colour: true touch vs hover
Sync camera to projector blanking	hard	look only when the projector is dark

This is a great discussion point. Every interactive projection system makes a choice here.

6. Going further

• Automatic calibration: project dots / Gray-code stripes (structured light) and find them in the camera, with no clicking.
• ArUco / AprilTag markers: printable fiducials that give an object’s ID, position and rotation.
• Lens distortion: calibrate the camera with a checkerboard (OpenCV calibrateCamera) before the homography.
• Non-flat surfaces: mesh warping, or projector-camera calibration to a 3D model.
• Multiple projectors: overlap regions and edge-blending for bigger tables.
• Hand tracking with ML: MediaPipe Hands runs in the browser and finds fingertips directly.