Landmark Transform Lab

Three live computer-vision demos, built from scratch with plain matrix and pixel math: a face with a rotation/translation matrix and blendshapes, a full 33-point body skeleton you can pose and rotate, and a synthetic camera feed showing how a segmentation model separates a person from the background.

Where this shows up in practice: MediaPipe's Face Landmarker returns 478 3D landmarks, a 4×4 facial transformation matrix, and up to 52 blendshape scores. Its Pose Landmarker returns 33 3D body landmarks plus a visibility score for each. Its Selfie Segmentation model returns something different in kind — one probability per pixel, "how likely is this pixel part of a person" — with no landmarks or joints involved. The three tabs below are hand-built, simplified stand-ins for those three real outputs.
drag to orbit the camera
face mesh (wireframe)
feature landmarks
head's local X axis
head's local Y axis
head's local Z axis

Head pose

Rotation (pitch / yaw / roll) and translation, combined into one 4×4 matrix.

Pitch (nod)0°
Yaw (turn)0°
Roll (tilt)0°
Translate X0.00
Translate Y0.00
Translate Z0.00
this is what actually gets applied — p' = R · p + T

Blendshapes

Each slider is a weight on a preset facial deformation, blended on top of the neutral mesh.

Eyebrows raised0.00
Eyes closed0.00
Jaw open0.00
Smile0.00
Mouth pucker0.00
face = neutral + Σ (weighti × Δshapei) no blendshape is active — the mesh is neutral

Rotation: turning without moving

Pitch, yaw and roll each rotate around one axis of the head — X, Y and Z. A rotation matrix R is what you get when you multiply those three simple rotations together in a chosen order. R never changes the size or shape of the face, only its orientation: its rows are perpendicular unit vectors, so Rᵗ = R⁻¹.

Order matters. Rotating yaw-then-roll lands you somewhere different than roll-then-yaw, because each rotation happens in the frame left behind by the one before it.

Try this: set yaw to 45°, watch the local axes in the viewport, then bring roll up too. The blue Z axis is no longer pointing where it started — that's the order-dependence, not a bug.

Translation & the 4×4 matrix

Translation just adds an offset vector T to every point: p' = R·p + T. To do rotation and translation in a single matrix multiply, computer vision pads everything into homogeneous coordinates — a 4th "1" appended to every point — so R and T live together in one 4×4 matrix, shown live in the panel on the left.

This is exactly the shape of matrix a head-pose or camera-pose model hands back: the top-left 3×3 block is orientation, the last column is position.

Try this: push Translate Z while yaw is nonzero. The head moves along a straight line in world space, but because it's rotated, that line no longer looks aligned with the face's own "forward."

Blendshapes: expression as a mix

A blendshape is a fixed deformation — "jaw open," "smile," "left eye closed" — stored as an offset from a neutral mesh. A live expression is just the neutral face plus each active blendshape scaled by a weight, usually 0 to 1.

This is additive and independent of head pose: the same blendshape weights produce a smile whether the head is facing forward or turned 40° away, because rotation acts on the whole mesh afterward.

Try this: set smile and jaw open together, then yaw the head. The expression stays intact through the rotation — pose and expression are separate numbers describing separate things.
drag to orbit the camera
skeleton (bone connections)
33 body landmarks
body's local X axis
body's local Y axis
body's local Z axis

Body pose (root transform)

The same rotation + translation matrix as the head, applied to the whole skeleton at the hip.

Pitch0°
Yaw0°
Roll0°
Translate X0.00
Translate Y0.00
Translate Z0.00
applied at the hip (landmark 23/24 midpoint) — p' = R · p + T

Joint angles

Each limb is a small kinematic chain — every joint's rotation multiplies onto the one above it.

Arms raise0°
Arms spread0°
Elbow bend0°
Torso lean0°
Squat (hips + knees)0°
Leg spread0°
wrist rotation = shoulder_R × elbow_R no joint angle is active — standing neutral

33 landmarks, one skeleton

MediaPipe's Pose Landmarker numbers its 33 points by region: 0–10 trace the face (nose, eyes, ears, mouth corners), 11–22 cover shoulders, elbows, wrists and fingers, and 23–32 cover hips, knees, ankles and feet. Each point ships with x, y, z and a visibility score.

The skeleton here follows that same numbering — the amber dots are literally indexed 0 through 32, and the cyan lines are the standard bone connections between them.

Try this: raise the arms and watch which index range moves — only 11–22 change, because the leg indices don't depend on the arm's rotation.

Kinematic chains: rotations that stack

A wrist doesn't get its own independent rotation — it inherits the shoulder's rotation, then adds the elbow's on top: Rwrist = Rshoulder · Relbow. This is the same matrix composition from the rotation tab, just repeated joint after joint down a limb.

That's why a small change near the shoulder moves the whole arm, while a change at the elbow only moves the forearm and hand — each joint only affects what's downstream of it in the chain.

Try this: set arms raise high, then sweep elbow bend. The hand swings through a wide arc because it's inheriting the shoulder's large rotation on top of its own.

Pose vs. blendshapes

Pose Landmarker tracks the body's large-scale structure — where the joints are in space. Face Landmarker's blendshapes track something finer — how the surface of the face deforms on top of wherever the head happens to be.

A full avatar or motion-capture pipeline typically runs both together: pose for the skeleton's position, blendshapes for the face riding on top of it, each solved independently and then combined.

Try this: flip back to the face tab, set a strong smile, then come back and pose the body. Nothing conflicts — position and expression are separate systems by design.
Input frame
Predicted mask — P(person)
Composited output

Frame & mask

A synthetic person walking through frame, and the per-pixel mask a model would predict for it.

Person position0.00
Person distance100%
Edge feather6 px
Foreground threshold50%
foreground pixel ⇔ P(person) ≥ threshold threshold = 0.50, feather = 6px

Background effect

The mask only decides which pixels are "person." Everything else is free to replace.

A mask is a probability map

Selfie Segmentation doesn't return points at all — it returns one number per pixel, between 0 and 1, estimating how likely that pixel belongs to a person. The middle panel shows this directly: white means confident "person," black means confident "background," and the soft gray fringe in between is genuine uncertainty near the edges — hair, motion blur, semi-transparent clothing.

Try this: push edge feather to maximum. The uncertain fringe around the silhouette widens, the same way a real model gets less certain the closer a pixel is to an edge.

Threshold turns probability into a decision

A probability map isn't directly usable for cutting someone out of a frame — something has to decide which pixels count. That's what the threshold slider does: any pixel at or above it becomes foreground, anything below becomes background. Low thresholds let uncertain edge pixels through, so the cutout grows a soft halo; high thresholds keep only the most confident pixels, which can bite into thin details like hair or fingers.

Try this: set threshold near 0 and compare the output panel to setting it near 100. There's no single "correct" threshold — it's a trade-off a real pipeline has to tune.

Segmentation vs. landmarks

Face and Pose Landmarker both answer "where is this specific, named part" with a sparse set of structured points that carry meaning — point 15 is always the left wrist. Selfie Segmentation answers a different question, "which pixels are a person at all," with a dense, unstructured mask that has no concept of joints, expressions, or identity.

A real product often layers these: segmentation to isolate the person, pose or face landmarks on top to understand what that person is doing.

Try this: notice the output panel has no notion of a "wrist" or a "smile" — it only knows foreground and background. That's the trade: less structure, but it works on anything shaped like a person, not just faces or bodies in a known pose.