Estimating a pulse from a webcam in Rust and WebAssembly

Heartrate is a browser page that estimates a pulse from a webcam, without contact. With each heartbeat a little more blood reaches the skin of the face, haemoglobin absorbs a little more green light, and the colour of the skin moves by roughly 0.1 to 0.5 percent: invisible to the eye, measurable in the average of thousands of pixels over a few seconds. The technique is called remote photoplethysmography, rPPG.

It is live at heartrate.tamasvigh.com. There is no server-side code. The video never leaves the machine. Without a camera, heartrate.tamasvigh.com/?demo shows a synthetic low-poly face pulsing at 72 BPM, measured by the same signal path with the face tracker left out. It is not a medical device. What follows are the parts that turned out stranger than expected.

A face becomes one colour per frame

MediaPipe’s face landmarker fits a 468-point mesh to the face, and three regions are built from it: the forehead and both cheeks, skin with a good blood supply away from the eyes, mouth and hairline. On MediaPipe’s canonical face, a reference face in centimetres, no eye, brow, lip or nose point falls inside them. The forehead stops 2 mm short of the brows, the cheeks 8 mm from the nose.

The face is cropped to 256x256 and handed to a Rust core compiled to WebAssembly, which averages the region pixels into one RGB value per frame. It skips pixels that are too dark or clipped, and those that are not skin-coloured unless fewer than 30 percent pass that test, as under coloured light. Frames are timestamped by requestVideoFrameCallback (capture time where the browser provides it, presentation time otherwise) and resampled to an even 30 Hz, since a spectrum assumes even spacing. Every half second the last ten seconds become one estimate.

The pulse is smaller than one brightness level

Each channel of a pixel is an integer from 0 to 255. Blood does not change the channels equally: relative to their means, red, green and blue move in the ratio 0.33 : 0.77 : 0.53 (de Haan and van Leest, 2014), the blood-volume direction. A pulse of 0.3 percent along it moves green by 0.23 percent, which on a green value of 130 is 0.3 of one level. The heartbeat lives below the resolution of a pixel.

A unit test of the demo’s frame generator takes 40,000 pixels with a green value of 150 and adds a pulse worth 0.29 of a level. Without noise the mean green is exactly 150: every pixel rounded the pulse away. With Gaussian noise of standard deviation (σ) two levels added before the rounding, it comes back within 0.05 of 150.29.

This is dithering. With noise wider than a level, each pixel’s rounded value is random, but its expected value is the unrounded one, so the rounding error becomes ordinary noise that averaging divides by the square root of the pixel count: were the roughly 3,700 pixels of the three regions independent, two levels would become 0.03. A canvas of flat triangles is noiseless, so the demo adds noise, as does the fake camera. The calibration below uses 0.3 levels after averaging (0.1 in the strong case), about ten times that, but still independent white noise. Real camera noise is not: compression, flicker and movement act on whole regions.

Uniform brightness changes cancel exactly

The colour also moves when the light does: a cloud, or automatic exposure. POS, Plane-Orthogonal-to-Skin (Wang, den Brinker, Stuijk and de Haan, 2017), is built to remove that. In each 1.6-second window every channel is divided by its own mean, and two combinations of the channels are formed:

S1 = G − B
S2 = G + B − 2R

A change that scales all three channels by the same factor leaves the normalised channels identical, and each combination’s weights sum to zero, so both are zero: not small, zero. The unit test modulates brightness by 5 percent at 1.2 Hz and requires the output to stay under 1e-4. It is about 5e-6, floating-point rounding, against roughly 0.07 for a 0.3 percent pulse. A real exposure change is such a factor only approximately, and white balance is not one at all (see the last section).

The pulse is not common to the channels: along the blood-volume direction it keeps 0.24 of its amplitude in S1 and 0.64 in S2. The two are summed, weighted by the ratio of their standard deviations in each window. Nothing is fitted per person or camera.

Zero-padding locates one peak. It cannot separate two

The last ten seconds, 300 samples, have a fitted straight line subtracted, are tapered to zero at both ends by a Hann window so the cut edges do not smear the spectrum, and are padded with zeros to 4,096 points: FFT bins 0.44 BPM apart. Padding adds no resolution. Ten seconds still blur rhythms several beats per minute apart. But one dominant peak can be located far more finely than two can be separated: a parabola through the log power of the peak bin and its neighbours puts clean synthetic sines within 0.5 BPM.

A falling spectrum has its maximum at 42 BPM

The search, from 42 to 240 BPM, takes the strongest true local maximum, a bin higher than its neighbours, not the plain maximum. Slow drift makes a spectrum fall with frequency, and the maximum of a falling curve is its left edge: 42 BPM, a believable heart rate. When the band holds only a slope, there is no peak.

The quality gate throws away good estimates

Each estimate carries an SNR after de Haan and Jeanne (2013): the power near the peak and near twice its frequency against the rest of the band, equal at 0 dB, double at 3 dB. Good starts at 3 dB, fair at 0. The thresholds come from synthetic colour traces, 20 random seeds of 20 seconds at 72 BPM (SNR as 5th, 50th and 95th percentiles, noise σ in levels after averaging):

Case	SNR p5 / p50 / p95 (dB)	Raw peak within ±5 BPM
Noise only, σ 0.3	−7.1 / −4.6 / −0.3	n/a
Weak: 0.1% pulse, σ 0.3	−6.9 / −3.6 / 0.2	24%
Middle: 0.3% pulse, σ 0.3	−4.1 / 0.3 / 3.2	92%
Strong: 0.3% pulse, σ 0.1	5.0 / 7.4 / 9.7	100%

Fewer than one noise-only window in twenty reaches 0 dB, so the fair threshold keeps almost all pure noise off the screen. In the middle case the noise is as large as the green channel’s pulse and 92 percent of raw peaks, before any gating, fall within ±5 BPM, but the median SNR is 0.3 dB, so only about half the windows pass. The gate discards correct estimates to keep wrong ones out, and pays in time to the first number.

The display shows the median of the last five fair-or-better estimates once there are three, and only while the current one is not poor. Six poor estimates in a row clear them. A harmonic winning one window, 140 among 70, 71, 72 and 70.5, leaves the median at 71, where the mean would be 84.7.

The fake camera plays a whole number of heartbeats

Chrome can use an uncompressed Y4M video file as its camera, so the end-to-end test generates one: 320x240, 30 fps, ten seconds of flat skin colour carrying a 78 BPM pulse at 0.5 percent along the blood-volume direction, with σ 2 pixel noise and a 3 percent, 0.1 Hz light drift.

The fake camera loops the file, and a fractional number of beats would leave a jump in the pulse wave every ten seconds, inside nearly every analysis window. 78 BPM is 1.3 Hz: thirteen whole periods, and the drift exactly one, so the loop has no seam.

The test leaves out one part of the pipeline: a flat colour gives MediaPipe no face to find, so ?landmarks=static puts the canonical face in place of the tracker. Everything else runs as in production, from getUserMedia to the DOM, served locally with the production headers, including script-src 'self' 'wasm-unsafe-eval'. The test waits for 78 ± 3 on screen, quality not poor and no console errors. MediaPipe’s injected loader <script> was checked under that policy separately.

Asking for 30 frames per second delivered 26

The demo is the same idea for visitors: 898 flat-shaded triangles of MediaPipe’s canonical mesh, with a pulse on the skin and drift and noise on the whole face. Only face pixels are recomputed, about 1 ms per 640x480 frame, and canvas.captureStream() feeds the canvas to the <video> element a camera would use.

captureStream(30) treats its argument as a limit and dropped frames whose timing wobbled slightly around 33 ms: about 26 arrived per second. With no limit and the page pacing itself, 29 to 31 arrived, unevenly spaced, which the resampler absorbs. That first pacer drew once about 29 ms had passed since the last draw, but requestAnimationFrame fires only at the display refresh: at 50 Hz every 20 ms, so the first callback past 29 ms comes at 40, which is 25 fps. 75 Hz lands on 40 as well. The replacement advances a target time by a thirtieth of a second, so a late frame does not push the schedule back. Its unit test counts 299 to 301 frames in ten seconds at nine refresh rates from 50 to 240 Hz.

The signal processing is the cheap part

Signal processing on every frame sounds like the thing to move off the main thread. Measured in Node on a 256x256 crop with a 31,000-pixel region, about eight times the real ones, averaging a frame takes 0.37 ms and one estimate 0.25 ms, about 12 ms of work per second. The face tracker takes a few milliseconds per frame on the GPU. So there is no Worker, no SharedArrayBuffer and none of the COOP and COEP headers it would require.

The core is a dependency-free Rust crate, with its own FFT and a biquad filter used only for the drawn waveform, behind a raw C ABI of 20 functions that take and return only numbers and pointers. Without wasm-bindgen, one cargo command builds a 60 KB module, about 23 KB gzipped. The price is a visible boundary. Rust owns the buffers and JavaScript writes into them, and a call that allocates may grow the wasm memory, which detaches the old ArrayBuffer. A view made before the call is dead after it:

setFrame(rgba: Uint8ClampedArray, width: number, height: number): void {
  const ptr = this.x.hr_frame_ptr(this.session, width, height)
  // The call may have grown memory: create the view only after it.
  new Uint8Array(this.x.memory.buffer, ptr, width * height * 4).set(rgba)
}

The download is lopsided the same way: about 11 MB of self-hosted MediaPipe runtime and a 3.7 MB model, fetched only after Start, against 60 KB of signal processing.

What does not work

Accuracy on real faces is unmeasured. The target is ±5 BPM sitting still in good light. Nothing has been checked against a reference device. The page can record a trace with a reference BPM for a Rust regression test, but there are no recordings yet to retune the SNR thresholds, which come from synthetic white noise.

The demo is too clean to say anything about accuracy. It reaches 72 BPM rated good in about eight seconds at an SNR of 18 to 20 dB, far above the strong synthetic case. It shows the pipeline running, not how accurate it is on a real face.

No automated test tracks a face. The fake-camera test and the demo test both use static landmarks, and the end-to-end suite runs only in Chromium.

White balance is not cancelled. POS removes a change common to all three channels, not a camera adjusting each channel differently.

Skin tone, light, motion. Darker skin gives a weaker signal, because melanin absorbs more of the light: a known bias of rPPG, untested here. Dim light, movement, talking, beards and glare from glasses are expected to degrade the estimate. None has been measured.

The beat is not a beat. The flashing dot and the optional heartbeat sound follow the measured rate, not individual beats, and the page says so.

Each synthetic input needed one deliberate choice to be a fair test: noise so that rounding would not erase the pulse, a loop with no seam. None of them reaches the tracker. What remains is what no synthetic input can supply: a real face, a real camera, and a device beside them that knows the answer.