07/08

Offbeat

Your photos, cut on what you hear.

About

Turns a folder of camera photos and a song into a short video where every cut lands on a kick, a snare or an accent. Photos ordered by colour, the cut playing live before anything renders, all on your own computer: no uploads, no accounts. Web app and native apps for Mac, iPhone and iPad.

In numbers

transitions, chosen by each hit’s weight: 20
openings that move with the song, the Vinyl among them: 8
looks that set the whole frame: 6
automated tests across web and native: 800+

Problem

Cutting photos to a song by hand means hunting for every kick and snare and nudging each cut into place. Offbeat does the listening, and leaves the choices to you: every cut can be moved, added or tapped in.

Constraints

Everything runs on your own computer: no uploads, no accounts, no telemetry
Big camera files: 40-megapixel JPEGs and RAW
Formats for Reels, TikTok, Shorts and YouTube, up to 4K

Decisions

A small local model (Beat This!) finds beats and downbeats with PyTorch, with librosa as the fallback.
The song and the photos never leave your computer.
Not chosen: ~~Cloud audio analysis, which would mean uploading the music and the photos.~~
Keep to the beat is on by default: cuts sit on the beats the kick and snare land on, and an off-beat accent cuts only when it stands out.
Cuts lean to the kick, flashes and effects go to the snares and claps, and every cut lands on an export frame no more than 5 ms after its hit.
The live preview and the final export read the same timeline, and the preview draws transitions and effects on the GPU.
A parity test holds the preview’s frames to the export’s, so what plays is what renders.
Limit flashing is on by default: nothing flashes more than three times a second.
Flashes, dips and shutters are spaced the same way in the preview and the export (WCAG 2.3.1).
Match the cover reads the album art inside the song file and picks photos in those colours.
The video takes its colours from the song’s own sleeve; a black and white cover picks the black and white photos.
Song on Instagram exports the video without sound and writes the title, artist and start time beside it.
Instagram mutes a video whose song it has no licence for, so the same song can be chosen from Instagram’s library and lined up.

Outcome

A web app, and native apps for Mac, iPhone and iPad, kept in step through the later rounds: the new transitions and effects, title styles, Match the cover, the Vinyl opening and Song on Instagram.