A Screen Recording Is Not a Spec: Rebuilding an Interface State by State

That instruction skips the hard part. The recording shows one journey through one viewport. It does not tell you what the interface should do when content is long, the network fails, the user presses Escape, or an updated item disappears from the current filter. The pixels are visible; the product decisions are not.

We explored that gap with Lumen Review Desk, a fictional dashboard made for this article. The useful result was not a magic prompt. It was a workflow: turn the clip into six named states, separate observation from interpretation, give the coding agent a behavior contract, and test the transitions that no screenshot can prove.

The demo was convincing. The missing middle mattered more.

This experiment began with an OpenAI Developers roundup of work presented as GPT-6 Astra examples. In one linked thread, Pietro Schirano shows polished interfaces and says a video was converted into code. It is a compelling demonstration. It is also, quite reasonably for a social post, a highlight reel rather than a build log.

We could not see the original video, exact prompts, discarded attempts, source code, test criteria, or where human editing began and ended. Without those pieces, “recreate a UI from video” is a result, not yet a method. The more valuable article was therefore not a speculative prompt for copying the showcase. It was an attempt to make the missing middle inspectable.

There is a practical API detail too. The current GPT-6 Astra model page lists image input but not native video input. A reproducible workflow should not quietly imply otherwise. In our version, the model-facing material is a set of selected frames, a transition timeline, and written constraints. The video remains the source we inspect, not a file we claim the model watched end to end.

Five seconds of video left us with a dozen unanswered questions

Our clip follows a simple path. The user filters a queue to “Ready,” opens project LN-104, expands its activity, marks it reviewed, and closes the detail drawer. Nothing about the sequence looks mysterious until you try to implement it.

Was the drawer modal, or could the page behind it still be used? Did focus move inside? Would Escape close it? Were the counts calculated or hard-coded? Did the status persist? What should happen on a narrow screen, at 200% zoom, with a long project title, or with reduced motion enabled? The recording answered none of those questions. It also revealed only success: no loading, empty, error, permission, or stale-data state ever appeared.

One ambiguity became surprisingly concrete. LN-104 begins in the Ready filter. Marking it reviewed removes it from that filtered list. When the drawer closes, the control that originally opened it no longer exists. “Return focus to the opener” sounds like a sensible accessibility rule, but here it is impossible. The product still needs a fallback—perhaps the next card, perhaps the active filter—and the clip cannot choose for us.

That was the point at which the project stopped being a visual copying exercise. We wrote the unresolved focus destination down instead of letting the implementation make an accidental decision.

Before doing any of this with a real recording, settle a more basic question: are you allowed to use it? Remove customer information, credentials, private URLs, internal metrics, and account details before frames enter a model or shared workspace. Distinguish the behavior you are studying from protected copy, imagery, branding, or source code. Lumen is synthetic precisely so the downloadable package can include the full recording and implementation without borrowing somebody else’s product.

We turned the clip into six states, not hundreds of frames

Sampling every fraction of a second produces a stack of near-duplicates. We kept a frame when something meaningful changed: visibility, selection, content, status, count, focus, or position. Each surviving frame received a state name and a note describing the action that led to it.

For a longer recording, FFmpeg can create candidates before you review them. A fixed-rate pass makes a contact sheet easy to scan; a scene-change pass can find large visual jumps:

mkdir -p frames candidates
ffmpeg -i recording.mp4 -vf fps=2 frames/frame-%04d.png
ffmpeg -i recording.mp4 \
  -vf "select='gt(scene,0.35)',showinfo" -fps_mode vfr \
  candidates/scene-%03d.png

The FFmpeg filter documentation describes the scene score and suggests values around 0.3 to 0.5 as a generally useful range. It is a starting point, not a state detector. A subtle focus move or text change may score lower than a decorative animation, so we still walked the recording manually and kept the moments that changed behavior.

The state contract did more work than the prompt

For each state, we wrote four things: what the frame visibly established, what we inferred, how confident we were, and what a browser could check. That small distinction prevented guesses from hardening into “requirements.”

We then reduced those notes to a compact contract. It describes outcomes instead of prescribing component names or internal variables:

{
  "id": "S3-activity-open",
  "precondition": "S2-detail-open",
  "trigger": "activate the Show activity control",
  "assert": [
    "three activity events are visible",
    "the control label changes to Hide activity",
    "the dialog remains open and labelled"
  ]
}

The prompt became almost plain after that. We asked for an original implementation of the six states, named the target viewports, required derived counts, keyboard operation, a labelled dialog, a right-edge desktop drawer, no horizontal overflow at 390 pixels, and a reduced-motion path. We explicitly left DOM structure, component boundaries, state representation, and CSS organization to the agent. We also told it not to invent APIs, persistence, analytics, permissions, or routing.

Build an original implementation from the supplied frames and state contract.
Preserve the named transitions, visible counts, keyboard path, dialog name,
desktop drawer geometry, 390px layout, and reduced-motion behavior.

Choose your own DOM, components, state representation, and CSS structure.
Do not assume hidden APIs, persistence, permissions, analytics, or routing.
Return runnable files, automated transition checks, and unresolved decisions.

This is less dramatic than “copy this perfectly,” but much harder to misunderstand. It gives the agent room to engineer while making product uncertainty visible to the reviewer.

Our fast mock copied the mood and missed the interaction

The first version looks plausible at a glance. It has the navy rail, warm background, purple accents, serif heading, cards, and dimmed page. It also puts a narrow modal in the center when the recording shows a drawer attached to the right edge. Activity history is missing. Counts do not change. “Reviewed” is decorative rather than a state mutation. There is no reliable focus path, mobile rule, or confirmation message.

We made this rough version deliberately; presenting it as raw AI output would create a benchmark that never happened. Its role is to reproduce a familiar review failure: visual vocabulary wins approval while interaction grammar goes unexamined.

We rebuilt in a different order. First came semantic controls and all six transitions. Then focus behavior, announcements, Escape handling, and reduced motion. Only after the state graph worked did we tune the large visual relationships—rail, content grid, drawer width, headline scale—before moving to padding, borders, shadows, and timing.

The order saved time. A drawer that is 30 pixels too far left leaves a large, legible signature in a screenshot diff. A wrong dialog pattern can force markup, focus, and layout changes at once. There is little value in polishing the shadow on the wrong structure.

Transition tests caught what screenshots missed

The final browser run contains 28 assertions. It checks the initial items and derived counts, filters the queue, opens LN-104 from the keyboard, confirms that the drawer is a named modal dialog, expands three activity rows, changes the project status, verifies the live feedback, closes with Escape, and confirms that the filtered list now contains only LN-111. The same run checks desktop and simulated mobile overflow, a full-width mobile drawer, reduced motion, console health, and page errors.

That calibration matters. A 3% threshold chosen after seeing the result is decoration. We kept the limit fixed and introduced a defect we expected humans to notice: moving the open drawer 32 pixels left. The comparison rejected it. That does not make 3% universally correct; it tells us this threshold is sensitive to one important failure in this controlled viewport.

We used a 0.12 per-pixel comparison threshold and held the browser, viewport, fonts, content, motion preference, and capture timing steady. Playwright's screenshot assertion documentation explains the available pixel and ratio tolerances. Without a stable environment, the number mostly measures font rendering and timing noise.

Accessibility needed behavioral checks of its own. A pointer trail does not prove keyboard access, a dimmed page does not prove dialog semantics, and an animation does not reveal its reduced-motion branch. W3C guidance covers both content that appears on hover or focus and honoring reduced-motion preferences. Our automation checks part of that surface; screen-reader and hands-on assistive-technology review are still deliberately open.

Reproduce the workflow on your own interface

Download the Screen Recording UI Rebuild Kit

The ZIP includes the five-second reference recording, six keyframes, state contract, build notes, intentionally incomplete mock, separate rebuild, browser results, visual diffs, the seeded calibration failure, a review checklist, and a dependency-free Python release gate. The build notes say who did what and where this exercise stops being independent; they are part of the method, not fine print.

After extracting the kit, serve the directory and replay the interaction:

python3 -m http.server 8766
# open http://127.0.0.1:8766/rebuilt/
# Ready → LN-104 → Show activity → Mark reviewed → Escape
python3 release-gate.py

The release gate should report six states, 28 browser checks, four passing visual comparisons, the detected 32-pixel mutation, and two unresolved human gates. It verifies that the package is internally consistent. It does not pronounce the interface ready for production.

To adapt the kit, replace the recording, frames, content, and state contract with material you are authorized to use. Review the video at normal and slow speed. Name every meaningful transition. Write uncertainties before prompting. Add tests for the parts your recording omitted—loading, empty data, errors, permissions, persistence, destructive actions, long content, zoom, and localization—according to the risks of your product.

Where judgment still beats automation

A coding agent can move quickly once the problem has edges. It can turn a contract into controls and state, generate repetitive assertions, compare stable renders, and fix the obvious gap between a centered modal and a right-side drawer. It cannot recover product intent that was never visible. A screenshot score cannot decide whether the experience is understandable, whether the focus fallback feels natural, or whether the reconstruction is sufficiently original and authorized.

That is why the package leaves two gates open: assistive-technology review and production release. A real team would also review security, maintainability, browser and device coverage, data behavior, language, zoom, and failure states before shipping.

The durable lesson from the five-second clip is simple: a recording is a useful observation, not a specification. Give the agent selected frames and a timeline, but give it something better too—a state contract, named unknowns, and tests for the moments between the screenshots. The result looks a little less like magic and a lot more like software another person can inspect, challenge, and improve.