REDSMARTTV

Emulator / Testing

The emulator also writes the report

Both ways press remote-control keys on your app and produce the same kind of report. The report says which one produced it.

Two ways

Exploration and Scenario both produce the same report Two paths run from the application under test. Exploration presses keys on its own and walks the app breadth-first. Scenario replays a path recorded by hand. Both converge on one report format, which records which of the two produced it, and that report exports as HTML, Markdown or JSON so two runs can be compared. Exploration presses the keys itself Scenario replays your recording One report findings · limits · gaps and which run produced it Export HTML · Markdown · JSON two runs can be diffed
A search reaches breadth. A recording reaches what a search structurally cannot: number keys, free-text fields, a video operated deliberately. The report says which one it came from.

Exploration

Exploration

The emulator presses the keys itself and visits as many screens as it can find. Nothing to set up. It will not reach everything — and the report lists what it did not reach, which is the part most tools leave out.

Scenario

You walk through the app once with the keys, the recording gets a name, and you replay it after every change. The first replay becomes the baseline; later runs are compared against it.

This is the way for things a search structurally cannot reach: number keys, free-text fields, a specific video operated deliberately.

The report

What a report contains

Findings
Where your app does something a television would not accept. Most name the passage in the standard; where no passage covers the case, the finding says so. “Replay” presses the same keys again so you can watch it happen.
Emulator limits
Where the emulator behaves differently from a television. These are not defects of your app, and keeping them in a separate section is the difference between a report you act on and a report you argue with.
Not reached
What the run did not cover, so you know what the report cannot say.
AI summary
Optional, with your own Anthropic key. A model rephrases and prioritises the findings. It never adds or removes one: severity and the standards citations are decided before the call. The model phrases and weighs; it does not classify.
An extract from a test report: the Not reached and Emulator limits entries, each citing the passage in the standard it follows from
From a real report. Each entry says why the run could not go further — and cites the passage it follows from, down to the table and page number.

A real report

An HbbTV test report: run metadata, the standard it was measured against, and one grouped finding with its occurrences
An actual export. The host has been replaced for publication; everything else is a real run. Note the line naming the standard the run was measured against, and that like findings are collected into one block with every key sequence kept.

Citations

Around 80 rules, each anchored in the standard

Findings are not opinions. Each rule carries the clause it comes from — keyset#group-not-available, av-control#autoplay-blocked, page#mixed-content-blocked, run#reset-incomplete — and the citations are machine-checked on every release of the tool, along with the page numbers.

Readability

Why the report is short

An early run against a real broadcaster application produced 53 findings. Twenty-three of them were one rule and twenty-one were another: 299 of 348 lines, 86 %, were two rules — and the four playback errors that actually mattered were buried underneath.

Like findings are now shown as one block: the explanatory prose once, every key sequence in full. 456 lines became 220; 53 findings became 9 blocks. What is grouped is the display, not the measurement — the JSON export and the run-to-run comparison still carry every finding individually.

We mention this because it is the kind of thing a test tool usually hides. A report nobody reads is not a test result.

Repeatability

Every run starts from the same place

Each run reloads the site with its cookies and storage cleared, so the app begins identically every time. Anything you are signed in to on that site will be signed out, other subdomains included. What the run cannot clear — HttpOnly cookies, a session held on the server — the report names as “reset incomplete” rather than quietly ignoring.

Comparing two runs

Export as HTML to send on (self-contained), JSON for a build pipeline, or Markdown for a ticket. Two runs can be diffed, so after a change you see 12 fixed · 0 new rather than reading 200 lines again.

Measured

Our reference app is a production broadcaster application we also built. Under a recorded scenario it produced 0 findings of run#not-reproducible, against 23 in an exploration run of the same app.

Read the Bibel TV case study

Next