A room photograph can change in different ways during virtual staging. A tool may crop the view without moving a window inside it. It may preserve the camera frame while altering a permanent feature. An automatic check may also lack enough evidence to decide whether framing survived.
Those possibilities need separate labels. StagingBench records framing as passed, failed or unknown. Counting those outcomes is a useful first inspection of its partial data. It also forces a basic question: which room records are actually present in the file being analyzed?
The inputs are real CC0 photos of real rooms. The building in each original provides the reference. The aim here is to summarize recorded framing outcomes, while keeping claims about moved walls, doors, ceilings or floors tied to supporting evidence.
Read the room records
The JSON has a top level tools array. An initial result can live in a tool’s room01 dictionary. Additional results can live in its rooms list, where each record has a room field.
A script that reads only the list misses initial results. Here, a dictionary keyed by room identifier combines both locations. If room 01 appears in both, the explicit room01 dictionary takes precedence.
Run this snippet from the repository root. It uses only Python’s standard library.
import json
from collections import Counter
from pathlib import Path
data = json.loads(Path("data/results.json").read_text())
print("tool | passed | failed | unknown")
for tool in data["tools"]:
rooms = {r["room"]: r for r in tool.get("rooms", [])}
if isinstance(tool.get("room01"), dict):
rooms["01"] = tool["room01"]
counts = Counter(
"passed" if r.get("framing_pass") is True else
"failed" if r.get("framing_pass") is False else "unknown"
for r in rooms.values()
)
print(tool["id"], *(counts[k] for k in
("passed", "failed", "unknown")), sep=" | ")
The explicit identity checks matter. A null value must not quietly become a failure through a generic truthiness test. An absent framing value within an existing room record is counted as unknown. A tool with no room records contributes no outcomes.
What the executed script returned
These counts come from the supplied JSON snapshot, credited to StagingBench. The script was run against that file.
| Tool ID | Passed | Failed | Unknown |
|---|---|---|---|
| reimaginehome | 1 | 0 | 0 |
| collov-ai | 4 | 0 | 1 |
| apply-design | 1 | 0 | 0 |
| boxbrownie | 0 | 0 | 0 |
| styldod | 0 | 0 | 0 |
| rooomy | 0 | 0 | 0 |
| stuccco | 0 | 0 | 0 |
| photoup | 0 | 0 | 0 |
| virtualstaging-ai | 0 | 1 | 0 |
| pedra | 4 | 1 | 1 |
| roomagen | 2 | 0 | 0 |
| tinty | 0 | 1 | 0 |
| homai | 0 | 0 | 0 |
| meltflex | 0 | 0 | 1 |
| immostage | 0 | 0 | 0 |
| virtualstagingflow | 9 | 0 | 1 |
| ai-homedesign | 0 | 0 | 0 |
| trolto | 0 | 0 | 0 |
| chatgpt-baseline | 0 | 0 | 0 |
| gemini-baseline | 0 | 0 | 0 |
Zeroes across a row mean no room records were counted. They do not mean the tool failed, passed or received an unknown result on every intended test room.
The test log contains later REimagineHome and TINTY room runs that are absent from this JSON snapshot. The table deliberately does not merge them. Mixing sources silently would make a reproducible file summary look more complete than its input.
A failed frame does not prove a moved wall
TINTY room 01 provides a clear recorded example. Its note says the default crop returns a 3:2 image from a 4:3 original. About 54 pixels were cut from the top and bottom. Framing failed, while the note says the architecture inside the visible area matched closely.
That is a crop finding. It does not establish that a window, wall or door moved. The TINTY tool page gives the result’s context. The test log records this run as an anonymous preview.
The methodology separates framing from architecture drift for this reason. Alignment precedes the structure comparison. When alignment fails, framing is unknown and drift falls back to an unaligned value. A count of framing labels cannot replace that interpretation.
Keep compliance outside the counter
The MLS rules summary, last checked on 19 October 2026, describes MLS Now as prohibiting changes to permanent features including walls, windows, doors, ceilings and floors. It gives different disclosure locations for different MLS policies. CRMLS uses the photo description field. MLS Now requires a label on the image itself.
No branch in this script checks those requirements. It does not inspect a floor, recognize a removed door or confirm a required label. A passed frame therefore cannot be presented as an MLS compliance approval.
The benchmark remains partial. Blind human scoring is planned and has not started. These counts describe the available records, with unequal coverage across tools. They are a starting point for opening the relevant notes and comparisons, not evidence of a universal success rate.