One iPhone LiDAR walk becomes a measured digital twin of your home. A home robot finds itself in it with no stickers, takes spoken orders in any language through NVIDIA Nemotron, and asks you when it isn't sure.
It has never seen your rooms. The usual fixes cost the person time, or the robot maker a test home. Everyone already carries a phone, and many phones have LiDAR.
Markers in every room, put up by hand.
Extra hardware to buy, place and power.
The robot crawls the whole home first, often with a person watching.
Walk through once, tap each room. The map is ready for any robot.
The app records colour, LiDAR depth and the ARKit pose at about 4 frames a second. You tap each room as you enter it, then upload. Sign in, record, upload, My maps.

OpenCV 5 turns the walk into floor, walls, doors, rooms with their sizes, furniture and landmarks, as one house.json, plus a 3D view, a MuJoCo scene and now a USD stage for Isaac Sim. A browser editor fixes room names.
The robot turns once and stacks its 8×8 depth readings into a 360° scan, then matches it against every free spot and heading in the twin. It only counts as found when the best place clearly beats the runner-up, and keeps checking the walls as it walks.

Nemotron 3 Nano Omni writes the words down. Nemotron 3 Ultra picks this home's rooms through tool calls over house.json, and the robot says its plan aloud. Vague or unsafe? It asks. “Stop” works at once, without a model.
Pollen's Microduck software and its trained walking policy drive a simulated body in the twin of the real flat. When it is lost, or doubts an arrival, it asks a person. Every step goes in a trace, and a film replays it.
This is the twin of the flat we scanned, drawn live from its house.json. Each room is coloured by its floor height from the scan, so a robot knows where the floor steps before it gets there.
The route stops 40 cm before the first step taller than the robot can climb. A few cm of ARKit drift stay; steps under 5 cm are noise.
The same twin exports to USD with physics: floor tiles at their measured heights, walls with door gaps, a collider on every solid, and the scan's colours on top. In NVIDIA's simulator, Nova Carter drove the route Nemotron planned on its own wheels, and stopped 7 cm from the planned stop, before the 14 cm step.
Each body gets its own route (A* with its radius) and its own stop. *Step limits are assumed, not measured. Clip: 12.8 s, rendered headless on an RTX 4090.
Every call goes through one OpenAI-compatible client and is logged with its provider, model id, latency and tokens. Numbers measured in development on NVIDIA's API catalog, same Nemotron models.
Spoken command → which rooms, in which order, by tool calls over this home. Also the robot's questions and its patrol report.
Microphone audio → text, once per spoken command, in the speaker's own language.
The same 162-command test at a fraction of the price.
Camera frame → which room, on 191 real iPhone frames, against the colour vote.
Commands with typos, vague, impossible and unsafe requests, on the sample home and the real flat (8 rooms). out/commands/results.json
The robot's language runs through one OpenAI-compatible client pointed at Nebius Token Factory, with NVIDIA's open Nemotron models. Switching providers is one setting, and one script re-runs every test quoted on this page.
One set of numbers. Being sure fast is easy; being sure and right is the job.
Spoken commands mapped to the right rooms, 162 commands, 6 languages.
Goals reached in 22 test houses × 4 seeds, rough sensor.
Runs that locked onto a wrong place. False arrivals: 56 → 0.
Live walk in the real flat's twin: sure and right about where it is.
From the garden goal, after “revisa la cocina, luego el pasillo y después el jardín”. 0 bumps.
Spanish commands right with Nemotron 3 Ultra.
out/commands/results.json · eval/selfloc.py · the live walk in out/walk/. Model numbers: dev, NVIDIA API catalog.
Nemotron VL on 191 real frames: 78.9 % vs 75.2 % on the room the camera sees, and its “sure” is ≥ 0.9 right or wrong. So it is not in the robot's loop.
Twice Ultra walked to the pool or the garden on an unsafe request; twice a failed call fell back to keywords. The fallback needs to ask instead.
From an unknown start on real chest-height frames the robot is within 0.3 m 52 % of the time; from a known start, 85 %.
Measured on NVIDIA's API catalog. Omni on the free tier failed 5 of 14 live calls (503).
For home-robot makers first. The end use: check-in rounds for someone living alone. “Is the stove off?”, asked in their own words.