Nebius × NVIDIA Global AI Hackathon

Walk your home once. Any robot knows it.

One iPhone LiDAR walk becomes a measured digital twin of your home. A home robot finds itself in it with no stickers, takes spoken orders in any language through NVIDIA Nemotron, and asks you when it isn't sure.

Nemotron 3 Ultra Nemotron 3 Nano Omni Isaac Sim 6.2 Built for Nebius Token Factory
Replay · a real run in the scanned flat
You
Nemotron 3 Ultra · tool call over this home · 1.4 s
kitchen “Checking the kitchen for the stove.”
A* route · Nova Carter, 30 cm wide
passway → kitchen
Floor steps 14 cm · this body climbs 3 cm
“I can climb 3 cm, but the floor steps 14 cm from passway into kitchen. Can you check kitchen for me?”
From route.json: Ultra's plan, the route and the question it wrote · Isaac Sim, real flat
The problem

A new robot in your home is lost.

It has never seen your rooms. The usual fixes cost the person time, or the robot maker a test home. Everyone already carries a phone, and many phones have LiDAR.

Stickers on the walls

Markers in every room, put up by hand.

Beacons

Extra hardware to buy, place and power.

A long mapping run

The robot crawls the whole home first, often with a person watching.

One phone walk

Walk through once, tap each room. The map is ready for any robot.

How it works

From a phone walk to a robot that asks.

01 · Walk

Record the home with the iPhone app.

The app records colour, LiDAR depth and the ARKit pose at about 4 frames a second. You tap each room as you enter it, then upload. Sign in, record, upload, My maps.

The iPhone app: sign in, My maps, floor plan with room sizes, 3D view
02 · Twin

The server builds a measured twin.

OpenCV 5 turns the walk into floor, walls, doors, rooms with their sizes, furniture and landmarks, as one house.json, plus a 3D view, a MuJoCo scene and now a USD stage for Isaac Sim. A browser editor fixes room names.

03 · Find itself

One turn on the spot, no stickers.

The robot turns once and stacks its 8×8 depth readings into a 360° scan, then matches it against every free spot and heading in the twin. It only counts as found when the best place clearly beats the runner-up, and keeps checking the walls as it walks.

The MuJoCo twin of the flat
04 · Listen

Spoken orders, in your words and language.

Nemotron 3 Nano Omni writes the words down. Nemotron 3 Ultra picks this home's rooms through tool calls over house.json, and the robot says its plan aloud. Vague or unsafe? It asks. “Stop” works at once, without a model.

Españolrevisa la cocina, luego el pasillo y después el jardínEnglishgo check the kitchen, I think I left the stove onFrançaisvérifie la cuisineDeutschgeh ins SchlafzimmerNegationdon't go near the pool, just check the garden Españolrevisa la cocina, luego el pasillo y después el jardínEnglishgo check the kitchen, I think I left the stove onFrançaisvérifie la cuisineDeutschgeh ins SchlafzimmerNegationdon't go near the pool, just check the garden
Portuguêsvai na cozinhaItalianovai in camera da lettoTyposchek the livng room n then the hallAll butall the rooms but not the garden or the poolMust askcheck on grandma's room Portuguêsvai na cozinhaItalianovai in camera da lettoTyposchek the livng room n then the hallAll butall the rooms but not the garden or the poolMust askcheck on grandma's room
05 · Patrol and ask

It walks each room and asks when unsure.

Pollen's Microduck software and its trained walking policy drive a simulated body in the twin of the real flat. When it is lost, or doubts an arrival, it asks a person. Every step goes in a trace, and a film replays it.

The twin

A real flat, measured down to its steps.

This is the twin of the flat we scanned, drawn live from its house.json. Each room is coloured by its floor height from the scan, so a robot knows where the floor steps before it gets there.

Nova Carter · passway → kitchen0.0 m
Rooms–
Walls–
Doors–
Furniture boxes–
Floor heights (pool → living room)–
Floor height
lowhigh

The route stops 40 cm before the first step taller than the robot can climb. A few cm of ARKit drift stay; steps under 5 cm are noise.

NVIDIA Isaac Sim 6.2

One scan, any robot body.

The same twin exports to USD with physics: floor tiles at their measured heights, walls with door gaps, a collider on every solid, and the scan's colours on top. In NVIDIA's simulator, Nova Carter drove the route Nemotron planned on its own wheels, and stopped 7 cm from the planned stop, before the 14 cm step.

house.jsonto_usd.pyhouse.usdIsaac Sim · RTX 4090

NVIDIA Nova Carter

width (radius)30 cm
climbs3 cm*

Pollen Microduck

width (radius)12 cm
climbs1.5 cm*

NVIDIA Jetbot

width (radius)10 cm
climbs1 cm*

Each body gets its own route (A* with its radius) and its own stop. *Step limits are assumed, not measured. Clip: 12.8 s, rendered headless on an RTX 4090.

NVIDIA Nemotron

Each model has one job.

Every call goes through one OpenAI-compatible client and is logged with its provider, model id, latency and tokens. Numbers measured in development on NVIDIA's API catalog, same Nemotron models.

Language → plan

Nemotron 3 Ultra

Spoken command → which rooms, in which order, by tool calls over this home. Also the robot's questions and its patrol report.

91.4 %commands right · p50 1.9 s
Speech → words

Nemotron 3 Nano Omni

Microphone audio → text, once per spoken command, in the speaker's own language.

2.0 sp50 · p95 4.9 s
Tested, not used

Nemotron 3.5 Lightning

The same 162-command test at a fraction of the price.

82.1 %commands right · p50 1.3 s
Tested, not in the loop

Nemotron 3.5 Super VL

Camera frame → which room, on 191 real iPhone frames, against the colour vote.

78.9 %vs 75.2 % colour vote: no real gain

162 spoken commands, two homes, six languages

keywords (no model)Nemotron 3 Ultra

Commands with typos, vague, impossible and unsafe requests, on the sample home and the real flat (8 rooms). out/commands/results.json

Safety by design

Models plan. They never steer.

The 5 Hz loop never waits. Sense, locate, decide, move: the robot's loop runs on its own; a model answer arrives when it arrives.
One call in flight. Stale frames are dropped; the loop uses the last answer.
A command becomes goals, not motion. A* plans every path; the particle filter decides where the robot is.
When unsure, it asks. A failed call falls back to a keyword plan and says so. “Alto” stops it without a model.
sense locate decide move 5 Hzrobot loop Nemotron
Nebius Token Factory

Built to run on Nebius.

The robot's language runs through one OpenAI-compatible client pointed at Nebius Token Factory, with NVIDIA's open Nemotron models. Switching providers is one setting, and one script re-runs every test quoted on this page.

Said plainly: a Nebius account could not be opened from Peru during the hackathon, so the numbers on this page were measured on NVIDIA's API catalog, with the same Nemotron models. Costs use Token Factory's list prices.
One client, one log for every call (provider, model, latency, tokens)
scripts/nebius_rerun.sh re-runs commands, camera and self-location tests
The 22-house self-location sweep as parallel Serverless Jobs
# config.toml [nebius] provider = "nebius" # Token Factory [nebius.nebius] base_url = "https://api.tokenfactory.nebius.com/v1" ultra = "nvidia/Nemotron-3-Ultra-550b-a55b" omni = "nvidia/nemotron-3-nano-omni-30b-a3b-reasoning" vl = "nvidia/Nemotron-3.5-Super-VL" lightning = "nvidia/Nemotron-3_5-Lightning" # then $ MODEL_PROVIDER=nebius scripts/nebius_rerun.sh
$0.97per 1000 commands, Ultra
$0.06per 1000 commands, Lightning
$0.0011the kitchen plan above
Results

Sure, and right.

One set of numbers. Being sure fast is easy; being sure and right is the job.

0%
keywords: 55.6 %

Spoken commands mapped to the right rooms, 162 commands, 6 languages.

0%
before: 51 %

Goals reached in 22 test houses × 4 seeds, rough sensor.

0/ 96
before: 89 / 96

Runs that locked onto a wrong place. False arrivals: 56 → 0.

0s
before: 61 s

Live walk in the real flat's twin: sure and right about where it is.

0cm
kitchen 28 · passway 13

From the garden goal, after “revisa la cocina, luego el pasillo y después el jardín”. 0 bumps.

0/ 23
keywords: 2 / 23

Spanish commands right with Nemotron 3 Ultra.

out/commands/results.json · eval/selfloc.py · the live walk in out/walk/. Model numbers: dev, NVIDIA API catalog.

Real

  • The iPhone LiDAR walks of a real flat: colour, depth, ARKit pose, room taps
  • The app, the upload, the twin build on the server
  • Pollen's robot software and its trained walking policy
  • Speech from a real microphone; every Nemotron call, logged with its latency

Simulated

  • The robot's body, its 8×8 depth sensor and camera, inside the twin (MuJoCo, Isaac Sim)
  • The person's answers in filmed walks, unless said otherwise
  • Furniture in the twin is boxes
  • Robot step limits in Isaac Sim are assumed

Vision didn't beat colours

Nemotron VL on 191 real frames: 78.9 % vs 75.2 % on the room the camera sees, and its “sure” is ≥ 0.9 right or wrong. So it is not in the robot's loop.

Unsafe commands: 4 in 40 got through

Twice Ultra walked to the pool or the garden on an unsafe request; twice a failed call fell back to keywords. The fallback needs to ask instead.

The body is simulated

From an unknown start on real chest-height frames the robot is within 0.3 m 52 % of the time; from a known start, 85 %.

Not yet on Token Factory

Measured on NVIDIA's API catalog. Omni on the free tier failed 5 of 14 live calls (503).

Built with
NVIDIA Nemotron 3 UltraNemotron 3 Nano OmniNemotron 3.5 Super VLNemotron 3.5 Lightning NVIDIA Isaac Sim 6.2 · OpenUSDNova Carter · JetbotNebius Token Factory iPhone LiDAR · ARKit · SwiftUIOpenCV 5MuJoCoPollen MicroduckFastAPI
House Twin

Turn every beta tester's home into a test bench.

For home-robot makers first. The end use: check-in rounds for someone living alone. “Is the stove off?”, asked in their own words.