How AR measurement works

The short answer

An AR measuring app builds a three-dimensional model of the space in front of the phone and tracks the phone’s position inside it. When you tap, the app projects a line from the camera through the point you tapped until it hits a surface in that model, and records where. Two taps give two points in the model, and the distance between them is the measurement. Everything else — calibration, drift, depth sensors — is about how good the model is and how accurately the phone knows where it is.

Visual-inertial odometry in plain words

The core technique is called visual-inertial odometry. “Odometry” means working out how far you have travelled; “visual-inertial” means doing it from two sources at once.

The visual part is the camera. Software picks out distinctive points in each frame — the corner of a picture frame, a knot in a floorboard, the edge of a skirting board — and follows them from frame to frame. If a point drifts left across the image as you move, the phone must have moved right; if points near the camera move faster than points further away, that difference reveals depth. Tracking many such points at once gives a running estimate of how the camera is moving through the scene.

The inertial part is the phone’s accelerometer and gyroscope, which report how the phone is accelerating and rotating hundreds of times a second. On their own they are terrible at position: integrate a slightly wrong acceleration twice and the error balloons within seconds. But they are fast, they work in the dark, and they know the scale of movement in real units.

Fusing the two is what makes the system work. The camera keeps the inertial sensors honest about where the phone is; the inertial sensors tell the camera what a metre actually is, and fill in the gaps when the image is briefly useless. A camera alone can tell that a wall is twice as far as a table but not whether the table is one metre or two; the accelerometer settles that, because it measured how far the phone moved between views.

World tracking and drift

The output of all this is a coordinate system anchored to the room — Apple calls it world tracking — in which the phone’s position is updated every frame. The points the app has learned to recognise become a sparse map of the room, and when the phone sees them again it can correct its position against them.

The catch is that every position update carries a small error, and errors accumulate. Walk from one end of a room to the other and the app’s idea of where you are may be a centimetre or two off by the time you arrive. That is drift. It is not a bug so much as arithmetic: the phone is adding up thousands of tiny estimates, and no estimate is perfect. Drift is why a measurement between two points placed far apart in time and space is less reliable than one placed quickly at arm’s length, and why accuracy is a property of the session, not the phone.

Good AR systems fight drift by recognising places they have seen before and snapping the map back into alignment. When that happens you sometimes see a placed point jump a few millimetres. That is the correction, not the error.

Plane detection

Tracked feature points on their own are a cloud of dots. To measure a wall, the app needs to know that a particular group of dots all lie on one flat surface. Plane detection looks for sets of points that fit a horizontal or vertical plane and grows that plane outwards as more points agree with it. A floor is found first because it is large and usually textured; walls follow; tabletops and doors come as the phone sees them.

This is why a plain white wall is a problem for a camera-only phone. No distinctive points means no dots, no dots means no plane, and no plane means the app has nothing to place your tap on. It is also why measuring apps prefer you to keep the floor-to-wall edge in view: the skirting line is a rich source of points that belong to both planes.

What happens when you tap

When you tap the screen, the app performs a hit test. It draws a ray from the camera’s position out through the point on the screen you touched and asks the model which surface that ray meets first. If a detected plane is in the way, the point is placed exactly where the ray crosses it. If there is no plane but there are feature points nearby, some apps estimate a surface from those. If there is nothing at all, the tap is rejected or placed at a guess, which is where wildly wrong measurements come from.

Depth from a sensor versus depth from motion

Everything above works with a camera alone, and the depth in the model comes from motion: the same feature seen from two positions. That has two weaknesses. It needs the phone to move before any depth exists, and it needs texture to find features at all.

A depth sensor removes both. LiDAR on Apple’s Pro iPhones and iPads, and time-of-flight sensors on some Android phones, emit infrared light and time its return, producing a direct depth map of the scene from the very first frame, plain walls included. Plane detection can then use measured depth instead of inferred depth, so surfaces appear faster, on more materials, in dimmer light, and a hit test lands on real geometry rather than an estimate. Apple’s RoomPlan, which scans an entire room into a floor plan, is built on this and needs a LiDAR device. Which iPhones and iPads have LiDAR has the list.

On Android, ARCore’s Depth API takes both routes: it uses a time-of-flight sensor where the phone has one, and estimates depth from motion where it does not. Motion-based depth is less accurate than a sensor, but it means plane detection and occlusion work across a much wider range of phones. TritonMeasure relies on LiDAR and RoomPlan on supported Apple devices and on ARCore with Depth on Android, which is why its capture options vary with the phone.

Why the first seconds matter

When you open a measuring app it asks you to move the phone. What it is doing is initialising: the visual system needs to see the scene from more than one position before it can estimate depth, and the fusion between camera and inertial sensors needs a few seconds of motion to settle on the scale of the world. Until that has happened the app is guessing, and any point placed during the guess inherits the error.

The prompt is asking for slow, smooth, sideways movement — a small figure of eight aimed at the floor and the nearest wall is ideal. Fast movement blurs the image and starves the tracker of features; pure rotation without moving the phone gives the camera no parallax and so no depth. Pointing at a textured floor gives it the richest possible start. Give it ten seconds, and you have bought yourself a better session than any amount of care later can recover.

The failure modes, and what they look like

• Lost tracking. A fast swing, a moment aimed at a blank wall or the ceiling, or a hand over the lens. The app usually shows a warning; placed points may freeze or jump. Recover by holding still on a textured area, then remeasure.
• Wrong scale. A hurried start. Every measurement in the session is proportionally off, which a known-size object will reveal immediately.
• Wrong surface. A tap on glass, a mirror or a gap lands on whatever the ray hit behind it. Tap frames and edges, never panes.
• Drift. Two points placed a long walk apart. Split long measurements into overlapping sections from a fixed feature.

Every one of these has a practical countermeasure, and they are collected in tips for accurate phone measurements.

Frequently asked questions

How does a phone measure distance without a ruler?

It tracks its own position in the room by combining what the camera sees with what the motion sensors report, builds a model of the surfaces around it, and places your taps onto that model. The distance between two placed points is the measurement. The motion sensors supply the real-world scale; the camera keeps the position honest.

Why does the app ask me to move my phone around first?

Because the camera cannot judge depth from a single position, and the fusion between camera and motion sensors needs a few seconds of movement to settle on the scale of the scene. Slow, sideways movement aimed at a textured floor gives it what it needs. Points placed before this finishes inherit the error.

Why do measuring apps struggle with white walls?

Camera-only tracking finds surfaces by recognising distinctive points, and a plain wall has none. With no points there is no detected plane for your tap to land on. Keep the floor-to-wall edge or a fixture in view, or use a phone with a LiDAR or time-of-flight sensor, which measures the wall directly.

Why did a placed point suddenly move?

Most often the app recognised a part of the room it had seen before and corrected its map, which shifts placed points by a few millimetres. A large jump means tracking was lost and re-established; remeasure anything placed just before it.

Estimates, not certificates: every phone-based AR measurement is an estimate, TritonMeasure included. Verify anything safety-critical, structural or expensive with a physical tape or laser before you cut, order or sign.