An AR experience toolkit on top of Flutter and Unity

#The division of labour

Augmented reality on a phone needs three things at once: camera tracking, three-dimensional rendering registered to the real world, and an interface the user can operate. The first two belong in an engine. The third does not, and building it in an engine is how AR applications end up with buttons that ignore the platform's accessibility settings.

The toolkit puts Unity in charge of ARKit and ARCore session management, plane detection, anchors and rendering, and leaves everything else to Flutter. The camera feed and the rendered content are one surface. The controls, the onboarding, the permission prompts and the sharing sheet are ordinary Flutter widgets composited over the top.

#Why that split is not obvious

The tempting alternative is to build the entire experience in Unity, since the engine already has a UI system. Every AR project I have seen do that has ended up rewriting native behaviour badly: a keyboard that does not match the system keyboard, a scroll view with the wrong physics, a permission dialogue that is not the operating system's.

Registration between virtual and real content is the hard technical problem in AR and has been studied for a long time [2] [3]. Interface conventions are a solved problem on both mobile platforms. Mixing the two into one runtime means solving the easy problem again, badly, while the hard one is still hard.

#Session lifecycle

An AR session holds the camera. That makes lifecycle handling stricter than it is for a plain 3D scene. If the user takes a call, the session must pause and release the camera, and on resume the tracking state may or may not be recoverable depending on how long the interruption lasted and whether the device moved.

The toolkit surfaces tracking state to Flutter as an explicit value rather than hiding it, so the interface can say that tracking is lost instead of leaving the user looking at a frozen scene. Hiding that state was my first attempt, and the resulting bug reports were all variants of "it stopped working".

#Anchors and where state lives

An anchor is a pose the tracking system promises to keep updated as its understanding of the room improves. That means anchor positions change after you place them, sometimes noticeably. Any application state that duplicates an anchor's position will drift out of agreement with the session.

The rule that came out of this is that Unity owns spatial state and Flutter owns application state. Flutter may ask for an anchor's current pose, and it never stores one. Everything that broke in early builds broke because I had cached a position on the Dart side.

#The longer write-up

I wrote this up in more detail for Potato, covering the project structure, the build configuration for both platforms, and the parts of the Unity export that need attention [1]. The demo repository shows a complete ARKit scene driven through the same path, which turned out to answer more questions than any amount of prose.