Earlier this month, at Apple’s annual developer conference, WWDC 2022, the company gave developers a first look at improvements to Apple’s ARKit 6 toolkit for building AR apps on iOS devices.
While Apple has yet to reveal (or even confirm) the existence of an AR headset, the clearest indication that the company takes AR absolutely seriously is ARKit, the developer’s toolkit for building AR apps on iOS devices. Apple has been developing since 2017.
At WWDC 2022, Apple unveiled its latest version, ARKit 6, which brings improvements to core capabilities so developers can build better AR apps for iPhones and iPads (and eventually headsets…probably).
During the ‘Discover ARKit 6’ Developer Session at WWDC 2022Apple ARKit engineer Christian Lipski, an overview of what’s next.
Better motion recording
ARKit includes a MotionCapture feature that tracks people in the video frame, giving developers a “skeleton” that estimates the position of the person’s head and limbs. This allows developers to create apps that overlay augmented things over the person, or move them relative to the person (it can also be used for occlusion to place augmented content behind someone to embed it more realistically into the scene).
In ARKit 6, Lipski says the feature is getting a “whole series of updates,” including improved tracking of 2D skeletons that now estimate the location of the subject’s left and right ears (which will certainly be useful for face filters, fitting a glasses with AR and similar functions involving the head).
As for 3D skeletons, which provide a pose estimate with depth, Apple promises better tracking with less jitter, more temporal consistency, and more robustness when the user is closed off by the edge of the camera or other objects (although some of these improvements are only available on iPhone 12 and later).
Camera access improvements
ARKit 6 gives developers much more control over the device’s camera while using it with an AR app for tracking.
Developers can now access incoming frames in real-time up to 4K at 30FPS on the iPhone 11 and later, and the latest iPad Pro (M1). The earlier mode, which uses a lower resolution but a higher frame rate (60FPS), is still available to developers. Lipski says developers should carefully consider which mode to use. The 4K mode might be better for apps geared towards watching or recording video (like a virtual production app), but the lower resolution 60FPS mode might be better for apps that take advantage of responsiveness, like games.
As with higher video resolution during an AR app, developers can now capture full-resolution photos even if an AR app is actively using the camera. That means they can pick out a 12 MP image (on an iPhone 13 anyway) to save or use elsewhere. This could be great for an AR app where taking pictures is part of the experience. For example, says Lipski, an app that guides users through taking pictures of an object to later convert into a 3D model using photogrammetry.
ARKit 6 also gives developers more control over the camera while it is being used by an AR app. Developers can adjust things like white balance, brightness, and focus as needed, and read EXIF data from each incoming frame.
More Location Anchor… Locations
ARKit includes LocationAnchors that can provide street-level AR tracking in selected cities (for example, to provide augmented reality turn-by-turn directions). Apple is expanding this functionality to more cities, including now Vancouver, Toronto and Montreal in Canada; Fukuoka, Hiroshima, Osaka, Kyoto, Nagoya, Yokohama and Tokyo in Japan; and Singapore.
Later this year, the feature will expand further to Auckland, New Zealand; Tel Aviv-Yafo, Israel; and Paris, France.
Plane Anchors are a tool for tracking flat objects such as tables, floors, and walls during an AR session. Prior to ARKit 6, an aircraft anchor’s origin would be updated as more of the aircraft was discovered (for example, moving the device to reveal more of a table than the camera previously saw). This can make it difficult to hold magnified objects in place on a plane if the origin was rotated after they were first placed. With ARKit 6, the rotation of the origin remains static no matter how the shape of the plane changes during the session.
ARKit 6 launches with the iOS 16 update which is now available to developers as beta and is expected to be released to the public this fall.