Apple released it’s ARKit3 with a couple of mind-blowing technologies.
Is it really as amazing as it looked during the presentation? What’s in it for developers? How does it affect AR industry in the short run and long-term?
We’ve spent some time here at WeAR studio exploring and discussing the most interesting parts of the topic to find out answers.

Daria Fedko — CEO & Founder
Maria Tintul — Head of Business Development & Marketing
Eugen Fedko — COO

1. Multi-camera tracking


ARKit now supports the use of front and back cameras simultaneously. For example: Interact with the AR scene generated by back camera using face mimics. Enables face tracking along with device orientation and position. Face Tracking tracks up to three faces at once, using the TrueDepth camera on iPhone X, iPhone XS, iPhone XS Max, iPhone XR, and iPad Pro.

E.F.: It’s like a holographic chat in Star Wars!
D.F.: It’s a cool idea to play with avatars. If you can control 3D avatar with your face you can have free hands for other interactions. So many more complex interactions are possible now.
M.T.: You can not only manage content on rear camera, but you can also respond to emotions with the environment, (e.g. if a person smiles and the sun shines).
D.F.: We had such a request from one of our clients. They wanted to put a promo vending machine with Cola at the bus stop. And make a Tetris-like AR game on the screen, where you whould control the behavior of cola can with emotions (rotate, move) in order to make a line. And for each line made in this game, a person would be rewarded with a can of Cola from the machine. But at the time, precision of emotion recognition tech didn’t match the goal so we declined had to reshape idea.
M.T.: It’s a whole new world for fairy tales. It’s a common case when a parent reads a tale for a kid remotely using front camera. But now it’s possible to simultaneously control 3D avatar with mimics (e.g. character of the tale) and also manage the content on a rear camera (e.g. control the environment where this avatar appears). So kid can watch a whole scene in AR multiplayer — with 3D-characters that tells a story in a fantastic environment whichchanges according to the story. Should be super fun

2. Machine Learning plain estimation


Accurate recognition of multiple surfaces

E.F.: Oh, wow!
D.F.: Now it understands floors and walls simultaneously. It allows to map a room and fill it with exactly fitted 3D mashes
E.F.: Now you can paint the real walls. Design the whole interior. Paint floor and walls simultaneously. There is no more switching between vertical and horizontal tracking. This solves the problem with the trial of air-conditioners in combination with other home appliances.
D.F.: And trial of various furniture in one scene. I wonder what about the ceiling. Looks like it is not tracked yet.
M.T.: You can make more realistic occlusion and collision simulations with this. Before the release of ARKit3, it was solved by a few separate SDK’s, which had to be integrated.

3. People occlusion


Hiding and showing objects as people move around in your AR scene. Quite hard to do from scratch, now available in ARKit.


D.F.: Camera now understands the human body and the content can appear as it should.
E.F.: (watching the presentation): quite thin applause. I would be super-excited at this moment if I’d been there!
D.F.: There will be no more stupid moments when, for example, at the conference, we showed a fridge, and everyone who passed by was covered by this fridge. So there will be no «special» points of view anymore. Now simulations will be fine from any point.
M.T.: This demonstrates the overall direction of the evolution of augmented reality as technology. Couple of years ago Disney showed their Magic bench and it was the first notorious project of exploring people occlusion in AR. But in 2017 Disney used a standalone professional depth camera to resolve a task. It’s 2019 and we can see that smartphones are already capable of such operations.

4. Motion capture


Tracks human body in 2D or 3D
Builds skeleton representation
Enables to drive virtual character

D.F.: Motion Capture systems are no longer necessary. This can push boundaries in animation. Now every person with a modern iPhone and properly constructed and rigged 3D model can be an animation studio. It’s also useful in some kinds of try-ons. You still cannot use it for measuring the exact size of human body to try on clothes. But system is already building a correct 3D skeleton, recognizes limbs and head, which is in combination with a future update of deep sensors and current computer vision algorithms can help to build a solution that could map and mesh the body in real time. Which is actually a key to correct clothes try-on.

5. Collaborative sessions


Multi-user experiences
Continuously map sharing between multiple devices
Objects (ARAnchors) sharing between multiple devices
Individual coordinate sharing

E.F.: Now it is possible to put objects in the AR with amazing accuracy. And since everyone understands the tracking of everyone, then the multiplayer compatibility problem is solved and objects can be positioned correctly for all participants. I wonder if it allows to quickly track the city for example? Having this data means that precision of navigation is not a problem anymore. And it finally allows to correctly place a layer of augmented reality perfectly connected to the real world. Games, local experiences, showplaces, art-installations literally anything could be accessible for everyone in exact places as in real world just as in virtual. No more markers.
M.T.: There are so many possibilities. The limitation is actually devices itself. It looks very romantic but now it’s only for A12+ processors. But this is a long-term strategy for Apple, that is needed in order to make developers familiar with the technology that will be widely available in the future.

6. Record and Replay environment


Recording of video and whole sensor data

D.F.: The client may remotely capture the environment, send us a video and we’ll work with it. All experiences that are tied to a specific place are now available for more simple development, galleries, shopping malls, hospitals — everything will be easier.


M.T.: Will ARCore catch the features too? This is an essential question because iPhones are only part of the market and our goal is to introduce the technology to everyone.
At the same time, it opens great opportunities for corporate customers who can use the needed amount of such devices for their business purposes. Sales presentations in widest sense will bloom shortly.

D.F.: And it also seems to me that these technologies aren’t just for iPhones, but for upcoming in 2021 Apple AR-headsets. Therefore, it is not bad with iPhones, but just imagine what these solutions could do with voice-guided glasses, which understand gestures, this is a completely different level of Augmented Reality. And, by 2021, the software part will be already widely tested by the developers.

E.F.: Many things that have been shown, look fantastic, not everything is quite suitable for the phones right now, but with the transition to a new device — their finest hour will come. Apple collects a huge amount of data for their machine learning systems — the accuracy and efficiency of these systems will increase significantly in the nearest future. And as we see here — machine learning is one of the main drivers of AR evolution. The next release, I’m sure, will impress us even more than the current one.

Subscribe now!
We won’t bother with the daily newsletter! Only important information once a month.