BETA
This is a BETA experience. You may opt-out by clicking here

More From Forbes

Edit Story

The Search Engine Of AR

Following
POST WRITTEN BY
Ori Inbar
This article is more than 6 years old.

The AR Cloud is to AR what search is to the Internet. I remember the Internet without search. Remember Prodigy? CompuServe? Search was perusing a directory. In 1993 we thought hypertext changed everything. That was just the beginning. That was the Internet without search. AR is very much in that position today. To move from enterprise to consumers, it needs a killer app. In "Why We Need An AR Cloud" below, Ori Inbar explains why there will be few killer apps for consumers without the AR Cloud. Tomorrow, in part two of this chapter, “Bigger Than Search”, Matt Miesnieks predicts the company that provides this solution will be more valuable than Apple, whose $870 billion market cap makes it the most valuable company in the world. - CF

Why We Need An AR Cloud

By Ori Inbar

With the release of Apple’s ARKit and Google’s ARCore , creating augmented reality (AR) apps became a commodity overnight. It’s free, it’s cool, and it just works. So far, the “mass adoption” is mostly among AR developers, and its generating tons of YouTube views. But developers are yet to prove their apps can break through the first batch of novelty apps and gain mass user adoption.

For sure, there will be a couple of mega-hits (e.g. Pokémon Go) that will enjoy the newly made mega distribution channel for AR apps on iPhones and on high end Android phones. But, I do not anticipate hundreds of millions using AR apps all day, every day. ARKit and ARCore based AR apps are like surfing the web with no friends. It’s so 1996.

The ARKit, released in September, 2017, is the best thing that happened to the AR industry. In just a couple of days, Apple’s AR enabling new iOS 11 was on hundreds of millions of phones. But massive adoption of AR apps will take much more than that. It will happen with the AR Cloud, when AR experiences persist in the real world across space, time, and devices.

Re'flect

I was hoping the iPhone X will introduce a key ingredient for massive AR adoption , not the ARKit . But rather, a back facing depth camera that will put in the hands of tens of millions a camera that senses the shapes of your surroundings, and can create a rich, and accurate 3D map of the world to be shared in the AR Cloud. That hasn’t happened yet. Hopefully it will in the next version. This will only slow down the creation of the AR Cloud , but it cannot stop the demand for it.

Apple

A real time 3D (or spatial) map of the world, the AR Cloud, will be the single most important software infrastructure in computing. Far more valuable than Facebook’s social graph or Google’s PageRank index. In a nutshell, with the AR Cloud, the entire world becomes a shared spatial screen, enabling multi-user engagement and collaboration.

The AR Cloud is a shared memory of the physical world and will enable users to have shared experiences, not just shared videos or messages. It will allow people to collaborate in play, design, study, or team up to problem solve anything in the real world. Multi-user engagement is a big part of the AR Cloud. But an even bigger promise lies in the persistence of information in the real world.

We are on the verge of a fundamental shift in the way information is organized. Today, most of the world’s information is organized in digital documents, videos, and information snippets stored on a server, and ubiquitously accessible on the net. But it requires some form of search or discovery.

Based on recent Google stats, over 50% of searches are done on the move (searched locally). There is a growing need to find information right there where you need it, in the now. The AR Cloud will serve as a soft 3D copy of the world and allow you to reorganize information at its origin, in the physical world (or as scientists call it in Latin: in situ). With the AR Cloud, the how to use of every object, the history of any place, the background of any person, will be found right there, on the thing itself. And whoever controls that AR Cloud could control how the world’s information is organized and accessed.

When Will This Happen?

The AR Cloud is not for the faint of heart. Industry leaders are imploring developers, professionals and consumers alike to have patience as it’ll take a while for this to materialize. But as investors in AR and frontier technology, this is the time to identify the potential winners of this long distance race.

In the past decade, several companies have been providing AR services from the cloud, starting with Wikitude as early as 2008. Then Layar, Metaio (Junaio), and later: Vuforia, Blippar, Catchoom, among new entrants. But these cloud services are typically one of two types: (1) Storing GPS or location related information (for displaying a message bubble in a restaurant) or (2) Providing image recognition services in the cloud to trigger AR experiences.

These cloud services have no understanding of the actual scene and the geometry of the physical environment. And without it, it’s tricky to blend virtual content with the real world in a believable way, let alone share the actual experience (not just a cool video) with others.

Imagine Pokémon Go With The Cloud

The incredible reach of Niantic’s Pokémon Go was a fluke. An outlier. A one of a kind. It will be super hard, if not impossible, to replicate its success with similar game mechanics. Though, we are about to find out: Niantic is preparing to launch a similar massive AR game based on the Harry Potter mega franchise in the first half of 2018.  For Pokémon Go, game servers store geolocation information, hyperlocal imagery, and players’ activity. but not a shared memory of the physical places in which its 65 million monthly active users are playing. Thus, no real shared experience can occur. For that it would need the AR Cloud.

Several scientists have conceived different aspects of the AR Cloud since the 1990’s, but I have yet to see a concise description for the rest of us. So here is my simplified version in the context of the AR Industry.

An AR Cloud System Should Include:

  • A Persistent Point Cloud

A point cloud as defined in wikipedia is “a set of data points in some coordinate system (x,y,z),” and by now is a pretty common technique for 3D mapping, reconstruction, surveying, inspection, and other industrial and military uses. Capturing a point cloud from the physical world is a “solved problem” in engineers’ jargon. Dozens of hardware and software solutions have been around for a while for creating and processing point clouds using active laser scanners such as LiDAR. Depth or stereo cameras such as Kinect, and monocular camera photos, drone footage or satellite imagery processed with photogrammetry algorithms. And even using synthetic aperture radar systems (radio waves) such as Vayyar or space borne radar.

For perspective, photogrammetry was invented right along with photography . So, the first point cloud was conceived in the 19th century. To ensure it has the widest coverage and always keeps a fresh copy of the world, a persistent point cloud requires another level of complexity: The cloud database needs to have a mechanism to capture and store a unified set of point clouds fed from various sources (including mobile devices) and its data needs to be accessible for many users in real time.

The solution may use the best native scanning or tracking mechanism on a given device (ARKit, ARCore, Tango, Zed, Occipital, etc.), but must also store the point cloud data in a cross platform accessible database.

The motivation for users to share their personal point clouds will be similar to the motivation Waze users have: Receive a valuable service (optimal navigation directions), while in the background sharing information collected on your device (your speed at any given road segment) to improve the service for other users (update timing for other drivers).

Nevertheless, this could pose serious security and privacy concerns for AR mapping services, since the environments mapped would be much more intimate and private than your average public road or highway. I see a great opportunity for crypto-point-cloud startups.

Google calls it “Area Learning . Which gives the device the ability to see and remember the key visual features of a physical space ( the edges, corners, other unique feature) , so it can recognize that area again later.” The AR Cloud’s requirements may vary indoors and outdoors, or whether it’s used for enterprise or consumer use cases. But the most fundamental need is for fast localization.

  • Instant Localization From Anywhere

Localizing means estimating the camera pose which is the foundation for any AR experience. In order for an AR app to position virtual content in a real world scene, the way it was intended by its creator, the device needs to understand the position of its camera’s point of view relative to the geometry of the environment itself.

To find its position in the world, the device needs to compare these key feature points in the camera view with points in the AR Cloud and find a match. To achieve instant localization, an AR Cloud localizer must narrow down the search based on the direction of the device and its GPS location (as well as leveraging triangulation from other signals such as wifi, cellular, etc.). The search could be further optimized by using more context data and AI.

Once a device is localized, ARKit or ARCore, using the inertial measurement unit (IMU) and computer vision, can take over and perform the stable tracking. As mentioned before, that problem is already solved and commoditized.

Dozens of solutions exist today to localize a device such as Google Tango, Occipital Sensor, and of course ARKit and ARCore. But these out of the box solutions can only localize against a local point cloud, one at a time. Microsoft HoloLens can localize against a set of point clouds created on said device. But, it can’t localize against point clouds created by other devices.

The search is on for the “ultimate localizer” that can localize against a vast set of local point clouds from any given angle and can share the point cloud with multiple cross platform devices. But any localizer will only be as good as its persistent point cloud, which is a great motivation to try and build both.

  • Place and Visualize Virtual Content in 3D

Once you have a persistent point cloud and the ultimate localizer, the next requirement to complete an AR Cloud system is the ability to position and visualize virtual content registered in 3D. “Registered in 3D” is the technical jargon for “aligned in the real world as if it’s really there”. The virtual content needs to be interactive so that multiple users holding different devices can observe the same slice of the real world from various angles and interact with the same content in realtime.

The AR Cloud landscape spells the biggest opportunity in AR to grow a Google-size startup since Magic Leap. Speaking of Magic Leap. It used to have the ambition to build the AR Cloud. Do they still have it? Should we care?

Tomorrow, Matt Miesnieks "Bigger Than Search" will follows up on the challenges of building the AR Cloud. 

The articles are part of my upcoming AR Enabled book, Charlie Fink's Metaverse which features contributions from thought leaders, executives, entrepreneurs, and artists

Cool Blue Press

 

Follow me on Twitter or LinkedInCheck out my website or some of my other work here