Digital humans might soon be guiding you around the world.
Picture this. You walk in, wide eyed, into the Vatican Museum. A tour guide dressed like Michelangelo greets you at the entrance and after a quick introduction, begins walking you through the galleries, explaining, in detail, the history of Vatican city and the great renaissance artists housed inside. He’s funny, charming and knows the answer to pretty much any question you have. He’s also not real… He’s a hologram.
Let’s teleport to another scene. You’re in the market for a new home and decided to stop by at a nearby open house. Upon reaching the house, you’re greeted by a realtor who asks you what you’re looking for in an ideal home and then walks you through the house giving you a complete overview of the features and history of the house. She’s helpful, knowledgeable and as you might have guessed, also a hologram.
This is the dream that got us most excited about augmented reality (AR) when we began our journey, at Placenote, with a guided tour of an office we built in early 2017. The idea was simple — AR lets us re-imagine how people interact with not just the digital but also the physical world and this fusion of digital and physical is what would guide the next big paradigm shift in computing.
Over the last year we’ve seen the AR community begin to connect the dots to this idea in a remarkable way. With this post, I wanted to shed some light on this trend, highlight interesting products built around it, and list the tools available to you, as a developer, if you’re interested in building something like this today.
First, to define the concept.
In the future, hologram tour guides in augmented reality might be the user interface through which we explore the world around us.
The keywords here are “user-interface” and “explore”. Let me explain why I think this is a big deal in the next 3 sections.
1. Augmented reality as a UI for navigation
As developers, we tend to think of our apps and websites as independent entities from the real world. You’ve probably heard the saying, “the UI is the product”, because it has almost always been true. So far, users have primarily interacted with our software through rectangular screens on their desks and in their pockets.
In the AR world, this completely changes. No longer is your app limited to a window. You can now create digital content all around your users and essentially “immerse” them in your app. It’s an understatement to say this requires a complete rethinking of our design conventions from the 2D world.
The second thing that changes in AR is that the world can now be integrated into your product. Ever since GPS showed up on mobile phones, we’ve had location-based apps that use maps as a canvas to display information, trigger actions and help us navigate. Great examples are Google maps, Foursquare, Uber and Yelp. In AR, however, the physical world itself can be used as a canvas for your UI. So what does this mean for location-based apps?
The GOOD — AR lets us leverage visual cues in 3D
When searching for directions on Google Maps, you’ve probably had to look up and down at your phone a few times to try to match the points of interest on your map with their real world locations. In AR you can highlight a point of interest directly in the real world to give users a much quicker way to understand contextual information. Here’s an example of how 3D visual cues make navigation a more intuitive.
The BAD — AR interfaces run the risk of overwhelming users.
The downside of AR for location-based apps is that you run the risk of overwhelming your users with “too many” visual cues. That means users have to work 10 times harder to search your maps for information that could pop up all around them. You definitely don’t want to end up like this:
That makes one think…how can augmented reality UI’s be designed to effectively get out of the way of people accomplishing a task — which in the case of location-based apps, is either searching, navigating or exploring the real world. AR forces you to constrain your UI to simple, limited interactions closely based on the user’s immediate context.
2. The BIG difference between navigation and exploration
The standard model of navigation apps has always been a very utilitarian experience with one goal — getting people from point A to point B. But think about the last time you used Google Maps as a sight-seeing tool. You first make a list of things to do, learn about each of them, string them together in an itinerary and then use Google Maps purely for the mechanical task of finding directions.
The new Google Trip Planner tries to solve this problem by providing better information and curated itineraries but the utilitarian experience persists here as well.
In my opinion, the killer feature of AR is enhancing a traditionally utilitarian experience with richer storytelling. If you’ve ever been on a walking tour of a city or a museum with an enthusiastic and insanely knowledgeable tour guide, you will understand why walking tours are a 16 billion dollar industry worldwide.
Walking tours are a 16 billion dollar industry worldwide.
With AR, finding your way doesn’t need to be a mechanical task anymore — It can be fun, educational and exploratory. Your digital tour guide could craft an itinerary for you, based on what it knows about you and then create a rich guided experience that lets you enjoy the best things a place has to offer in a simple, meaningful way.
3. Augmented reality as a digital assistant.
The concept of digital assistants is not new. There was a time when using a computer for the first time was an exploratory task for most people. If you used a computer in the 90s, you probably remember this little guy.
More recently, we’ve seen the rise of chatbots like Facebook messenger bots as well as voice assistants like Siri, Google Home and Amazon Alexa fighting to answer our questions with sassy AI bots.
These bots are incredibly useful but so far, we’ve only communicated with them through text. Even voice-enabled bots like Alexa don’t really hear us. Rather they read a transcription of our words to understand us. This is an important point because AR could lets us build bots that can communicate across a much larger range of channels — voice, body language, facial expressions and physical movement. I’m talking about importing the best parts of human face-to-face conversations into digital interactions with computers.
AR can let users have a face-to-face conversation with a bot.
The point I’m making here is that there is a strong need for guided location-based experiences that isn’t being met by navigation apps available today. Experiences that rely on exploration need a layer of storytelling with context and personality that can only be solved by (1) an intelligent guide and (2) deep knowledge of a users location and context.
AR can bridge the gap between the utility of navigation and the fun of exploration.
In the next sections, I’ll show you how to build an AR digital assistant and what the current progress in this space has been so far.
How to build an AR tour guide
Building an AR tour guide app essentially boils down to three main components:
- Character design
- Story design
- Location awareness
Let’s discuss each of these components and explore the latest tools you can leverage to build something like this today.
Character Design (Animation and AI)
One of the most important pieces of an augmented reality guided tour is the guide itself. This is a 3D animated character that will be the face of your digital assistant. Building animated characters is a well established field in the film and gaming industries. Characters are designed by artists, rigged to define facial features, joints and movements and then animated to match the script with careful timing. Watch the making of any Pixar movie and you’ll see what a tedious process this is.
Fortunately for developers, tools like Unity have pre-built animations that can be attached to any rigged character model for basic movements like walking, dancing, waving, frowning etc. But building more nuanced and custom animations has been quite inaccessible to the average mobile developer so far. This is changing now with startups building new and easier ways to create animated characters for VR and AR. Here are some examples:
Other companies like Mind Show and Limitless are making it easy to animate 3D cartoon characters, rather than real humans, using clever ways to create interesting characters for immersive environments.
Character AI is the second most important piece of character design. Digital assistants should be able to interact with users and respond to questions. In fact Amazon has built a product called Amazon Sumerian that uses the same AI engine that powers Alexa to create animated digital assistants for VR and AR apps.
Perhaps, alternatively, we might even be able to use something like Facebook’s Social VR to create live telepresence sessions with animated characters embodying real tour guides remotely guiding people in AR.
Story design encompasses everything that defines the “story” or the sequence of actions in an AR experience. In guided tours that basically means the sequence of tour stops, the story path alternatives and scripted content that will be played along the tour.
This part of the stack is independent of the AR interface itself and, as such, has existed in many forms like audio tours in museums for several years. There are a few startups building products that make it easy to design simple guided tour experiences on mobile and they each have built their own unique story design tools. Some neat examples are GuidiGo, Detour, Guide By Cell and MetaVerse.
The final piece of the guided tour puzzle is “location-awareness”. This has traditionally been done in the 2D world with GPS. GPS works well enough for 2D location-based apps because the accuracy needed for 2D maps is pretty low (3m–5m).
However, in AR, GPS doesn’t really cut it because rendering visual cues over the physical world in 3D requires a very accurate and responsive location-awareness system. Further, GPS doesn’t work indoors.
This a good segue for a quick note about my work with Placenote SDK.
Our team at Placenote is building an SDK to bring “visual location awareness” to AR apps. It works by letting you create visual 3D maps of outdoor and indoor locations and use them to create highly accurate location aware apps like guided tours.
The image below shows how you can use Placenote SDK to visually map and define points of interest in your guided tour, effectively replacing GPS as a location-awareness solution.
A combination of these three components (Character design, Story design and Location awareness) could be put together in creative ways to build incredibly compelling guided tour experiences that give users a fun, educational and engaging way to explore the world.
I think this is absolutely a killer app for AR and I would love to see more developers start experimenting with character led tours.
If you’ve still reading until this point, Thank You! If you’re interested in learning more about what we’re building at Placenote,
Also, if you’ve seen any interesting examples of virtual tour guides, please do share them with me! Thanks!