Hands-On: Facebook Pioneers Augmented Reality for the TV with Latest Portal Hardware

When Facebook launched its first hardware products last year, the Portal and Portal+ smart displays, the company mostly touted its video-calling features as it faced off against Amazon and its Echo Show.

But Facebook also had an interesting feature that mostly flew under the radar. The Portal devices were also capable of augmented reality, with Spark AR, the platform responsible for mobile augmented reality experiences on Facebook, Messenger, and Instagram, supplying the virtual effects.

• Don't Miss: Facebook Searching for More Troops to Work at Oculus on AR Product Design & Marketing

Roughly a year later, with a former Google engineer leading the Portal team, Facebook has expanded the Portal family.

In addition to last year's Portal+ (now $279), Facebook now offers a redesigned Portal ($179), which now resembling a digital photo frame instead of a Google/Nest Home Hub, the similarly-designed-just-smaller Portal Mini ($129), and a brand new form factor, the Portal TV ($149).

The devices are available for purchase via the Portal website, Amazon, and Best Buy in US and Canada. Customers who opt to buy any two devices will enjoy a $50 discount on their purchase. Facebook has also expanded its sales territory for the Portal collection of devices, with customers in the UK, France, Italy, Spain, Australia, and New Zealand now able to purchase the hardware.

For our purposes here, we are focusing on Portal TV, because it achieves a rather unique milestone: it brings augmented reality to your television. And that's the unit that we asked Facebook to send to use for a hands-on exploration.

Hardware Specs

Measuring in at 7.48 inches by 1.18 inches by 2.24 inches and weighing in at 0.68 pounds, the Portal TV device comes in any color, as long as it's black.

Portal TV's central hardware feature is its camera, a 12.5 MP sensor with 120 degrees field of view. The device also includes a full-range speaker for audio output and far-field array of eight omnidirectional microphones for voice input. As one would assume, it has an HDMI port to connect to a TV for video output (HDMI cable not included).

In terms of accessories, the Portal TV comes with a compact remote controller with a directional navigation pad and a handful of input buttons.

In the Portal TV's default configuration, it sits on a short pedestal for users who prefer to sit it below their television. For those who wish to mount the device on top of their television, the pedestal folds out to anchor the Portal TV to the rear of the television, and a shorter brace folds out from underneath the device to attach to the front of the television's top bezel.

Finally, for peace of mind on the privacy front, Portal TV has a shutter door for obscuring the camera, and a button to deactivate the camera and mute the microphone, along with a red LED to confirm that comms are dark. (There are also some privacy features on the software side that we'll get into later on.)

Software Features

Before we jump into the AR portion, Portal TV offers a slew of amenities that appeal to the average consumer.

First and foremost, there's video calling, which is where the augmented reality experiences come into play. One of the unique features of Portal's camera, with an assist from AI, are automatic pans and zooms as callers move through their space to keep them in frame.

Portal TV supports video calls via Messenger and WhatsApp, enabling Portal users to contact friends and family even if they don't have their own Portal device. English-speaking users in US and Canada can use the "Hey Portal" key phrase for hands-free video calling.

• Don't Miss: Facebook Brings Its AR Camera Effects to Instagram & Messenger Apps

The Portal TV also acts as a multimedia device, with a limited selection of media apps such as Facebook Watch, which offers a feature called Watch Together that enables video calls that allow users to veg out remotely together while watching a video. Other apps include Amazon Prime Video, Showtime, CBS All Access, Starz, Pluto TV, Red Bull TV, and Neverthink for video and Spotify, Pandora, iHeartRadio, and iHeartRadio Family for audio to take advantage of that full-range speaker. While the selection is relatively small, Facebook says more apps are on the way.

In addition, the Portal TV integrates Amazon Alexa for access to all of the assistant's smart skills and packs an app called Superframe for displaying photos, birthday reminders, and online friends.

But this is Next Reality, and we're here to talk about one thing: augmented reality.

Spark AR on Portal

Portal has four augmented reality categories for users to choose from, one that is familiar to Facebook Messenger users, two that are unique to Portal, and another that is still in the oven.

First, the Portal TV offers the same AR camera effects that are available on the Messenger mobile app. These AR effects work on calls between Portal users as well as in calls between Portal and mobile Messenger users.

Second, there's Storytime, which works for Portal users calling other Portal users or Messenger users on mobile or desktop. As the name implies, the feature assists users in telling children's stories.

Storytime works much like a traditional book, with virtual frames visible to the audience and captions visible to the reader. Readers advance each "page" with the handheld remote. As the story progresses, Portal adds virtual masks to the reader to further immerse the reader as a character in the story from the audience's point of view.

At launch last year, Portal's Storytime library consisted of just six stories. According to a Facebook spokesperson, the library will expand to include 20 stories by the end of the year.

• Don't Miss: Facebook Notches Win over Snapchat with AR Gaming for Messenger Video

The third AR feature, available on Portal to Portal calls only, is called Shared Effects, but it really could be called Portal Games. These four AR experiences are designed to take advantage of the hands-free nature of the Portal experience. I had the opportunity to playtest these games in a briefing with Facebook representatives.

One game overlays an image of a cat on each caller's face, and each player must move their head within the camera view to catch the matching cat face as it falls into view from the top of the screen. I misunderstood the rules in the first round, thinking that the red beams of light tracked to my face meant I won. Knowing the rules the second time around, I still lost due to a lack of dexterity.

Another game uses face tracking to create a smiling contest, where whoever smiles last wins. I won.

The next game was cooperative rather than competitive. This one places bunny ears on the users' head and each player takes turns catching as many virtual doughnuts as they can within their allotted time.

Finally, the Emoji game acts as an icebreaker game, enabling callers to get to know each other. Each round of the game presents the participants with a pair of emojis. Players select their favored emoji by leaning their head in the direction of the emoji.

If there's one drawback to these games, it comes in situations where there's more than one person in the camera view. The Portal camera has the tendency to lose track of which face it should focus on. But, hey, this isn't professional e-sports we're playing here.

Finally, a new Portal AR experience launching later this year is called Mic Drop. This experience dares users to lipsynch to songs while wearing AR costumes, with callers taking turns performing. I'll have to admit that I'd like to revisit Portal TV for this feature.

Because the AR effects are built on Spark AR, the potential for independent developers to create their own AR experiences for Portal are within the realm of possibility. According to a Facebook spokesperson, the company is currently working with select partners for new Storytime and Shared Effects experiences and is exploring how to open the platform up in the future.

Portal Privacy

So, there's an elephant in the room when it comes to Facebook, and its name is privacy. The company's struggles over the past few years are well-documented, and they cast a shadow over this product.

But Facebook maintains that the device has been designed with privacy in mind. Here's what the company has to say about privacy on Portal:

Portal has clear and simple settings for privacy and security. You can disable the camera and microphone with a single tap or a sliding switch. A red light next to the lens indicates the camera and microphone are off and there's an integrated camera cover if you want to physically block the camera lens.

For added security, Smart Camera and Smart Sound use AI technology that runs locally on Portal, not on Facebook servers.

If you have "Hey Portal" enabled, Portal listens for the phrase "Hey Portal." If it's detected, Portal sends a short audio recording and transcript of the "Hey Portal" voice interaction to Facebook. A trained team may review a sample to make our voice services smarter and more accurate for everyone. You can view, hear and delete any of your "Hey Portal" voice interactions in your Facebook Activity Log. You can also turn off voice storage in Settings anytime, which means that your voice interactions are not stored or reviewed. To learn more about Portal's privacy features, visit portal.facebook.com/privacy.

In addition, a Facebook spokesperson confirms that any interactions with Alexa are only sent to Amazon's servers.

How does this information impact consumers' outlook on the device? Some skepticism is warranted. Between my sessions with the device, a bug discovered on the iOS version of the Facebook app allowed the app to access the iPhone or iPad's camera. With privacy concerns enveloping the company, this is a hard accident to merely overlook.

Then again, the average consumer seems to be unphased by Facebook's track record. The social network continues to grow its daily active user base, rather than see it erode.

Conclusion

During a briefing, Facebook product manager Kelly Zhou emphasized that the overarching theme of the AR features, and the Portal platform as a whole, is building connection.

This is evident in the selection of AR experiences. There's no capture button for sending photos in direct messages or sharing in stories. These are meant to be real-time experiences in live conversations. As a communications tool, the Portal TV and its AR features succeed.

But, privacy fears aside, the Portal TV seems like niche device for fringe audiences, like grandparents and their grandchildren. To illustrate my point, in a conversation with a millennial friend, I shared that I was testing out a Portal TV. When asked what it was, I explained that it is a connected camera that attaches to a TV for making video calls.

"You mean like I can already do with my phone?" was the response.

This is a small sample, sure. But a 2016 report revealed that 52% of smartphone users ages 18 to 34 used smartphones for video calling. And that was three years ago.

Nonetheless, Portal TV is breaking new ground as a fun AR communications device for televisions. And it may just be the gateway to AR for our oldest and youngest generations, carefree enough to throw caution to the wind in order to connect.