14-December-2019 By Jeffrey Cooper

Immersive Spaces: UX In The Real World (Part 2)

In the previous article, I laid out my thinking process towards approaching most problems and concepts. In this second foundation article, we’ll dive one level deeper.

For immersive solutions that involve the interaction of components, I follow an approach such that the usage of the system is as evident as possible and uncomplicated. These systems can be with or without human interaction, but for the articles following this one, I’m going to consider human actors at the center of it.

Philosophy

While I paint the following scenarios as fully-realized, immersive systems, it is important to note that when I develop these concepts, I attempt to do so with modularity in mind, so you don’t have to implement everything at once to get results. In the articles following this, I will point out where elements can be layered in piecemeal, each adding additional value once it’s implemented. Ideally, a minimum first step can be implemented that will provide value to the business, either in the form of valuable data (new products or incremental revenue), an improvement in efficiency of operations, or both.

Immersive Spaces

What do I mean by Immersive Spaces? Simply put, I see the ultimate User’s Experience (UX) taking place in the world itself, in the space that we inhabit in a given situation. It can be your place of employment, your home, a shopping mall, a gym, a vehicle or even a city. An individual interacts with an array of devices (1:many) and the building or space interacts with many people at the same time (many:many), which I usually refer to as aggregate.

The diagram above is a shopping mall and shows various networks both meshed and interacting with the shopper. In these interactive environments, there are usually many inputs and many potential outputs, functioning in a coordinated way to provide an enhanced experience that is intuitive. I like to express it as the system “Just Is” and “Just Does.”

Please keep in mind that UX is not UI. It is, in part, a collection of UIs and the experiences between them. It contains the totality of a user’s experience within a given system.

For example, one’s Smartphone has a UI. A Smart Watch has a UI. But consider also that in a building, a collection of sensors (beacons for location, motion detection, accelerometers, etc…) also serve as inputs. And an output can be to a phone display, or to lighting, elevators or environmental controls such as heating and white noise, as an example. Collectively, all these elements working together mold the UX.

Let’s use Work as an example- you walk in the front door, and the mere presence of your phone with you identifies you (which is enhanced if you are wearing a Smart Watch). Security gates unlock (secondary security features can be included for sensitive locations). You walk up to the elevator, and your calendar shows you have a meeting on the 6th floor in 2 minutes, so even though you sit on the 9th floor. The elevator defaults to the 6th floor, since the meeting is about to start.

This is a simple example of an immersive use case. For default (presumed) actions, there is no action required by the user. If the user knew the meeting was canceled, for example, and it had not been removed from their calendar, they can choose to override the default, and hence choose to use a UI to override it, but in normal situations, the best UX contains automated choices with no direct user interaction with any of the associated UI elements (phone screen, elevator buttons, etc…)

The Elements of an Immersive UX in the Real World

Note: I use the term Immersive UX frequently to apply to the UX within three-dimensional physical spaces. Note that the term can also apply to virtual reality environments, which are also immersive in a different way. Just to clarify- nothing in this article refers to VR.

Within an Immersive UX, there are several elements:

• Nodes– these are intermediary devices, or hybrid devices (see below) that relay information within a system. These may be radio elements such as WiFi and Bluetooth, phones, watches, touch displays

• Endpoints– these are input devices such as sensors and sensor meshes, microphones, pieces of equipment, and output devices, such as lighting, ventilation, speakers, a display or phone displaying information

• Hybrid devices– these serve as both types of devices. A Smartphone, for example, can provide input to a system, while also interacting with a user or passing information along to an accessory, such as a Smart Watch, which itself can server as both input and output.

Five Tenets of Immersive UX’s

There are five basic tenets I have about Immersive UX’s:

• Fusion of Data Sources– simply put, this is the entirety of relevant sensor information for a given scenario, which can also include voice. Combining locally-acquired data gives context (who, what, where and when) that enrich the experience.

• Incorporation of Big Data– this will vary by scenario and may or may not be needed, depending on your needs. For example, a shopping mall might want to bring in external data sets that indicates weather conditions- current and predicted, traffic patterns, and current popularity trends. These add value to current context.

• Modularity– While I paint the scenarios in the following articles as fully-realized, immersive systems, it is important to note that when I develop these concepts, I attempt to do so with modularity in mind, so you don’t have to implement everything- a big undertaking, to get results. In the articles following this, I will point out where elements can be layered in piecemeal, each adding additional value once it’s implemented.

• Security is essential– this goes without saying, but a mesh of connected devices can allow for more openings for bad actors to access. In home networks, we hear stories of WiFi networks with default login credentials still in place and baby monitors being hijacked. And in corporate scenarios, breaches have happened by a number of different vectors.

• Privacy is essential– depending on your relationship with the user, data needs to be handled accordingly. While an employer may own records of employee attendance, for example, it must be kept private within the company. In other scenarios, such as retail or fitness, user behavior, while essential for providing a personal experience, should not be shared between them without consent. Collectively, the organization can use private data to offer new products or enhanced service offerings to the user, and, depersonalized, can use the data in aggregate to improve overall operational efficiencies.

The System

The total system consists of three main components- the Processing System in which the application and data resides, the Local Immersive Environment, or space, where the user is going to interact, and the User themselves.

Processing System: The Processing System is just the collection of servers and data sources, local and in the cloud, where the data processing and decisions occur.

Local Immersive Environment: Within the immersive environment, there are many potential components, in two major groups:

• Informational- those components that provide information to the system, such as sensors, NFC, Beacons, BLE, WiFi (virtual beacons for example)
• Reactive- those components such as lighting, environmental (heating, white noise, etc), access, informational displays

User: The User’s components can consist of:

• Phones & Watches- Computing and informational elements
• Sensors- most frequently these are contained on phones and watches, and consist of accelerometers, Heart Rate, GPS, NFC and other radios, or other sensors that use the phone or watch as their connection and don’t talk directly to the larger system. In the future additional devices will likely come into play.
• Vehicles- this is an emerging area as vehicles moving through a city can be viewed similarly to a person walking through a building. Besides sensors, voice is an important factor.

The Data

There are four types of data to consider in these systems.

1. Business data. This is data specific to your business and the physical parameters. It contains both constant and variable data. Constant data is mostly created during system setup. Locations of beacons, doors, sensors, equipment. Variable data consists of information such as schedules, employee or membership lists, etc…
2. Individual data. This is user-identifying data, and is used to provide personalized and optimized experiences for that individual. It can provide context as well to help interpret actions based on a collection of inputs or previous actions in similar situations.
3. Aggregate data. The is the collective data from all individuals formally associated with the space. It can be all employees, shoppers, members of a gym franchise, etc… Aggregate data is depersonalized and is used to identify patterns, optimizations (perhaps rearranging the interior would optimize traffic flow), and identify trends (Tuesday evening are peak traffic times). It can also be used for suggestive purposes (the “people who bought X also bought Y” concept). Aggregate data can be aggregated within your four walls, or if you have multiple locations, across all of them.
4. Big (or External) Data. This of course comes from outside and adds context, be it weather, traffic, stock, pollen count- you name it. There will be big data sets available that can be tapped to help optimize business operations and add color to aggregate data.

Context

Context is at the heart of an immersive system. It takes the Who, What, Where, and When and combines it with sensor and environmental data, as well as other data sources, to help define what the user’s actions mean. It makes a system personalized and greatly improves the experience. With context, an immersive space can deliver a custom experience- a different experience, to each person present, at the same time.

The data sources highlighted above are what provides the context. That data can also be used to drive deep learning- AI modules that could be plugged into the system to provide fuzzier, but more likely interpretations. Non-fuzzy interpretations can sometimes arrive at the wrong conclusion unless carefully codified (just look at how many autocorrect typos you have to fix on every mobile-originated text message and email!).

As a simple example, correlating a few basic inputs can help the system interpret the actions of an individual. For example, GPS may indicate that you are at a gym. A beacon inside may determine that you are in the weight room, and that in turn tells how to interpret data from the accelerometer and heart rate sensor- you are doing bicep curls, set #2, rep #12.

Personal data can provide information for predictive behavior. For example, if you visit a shopping center a week prior to your anniversary, suggestions and offers can be given to the user accordingly- perhaps a gift and chocolates.

Context combines the Who, What, When and Where with other data sets to create a unique, personalized interpretation and experience.

Beacons (Bluetooth or increasingly virtualized), which are an important element, provide facility context. Which conference room are you in? What part of the gym are you in? Which store are you visiting, and which department are you currently in?

External data provides adds context. For an individual, aggregate data from other users of the facility can be used to predict most-used outcomes. Big Data, from outside, can be whatever is relevant to your business and can be used to predict user behavior or preferences.

When you have a contextual system, you have what I call a Non-fussy, frequently Touchless experience. You can use a device to query for specific information, but in general, just do what you would expect to do and the system should seamlessly react accordingly. The best UI is no UI at all.

The Results

When a system like this is designed and implemented, it will result in a happier customers and tenants because of improved, personalized experiences. It will help you improve the efficiency of your operation. And finally, though the collection and analysis of larger data sets, you can begin to look for new business opportunities.

Applications

I’ve cited a few examples in the article, such as your Employer, Gyms and Retail. There are a great many environments that can use elements of these systems to improve their customer’s experience. Any public space can be made immersive, from Cities and Campuses, to Sports stadiums, Restaurants, Museums, Airports, and many, many more.

The next article will get into a specific example: The Connected Gym: Tier 1.