23-June-2020 By Jeffrey Cooper

Sensored: Fusion

In the previous article, we talked about a number of sensors, but all of them were discussed using sensors a la carte- for a single purpose with a bespoke application. A few months back we lightly discussed sensor fusion in the Connected Gym, in Part 3 of that series. Now we will have a deeper look at Sensor Fusion, which brings in multiple sensors at the same time for different movements and sports to give a more holistic view of what your body is actually doing.

When you work out, any single device attached to you is essentially partially blind. They usually have one or two “senses” at their disposal, and therefore it can be very tricky to suss out your routine by themselves.

What Is Sensor Fusion?

Let’s use the human body as a baseline. It’s the best example of sensor fusion there is, and one we all understand innately. We have at least 6 senses, and a very powerful neural network to process the incoming data streams.

Why six senses? There’s the traditional five always mentioned, sight, sound, touch, smell and taste. But hidden from view and inside the ear, are the Semicircular Canals, which form another sense. They are different from the others- they sense gravity and motion, and are much like an accelerometer and gyro in our phones.

Rarely are all 6 used for the same task, but for any given task, multiple senses are at work. Take eating. It’s a combination of taste, smell, and touch. Texture of food matters. Would you want to eat a rib eye steak if it was served to you in the form of a pudding? Even vision can play a part- if your steak was bright blue, you may not want to eat it, either.

When running, you are using sight, touch, and those semicircular canals are playing a pivotal role in keeping you upright. It helps you keep your form and posture. Even the slightest deviation is detected. Touch comes into it as well with sensory input from your feet hitting the ground. Sound does too, more for safety, if you’re not drowning it out with music.

So, eating uses up to four senses, and running uses up to four senses. Our brains in near-real time fuse all this information together in an neuro-analog computer and spits out muscle commands. We are born with the raw capabilities and refine it early in life. We never think twice about it unless you have a medical condition that develops and interferes with it.

Digital Fusion

The same thing can be achieved digitally. In some situations, meshes of sensors can be wired together for easy data synchronization, such as structure monitoring and building management. In our case, the connections have to be wireless.

In the diagram above, I’ve placed a number of devices on the model that are either available today, or are in development. Most of these were discussed in the previous article, but now, we’ll go into fused use cases and how they might work.

The sensors above range from smart watches and smart rings, to smart shoes, heart rates straps (chest and arm), AR glasses, sensored ear buds and various other strapped on sensors. It is not common for heart rate sensors to provide motion data today, but it could be done and it would be useful for bigger picture analysis. Movesense HR+ is one such product that does measure motion + heart rate.

Current earbuds don’t have sensors, but Samsung’s first Galaxy Buds did have accelerometer and heart rate built in.

Sensor Fusion Scenarios

Let’s take a look at a few scenarios about how the devices above may work in various combinations in different types of workouts.

Running

When you’re running, it helps to have different perspectives of your body if you’re really serious about your running. In the previous article, I talked about Moov and Lumo. Imagine them working together. In a futuristic scenario you could wear the following sensors.

• Smart Watch- accelerometers, computing hub
• Smart Ring- accelerometers, temperature
• Chest or Arm strap- heart rate, accelerometers
• Smart Shoes or ankle bracelet- accelerometers and foot pressure if you are wearing shoes.
• Mid-body waist-mounted device- accelerometers
• AR Glasses
• Ear Buds

In this fully-outfitted scenario, you get HR from the chest or arm, for a more accurate read. Wear the ring on the opposite side of the body from the watch, and you get accelerometer data from both arms. The mid-back, waist-mounted sensor gives you body posture, and the glasses can feed back head tilt, in addition to overlaying running instructions for coached runs. Audio feedback is useful, too, of course. The ankle strap gives better cadence, stride length and strike force in g’s. If you opt for smart shoes, they can provide that as well as detailed foot-to-ground interface. That can tell you how much you pronate or supinate, and if you strike in the wrong part of your foot.

This data can be fused together. Some is useful immediately- “Lift your head!” “You are leaning too far forward!” Coached tips along the way can correct bad habits while they are happening and you can see the effects immediately. After the run, see a replay and get a deeper analysis, like a coach does after a game reviewing how a player could improve their performance. This is obviously useful for competitive running, too.

You can extend this to cycling pretty easily, and include the bike sensors from the previous article into the input stream, which will affect the coaching. The body sensors can help improve form and posture.

Weight Lifting

At the gym, the challenges are different. For this scenario, let’s imaging you are wearing:

• Smart Watch- accelerometer, computing hub
• Smart Ring on opposite hand- accelerometer, temperature
• Chest or Arm strap- more accurate heart rate
• Ankle strap
• Ear buds (with sensors)

Notice that in both scenarios, the watch and ring should be on opposite sides of the body. Measuring just one side of the body only gives you a partial picture, and it is harder to discern what the user is doing. Having sensors on both sides also help discern imbalances in posture in real time, so you can correct and avoid possible injury. As above, these heart rate sensors are still (for now) more accurate than smart watches.

The ear buds can provide coaching instructions, but can also have sensors. And the ankle straps can provide accelerometer data as well. Why would you need motion data from the head and feet when lifting weights? They give better context. Sensors just on the arms cannot detect leg exercises very well. And if you are doing an inverted leg press, sensors on the ankle can count reps and speed, and the ear buds could detect orientation (you are almost upside down). This combination of data makes it much easier to detect what exercises you are doing and if you are doing them right.

Yoga

Looking at a completely different sport, yoga can benefit as well. Data from sensors on the wrist, ankle and earbuds could record positions while you get audio coaching through the earbuds (for an at home scenario).

Cross Training and CrossFit

Cross Training and CrossFit are perhaps the most challenging of all. Though different, they involve a wide variety of moves, some quite inventive. When I go to the club, I split my time between weight lifting and cross training with a variety of exercises. Battle ropes, slam balls, platform jumps, BoSu activities and jump rope all form a part of my routine. More sensors mean that more activities can be identified.

The Challenges of Sensor Fusion in Wireless Devices

There are two big challenges (aside from batteries) facing scenarios using multiple sensors. One is data synchronization. When you have data coming from independent devices, from different manufacturers, you will need to align the data to be able to make sense of the combined data. The equivalent in human senses would be if your vision was out of sync with your inner ear- it would likely lead to a vertigo-like condition.

Context can help in this case. Despite being in different locations on the body, different sensors may still be detecting common elements, such as foot strikes on the ground. That can make it much easier to align data across an array of sensors when analyzing a run.

Similarly, in weight lifting, if you know I’m in the weight room, you can look at a reduced set of matches and align data that way. Possibly all devices could emit a sync signal periodically, allowing a simpler ability to line things up. That leads to the second big problem.

Standardized Access

Today, different devices by different manufacturers don’t currently work together out of the box. Most don’t have open APIs aside from smart watches. And most don’t have provisions for a universal sync signal, which would be based on an incoming “Start workout” signal to base the sync signal on. This is something that would need to be standardized. Once done, though, the possibilities would be incredible.

And finally, there aren’t a lot of HR sensors with accelerometers, and Samsung took the sensors out of their ear buds (and they weren’t open access even so). But as sensor fusions slowly becomes a reality, sensors will creep into more devices.

Remember the arc of technology- lower power, higher resolution or processing capabilties, smaller size, lower cost and better batteries.