Simon Dogger – inclusive product developer – imagined this possibility with the Emotion Whisperer: A concept consisting of a pair of smart glasses, emotion recognition software and a haptic feedback sleeve that conveys emotions to the wearer through vibrations.

Imagine having only your ears and sense of touch to help you understand the people you talk to. You hear the words, the voice, intonation and pauses. You get the gist of the conversation, the basic information conveyed, but miss out on all the extra bits that make human interaction multi-layered and meaningful. Most importantly, you miss out on the facial expressions and body language that help you understand the emotions of the people around you.

Simon Dogger, experienced product designer, is working on a way to address this issue. Being visually impaired himself, he is uniquely qualified to relate. He explains, “There is a real loss for visually impaired people. Because they have trouble recognising the emotions of their conversation partners, their quality of communication suffers. A potential consequence is increased isolation and lower self-esteem. That’s why I came up with the Emotion Whisperer. It is a set of wearables that will help blind and low sighted people sense the emotions of their conversation partners. Eventually, it will allow them to have richer and more meaningful social interactions.”

The Emotion Whisperer concept works as follows: First, a pair of glasses with integrated camera captures the wearer’s conversation partner. This footage is then transmitted wirelessly to a smartphone and interpreted by an emotion recognition app. The app is capable of recognising both the type of emotion (out of the six base emotions: happiness, sadness, disgust, fear, surprise, and anger) and the intensity. This data is then relayed to a haptic feedback wearable that relays the type and intensity of emotion to the wearer through vibrations.

The concept has already won several awards and is being developed further with support from the University of Amsterdam and the Technical University of Eindhoven, among others.

Currently, the focus is on validating whether users are able to learn and recognise emotions through vibrations on the skin. In other words, can they distinguish between 18 different haptic feedback (vibration) cues that indicate the type and intensity of the emotion? With the original haptic feedback wearable, this turned out to be problematic.

Mr Dogger remembers, “The haptic feedback was just not precise and distinct enough for users to recognise the intensities of the emotions. That’s why I started looking for a partner with expertise in haptic feedback to develop an improved version.” This led him to Elitac Wearables with the following questions:

At Elitac Wearables, we were immediately enthusiastic about the project. Our focus is on developing wearables that add real value to the user. And helping to improve communication for visually impaired people definitely qualifies, in our view. We knew we had the required haptic feedback expertise to improve the original design. And because we could make use of our modular building blocks – the ScienceSuit haptic feedback hardware and our custom operating system TacOS – we were confident that we could deliver the wearable within the allotted budget and time.

Senior Researcher Wouter Vos remembers, “From the beginning, we thought it was a very inspiring project, helping visually impaired people sense emotions. It’s an amazing use of the sense of touch and haptic feedback to substitute other senses. And a real challenge too, because conveying 18 distinct haptic cues on just one limb is not easy to accomplish!”

All development projects at Elitac Wearable start with an intake meeting. This is important to understand the project background, prospective users and potential hurdles. Mr Dogger explained his expectations for the new wearable, what it should look like, what kind of haptic feedback patterns were required, software interfaces, etc.

We began by building a simple mock-up. This was relatively quick and easy because we were able to use our ScienceSuit tactor strings and straps. The mock-up consisted of a basic set of tactors (vibration motors) held in place on the arm by straps.

But it quickly became apparent that we needed to carefully consider the textile carrier: It had to be easy to put on for blind and low-sighted users. This meant that we could not use zippers or other closures. Similarly, the software had to be accessible and easy to operate for visually impaired users.

With this is mind, we set to work on the final Proof of Concept. We designed a triple-layered textile sleeve: a middle layer that contains the tactors, protected on both sides by additional layers. All layers are made of stretch material and the cables are specifically designed to stay in place when the sleeve is pulled over the (wider) hand. Some clever design choices make it suitable for visually impaired users: A thumb hole makes it impossible to wear the sleeve the wrong way around and pull straps on the bottom and top guide the wearer when pulling the sleeve on.

For textile designer Marloes Peters, it was a refreshing experience: “As a designer, I use a lot of visual design aspects when designing garments. When you are designing for visually impaired users, of course you still make it look good, but you also need to focus on the tactile sensation of those design aspects. Is it clear how to put it on even if you cannot see any visual indicators? How does it feel? And so on.”

With regard to the software, we were able to use TacOS, our in-house developed operating system, specifically for wearables. It only required minor amendments to reflect the sleeve’s tactor constellation and to add in the new patterns. To make it more accessible, we also changed the way the various patterns are selected.

The final version of the sleeve has 16 integrated tactors that are capable of delivering 24 pre-programmed haptic feedback patterns. The patterns consist of combinations of taps and strokes on various parts of the lower arm, making each pattern unique and distinctive. During the delivery of the sleeve and manual, we ran through the functioning of the sleeve, reviewed all the patterns, taught Mr Dogger how to use it, and made sure expectations were met.

Thanks to our haptic feedback expertise and modular building blocks, we were able to complete the sleeve in under 78 hours of development time and within the allotted budget. The sleeve features a wide variety of distinct haptic feedback patterns that can be used to continue research and development of the Emotion Whisperer.

First, the new sleeve will be used to test the extent to which users are able to learn and recognise emotions through the various haptic cues. A future development includes an iPhone app capable of recognising emotions from facial expressions and body language. When connected to the sleeve developed by Elitac Wearables, the sleeve will relay the appropriate emotions to the wearer.

The haptic feedback sleeve we’ve developed for this project also has huge potential for other projects as an information and communication display, for example:

Are you interested in incorporating the haptic sleeve into your wearable? Or looking for a wearables development partner with haptic feedback expertise? Please get in touch, we’d love to hear your ideas!