Title	SIGNS
Brand	GERMAN YOUTH ASSOCIATION OF PEOPLE WITH HEARING LOSS
Product/Service	SIGNS SMART TOOL
Category	A06. Data-technology
Entrant	MRM//McCANN Frankfurt, GERMANY
Idea Creation	MRM//McCANN Frankfurt, GERMANY
Idea Creation 2	McCANN FRANKFURT, GERMANY
Production	DROPOUT-FILMS Mainz, GERMANY
Additional Company	DAS WERK Frankfurt, GERMANY
Additional Company 2	STUDIO FUNK Frankfurt, GERMANY

Name	Company	Position
Sebastian Hardieck	McCANN Worldgroup Germany	Chief Creative Officer
Martin Biela	MRM//McCANN	Executive Creative Director, Head of LAB13 EMEA
Dushan Drakalski	McCANN Frankfurt	Executive Creative Director
Dominik Heinrich	MRM//McCANN New York	Vice President Global, Product Innovation & LAB13
Mark Hollering	MRM//McCANN	Director Creative Technology & LAB13, Germany
Chris Endecott	MRM//McCANN	Senior Copywriter / Concept & LAB13
Jan Portz	MRM//McCANN	Group Creative Director UX
Sebastian Klein	MRM//McCANN	IT Project Manager
Jawad Saleem	McCANN Frankfurt	Senior Art Director
Michael Klaiber	MRM//McCANN	Senior Motion Designer
Jerome Cholet	McCANN Woprldgroup Germany	PR & Communications Director
Nico Koehler	MRM//McCANN	Art Director
Sofia Paz-Vivo	MRM//McCANN	Trainee LAB13
Irini Sidira	MRM//McCANN	Junior Art Director
Klaus Flemmer	McCANN Worldgroup Germany	Head of Production
Michelle Mohring	German Youth Association of People with Hearing Loss	Chairwoman
Lucas Garthe	German Youth Association of People with Hearing Loss	Clerk

Why is this work relevant for Creative Data?

SIGNS is the first smart voice assistant solution for people with hearing loss worldwide. It’s an innovative smart tool that recognizes and translates sign language in real-time and then communicates directly with a selected voice assistant service (e.g. Amazon Alexa, Google Assistant or Microsoft Cortana). SIGNS is reinventing voice – one gesture at a time. Many people with hearing loss use their hands to speak. And that’s all they need to talk to SIGNS. How's the weather tomorrow? Change lights to blue. Find an Italian restaurant. Just speak, and SIGNS will answer.

Background

There are over 2 billion voice-enabled devices across the globe. Voice assistants are changing the way we shop, search, communicate or even live. At least for most people. But what about those without a voice? What about those who cannot hear? According to the World Health Organization around 466 million people worldwide have disabling hearing loss. Project SIGNS was developed to create awareness for inclusion in the digital age as well as to facilitate access to new technologies.

Describe the Creative idea / data solution (20% of vote)

Many people with hearing loss use their hands to speak. Their hands are their voice. However, voice assistants use natural language processing to decipher and react only to audible commands. No sound, no reaction. Voice is predominantly seen as something audible, something you can hear. But for people with hearing loss that use their hands. Voice is more. Voice is gestures, movements, and even facial expressions. SIGNS bridges the gap between deaf people and voice assistants, by recognizing gestures to communicate directly with existing voice assistant services (e.g. Amazon Alexa, Google Home or Microsoft Cortana). SIGNS is the first smart voice assistant solution for people with hearing loss worldwide.

Describe the data driven strategy (30% of vote)

SIGNS was pre-trained with video footage of people who use sign language. SIGNS includes a training interface that can be used to teach new gestures in real time. SIGNS then recognizes these gestures and acts as an interface to voice assistant systems such as Amazon Alexa, Google Home or Microsoft Cortana. SIGNS is based on an intelligent machine learning framework that is trained to identify body gestures with the help of an integrated camera. These gestures are converted into a data format that the voice assistant service understands. The voice assistant processes the data in real-time and replies appropriately. SIGNS replaces voice assistants’ typical form of communication through audio signals with a visuality – but not only by displaying the word. The visual interface of SIGNS fulfills various requirements that are necessary for an intuitive experience. SIGNS follows the basic principles of sign language. Therefore, the SIGNS dictionary was developed

Describe the creative use of data, or how the data enhanced the creative output (30% of vote)

SIGNS uses an integrated camera to recognize sign language in real-time and communicates directly with a voice assistant. The system is based on the machine learning framework Google Tensorflow. The result of the pre-trained MobileNet is used to train several KNN classifiers on gestures. The recognition calculates the likelihood of the webcam's recorded gestures and converts into text. The resulting sentences are translated into conventional grammar and sent to a cloud-based service that generates language from it. In other words, the gestures are converted into a data format (text to speech) that the selected voice assistant understands. In this case, shown Amazon Voice Service (AVS). AVS responds with meta and audio data, which in turn is converted from a cloud service to text (text to speech). The result is displayed. SIGNS works on any browser-based operating system that has an integrated camera and can be connected to a voice assistant.

List the data driven results (20% of vote)

Project SIGNS created awareness for inclusion in the digital age as well as facilitated access to new technologies. The response of the deaf community was overwhelming. Just like voice, gestures are an intuitive way of communicating, making it extremely relevant for the industry. Not just for the hearing impaired, but for everyone. People think it is awkward to speak to the invisible in public, that’s why we believe that invisible conversational interactions with the digital world are not limited to voice itself. Further we started a cooperation with the German Youth Association of People with Hearing Loss as a partner and extended the usability. Never before a sign language assistant was launched in that quality and with the prospect of becoming a worldwide platform, that can easily be accessible from all over the world, learning new signs and sign languages.