FINGERSPELLING WITH MACHINE LEARNING

Innovation Eurobest Award

Case Film

Presentation Image

TitleFINGERSPELLING WITH MACHINE LEARNING
BrandAMERICAN SOCIETY FOR DEAF CHILDREN
Product/ServiceFINGERSPELLING
Category A02. Applied Innovation
Entrant HELLO MONDAY Aarhus, DENMARK
Idea Creation HELLO MONDAY Aarhus, DENMARK
Production HELLO MONDAY Aarhus, DENMARK
Credits
Name Company Position
Rikke Agersnap Hello Monday Project Manager
Victor Dose Hello Monday 3D Designer
Toni Größchen Hello Monday Designer
Patrycja Hølm Hello Monday Designer
Line Jacobsen Hello Monday Designer
Anders Jessen Hello Monday Creative Director / CTO
Anders Mikkelsen Hello Monday Designer
Victor Rønnow Hello Monday Designer

Why is this work relevant for Innovation?

Instead of simply talking about - or visualising the ASL alphabet - we wanted to make a tool that can actually help you learn the ASL alphabet. We believe this is the first time advanced hand tracking (without any specialized equipment) - in the browser - has been used to teach people ASL Fingerspelling.

Background

Every year 2 - 3 out of every 1,000 children born in the US are deaf or hard-of-hearing. 90% of these children are born to hearing parents, and in many cases, their child is the first deaf person these parents have ever encountered. Without being introduced to sign language at an early stage, a deaf child may miss out on learning language. This can lead to language delay or deprivation, which has long-term negative impacts on a child’s life. That is why it is so important that parents of deaf children have the opportunity to learn American Sign Language

Describe the idea

Fingerspelling.xyz is a browser-based app that uses a webcam and machine learning to analyze your hand shapes so you can learn to sign the ASL alphabet correctly. To start, you are given a series of words and the computer shows you a 3D model of how your hand should be positioned for each letter. When you sign the word, the camera tracks your hand movements and gives you feedback so you can make corrections as needed. This helps you quickly develop your fingerspelling skills and move to the next level of the program.

What were the key dates in the development process?

N/A

Describe the innovation / technology

The fingerspelling game is a great way to introduce the basics of ASL in a fun and playful way. Instead of having to read or watch videos about fingerspelling, we offer an online teaching tool that guides you step by step in how to master fingerspelling - hands on! The game leverages advanced hand recognition technology, matched with machine learning, to give you real time feedback via the webcam for each sign and word you spell correctly. The game is designed for desktop, primarily to be used by parents for deaf children, but kids will also find it a fun way to practice and improve their fingerspelling technique. Fingerspelling is an essential part of ASL, the primary language of the Deaf community. It is often used for proper nouns or to sign a word you don’t know the sign for.

Describe the expectations / outcome

The site was launched in late April 2021 - and within 2 months over 150.000 correct hand-signs have been made. The site has been featured across a variety of news and blogs - including Engadget, Mashable and Yahoo. And we have helped bridge the gap between the hearing and the deaf.