Huawei's StorySign App Translates Kids' Books Into Sign Language

By Holly Brockwell on at

John Lewis Elton John ad didn't make you cry? Well, prepare your feels for this one from Huawei, because we watched it and now it's raining on our faces:

In short, Huawei's made an Android app that uses AI to 'read' storybooks to children page by page in sign language.

It's not widely known that deaf children often have trouble reading, but it makes sense when you think about it, because reading the words aloud is usually such a critical part of the process. When you consider that 90% of deaf children are apparently born to hearing parents, you can see the problem.

The avatar doing the signing in the app is called Star, and was designed by Aardman Animations (the studio behind Wallace & Gromit, Sean the Sheep and other such adorable critters):

Aardman used motion capture techniques to ensure the sign language was spot on.

Currently, it looks like the app only works with one book in English: 'Where's Spot?'. Presumably more will be added, but it doesn't sound like the app will be able to flawlessly translate any given book into sign language, which is a shame but not entirely surprising.

While the app does use OCR (optical character recognition) and image recognition, that's mostly so it can still 'read' the page with the phone held at an angle. The AI is used to speed up 'reading' so kids aren't waiting ages for the next page to load in the app.

(That's the Huawei Mate 20 Pro in the picture, of course).

To Huawei's credit, they worked with experts and charities from the deaf community to make something that's actually useful and not just a publicity stunt. They're also taking donations all December for deaf literacy projects, including more books for the app.

Andrew Garrihy, Huawei CMO for Western Europe comments:

"We created StorySign to help make it possible for families with deaf children to enjoy an enriched story time. We hope that by raising awareness of deaf literacy issues, people will be encouraged to donate to or support one of the fantastic charity partners we are working with across Europe."

European Union of the Deaf Exec Director Mark Wheatley adds:

"With StorySign, Huawei is using the power of its AI and innovative technology in a meaningful way. We’re very hopeful that it will make a significant impact in the deaf community, helping more deaf children learn how to read at the same level as hearing children.

We also hope the launch of StorySign will support a wider conversation about ensuring equality in every aspect of their lives for deaf people across Europe."

The app's available from today (Android only), and there's lots more info on the StorySign website.