Pushing the limits of remote RF sensing by reading lips under the face mask

Hira Hameed; Muhammad Usman; Ahsen Tahir; Amir Hussain; Hasan Abbas; Tie Jun Cui; Muhammad Ali Imran; Qammer H. Abbasi

doi:10.1038/s41467-022-32231-1

Pushing the limits of remote RF sensing by reading lips under the face mask

Hira Hameed, Muhammad Usman, Ahsen Tahir, Amir Hussain, Hasan Abbas, Tie Jun Cui, Muhammad Ali Imran, Qammer H. Abbasi^*

^*Corresponding author for this work

Electronic Engineering

Research output: Contribution to journal › Article › peer-review

Abstract

The problem of Lip-reading has become an important research challenge in recent years. The goal is to recognise speech from lip movements. Most of the Lip-reading technologies developed so far are camera-based, which require video recording of the target. However, these technologies have well-known limitations of occlusion and ambient lighting with serious privacy concerns. Furthermore, vision-based technologies are not useful for multi-modal hearing aids in the coronavirus (COVID-19) environment, where face masks have become a norm. This paper aims to solve the fundamental limitations of camera-based systems by proposing a radio frequency (RF) based Lip-reading framework, having an ability to read lips under face masks. The framework employs Wi-Fi and radar technologies as enablers of RF sensing based Lip-reading. A dataset comprising of vowels A, E, I, O, U and empty (static/closed lips) is collected using both technologies, with a face mask. The collected data is used to train machine learning (ML) and deep learning (DL) models. A high classification accuracy of 95% is achieved on the Wi-Fi data utilising neural network (NN) models. Moreover, similar accuracy is achieved by VGG16 deep learning model on the collected radar-based dataset.

Original language	English
Article number	5168
Number of pages	9
Journal	Nature Communications
Volume	13
DOIs	https://doi.org/10.1038/s41467-022-32231-1
Publication status	Published - 7 Sep 2022

Keywords

Lip-reading
face masks
COVID-19

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Documents and Links

10.1038/s41467-022-32231-1

Hameed, H. et al (2022) Pushing the limits of remote RF sensing by reading lips under the face mask
© The Author(s) 2022. This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/ licenses/by/4.0/.
Final published version, 2.27 MBLicence: CC BY

Cite this

@article{912de59c62cf48adbc23efd7deabaabb,

title = "Pushing the limits of remote RF sensing by reading lips under the face mask",

abstract = "The problem of Lip-reading has become an important research challenge in recent years. The goal is to recognise speech from lip movements. Most of the Lip-reading technologies developed so far are camera-based, which require video recording of the target. However, these technologies have well-known limitations of occlusion and ambient lighting with serious privacy concerns. Furthermore, vision-based technologies are not useful for multi-modal hearing aids in the coronavirus (COVID-19) environment, where face masks have become a norm. This paper aims to solve the fundamental limitations of camera-based systems by proposing a radio frequency (RF) based Lip-reading framework, having an ability to read lips under face masks. The framework employs Wi-Fi and radar technologies as enablers of RF sensing based Lip-reading. A dataset comprising of vowels A, E, I, O, U and empty (static/closed lips) is collected using both technologies, with a face mask. The collected data is used to train machine learning (ML) and deep learning (DL) models. A high classification accuracy of 95% is achieved on the Wi-Fi data utilising neural network (NN) models. Moreover, similar accuracy is achieved by VGG16 deep learning model on the collected radar-based dataset.",

keywords = "Lip-reading, face masks, COVID-19",

author = "Hira Hameed and Muhammad Usman and Ahsen Tahir and Amir Hussain and Hasan Abbas and Cui, {Tie Jun} and Imran, {Muhammad Ali} and Abbasi, {Qammer H.}",

year = "2022",

month = sep,

day = "7",

doi = "10.1038/s41467-022-32231-1",

language = "English",

volume = "13",

journal = "Nature Communications",

issn = "2041-1723",

publisher = "Springer Nature",

}

TY - JOUR

T1 - Pushing the limits of remote RF sensing by reading lips under the face mask

AU - Hameed, Hira

AU - Usman, Muhammad

AU - Tahir, Ahsen

AU - Hussain, Amir

AU - Abbas, Hasan

AU - Cui, Tie Jun

AU - Imran, Muhammad Ali

AU - Abbasi, Qammer H.

PY - 2022/9/7

Y1 - 2022/9/7

N2 - The problem of Lip-reading has become an important research challenge in recent years. The goal is to recognise speech from lip movements. Most of the Lip-reading technologies developed so far are camera-based, which require video recording of the target. However, these technologies have well-known limitations of occlusion and ambient lighting with serious privacy concerns. Furthermore, vision-based technologies are not useful for multi-modal hearing aids in the coronavirus (COVID-19) environment, where face masks have become a norm. This paper aims to solve the fundamental limitations of camera-based systems by proposing a radio frequency (RF) based Lip-reading framework, having an ability to read lips under face masks. The framework employs Wi-Fi and radar technologies as enablers of RF sensing based Lip-reading. A dataset comprising of vowels A, E, I, O, U and empty (static/closed lips) is collected using both technologies, with a face mask. The collected data is used to train machine learning (ML) and deep learning (DL) models. A high classification accuracy of 95% is achieved on the Wi-Fi data utilising neural network (NN) models. Moreover, similar accuracy is achieved by VGG16 deep learning model on the collected radar-based dataset.

AB - The problem of Lip-reading has become an important research challenge in recent years. The goal is to recognise speech from lip movements. Most of the Lip-reading technologies developed so far are camera-based, which require video recording of the target. However, these technologies have well-known limitations of occlusion and ambient lighting with serious privacy concerns. Furthermore, vision-based technologies are not useful for multi-modal hearing aids in the coronavirus (COVID-19) environment, where face masks have become a norm. This paper aims to solve the fundamental limitations of camera-based systems by proposing a radio frequency (RF) based Lip-reading framework, having an ability to read lips under face masks. The framework employs Wi-Fi and radar technologies as enablers of RF sensing based Lip-reading. A dataset comprising of vowels A, E, I, O, U and empty (static/closed lips) is collected using both technologies, with a face mask. The collected data is used to train machine learning (ML) and deep learning (DL) models. A high classification accuracy of 95% is achieved on the Wi-Fi data utilising neural network (NN) models. Moreover, similar accuracy is achieved by VGG16 deep learning model on the collected radar-based dataset.

KW - Lip-reading

KW - face masks

KW - COVID-19

U2 - 10.1038/s41467-022-32231-1

DO - 10.1038/s41467-022-32231-1

M3 - Article

C2 - 36071056

VL - 13

JO - Nature Communications

JF - Nature Communications

SN - 2041-1723

M1 - 5168

ER -

Pushing the limits of remote RF sensing by reading lips under the face mask

Abstract

Keywords

UN SDGs

Documents and Links

Fingerprint

Cite this