ArticleSensors (Basel, Switzerland)2023
Audio-Visual Speech and Gesture Recognition by Sensors of Mobile Devices.
Article in Sensors (Basel, Switzerland), 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 18 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
18 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Machine Learning and Deep Learning Approaches for Arabic Sign Language Recognition: A Decade Systematic Literature Review.Sensors (Basel, Switzerland) · 2024Pooled it
- An intelligent fusion-based transfer learning model with artificial protozoa optimiser for enhancing gesture recognition to aid visually impaired people.Scientific reports · 2025Article
- Unsupervised Clustering and Ensemble Learning for Classifying Lip Articulation in Fingerspelling.Sensors (Basel, Switzerland) · 2025Article
- Hand Gesture Recognition Using Ultrasonic Array with Machine Learning.Sensors (Basel, Switzerland) · 2024Article
- Real-Time Hand Gesture Monitoring Model Based on MediaPipe's Registerable System.Sensors (Basel, Switzerland) · 2024Article
- Intelligent Gesture Recognition Based on Screen Reflectance Multi-Band Spectral Features.Sensors (Basel, Switzerland) · 2024Article
- Hybridization of Acoustic and Visual Features of Polish Sibilants Produced by Children for Computer Speech Diagnosis.Sensors (Basel, Switzerland) · 2024Article
- Enhancing Air Traffic Control Communication Systems with Integrated Automatic Speech Recognition: Models, Applications and Performance Evaluation.Sensors (Basel, Switzerland) · 2024Review
- Verification of Criterion-Related Validity for Developing a Markerless Hand Tracking Device.Biomimetics (Basel, Switzerland) · 2024Article
- Nonlinear Regularization Decoding Method for Speech Recognition.Sensors (Basel, Switzerland) · 2024Article
- Real-Time Arabic Sign Language Recognition Using a Hybrid Deep Learning Model.Sensors (Basel, Switzerland) · 2024Article
- Efhamni: A Deep Learning-Based Saudi Sign Language Recognition Application.Sensors (Basel, Switzerland) · 2024Article
- End-to-End Ultrasonic Hand Gesture Recognition.Sensors (Basel, Switzerland) · 2024Article
- ST-TGR: Spatio-Temporal Representation Learning for Skeleton-Based Teaching Gesture Recognition.Sensors (Basel, Switzerland) · 2024Article
- The Comfort and Measurement Precision-Based Multi-Objective Optimization Method for Gesture Interaction.Bioengineering (Basel, Switzerland) · 2023Article
- Continuous Sign Language Recognition and Its Translation into Intonation-Colored Speech.Sensors (Basel, Switzerland) · 2023Article
- Re-Evaluation Method by Index Finger Position in the Face Area Using Face Part Position Criterion for Sign Language Recognition.Sensors (Basel, Switzerland) · 2023Article
- Domain Adaptation with Contrastive Simultaneous Multi-Loss Training for Hand Gesture Recognition.Sensors (Basel, Switzerland) · 2023Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
3 authors.
Funding
Abstract
Audio-visual speech recognition (AVSR) is one of the most promising solutions for reliable speech recognition, particularly when audio is corrupted by noise. Additional visual information can be used for both automatic lip-reading and gesture recognition. Hand gestures are a form of non-verbal communication and can be used as a very important part of modern human-computer interaction systems. Currently, audio and video modalities are easily accessible by sensors of mobile devices. However, there is no out-of-the-box solution for automatic audio-visual speech and gesture recognition. This study introduces two deep neural network-based model architectures: one for AVSR and one for gesture recognition. The main novelty regarding audio-visual speech recognition lies in fine-tuning strategies for both visual and acoustic features and in the proposed end-to-end model, which considers three modality fusion approaches: prediction-level, feature-level, and model-level. The main novelty in gesture recognition lies in a unique set of spatio-temporal features, including those that consider lip articulation information. As there are no available datasets for the combined task, we evaluated our methods on two different large-scale corpora-LRW and AUTSL-and outperformed existing methods on both audio-visual speech recognition and gesture recognition tasks. We achieved AVSR accuracy for the LRW dataset equal to 98.76% and gesture recognition rate for the AUTSL dataset equal to 98.56%. The results obtained demonstrate not only the high performance of the proposed methodology, but also the fundamental possibility of recognizing audio-visual speech and gestures by sensors of mobile devices.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.