Deep-Learning for Hand-Gesture Recognition with Simultaneous Thermal and Radar Sensors

In this paper, we present a framework for integrating two different types of sensors for hand-gesture recognition using deep-learning. The two sensors utilize completely different approaches for detecting the signal, namely; an ultra-wideband (UWB) impulse radar sensor and a thermal sensor. For robust gesture classification two parallel paths are utilized, each employs a combination of a convolutional neural network (CNN) and a long short-term memory (LSTM) network on both the radar signal and the thermal signal. The classification results from the two paths are then fused to improve the overall detection probability. The two sensors compliment the capability of each other; while the UWB radar is accurate for radial movement and less accurate for lateral movement, the thermal sensor is vice-versa. Thus, we find that combining both sensors produces near perfect classification accuracy of 99 % for 14 different hand-gestures.