Sign Language Recognition by Combining Leap Motion Controller and Hand Image Information

Sign language recognition systems help deaf people to access various media. In this paper, the Leap Motion Controller (LMC) and the image of the hand are exploited for sign language recognition. The LMC provides 3D position of the hand joints. The first set of features are extracted from the data provided by the LMC. When the hand is not located in vertical view of the LMC or when the hand posed like a fist, the precise position of the hand joints is not recognizable. The second feature extracted from hand image helps most hand gestures be recognized precisely. The second feature includes histogram of oriented gradients and the distance of the hand contour form the center of the hand. Also, a dataset composed of variant American sign language gestures is created which includes 64000 samples. In recognition stage, random forest is applied which is a good option for large datasets. The experimental results show that the proposed method performs better than similar methods.