Russian Sign Language Database for Clinical Use: Data and Annotation Peculiarities

This work has been carried out by the members of the Laboratory of Speech and Multimodal Interfaces of the St. Petersburg Federal Research Center of the Russian Academy of Sciences within an interdisciplinary research project aimed at the creation of an automatic Russian sign language translation system. The paper presents the design of a Russian sign language digital database for a specific subject area, namely, «The first visit to doctor».
Our database is meant to be used first of all a dataset for training neural network-based systems of automatic translation from Russian sign language. But also, it can be of interest for linguistics of sign languages in general since the approach elucidates solution of a number of distorting phenomena typical for continuous sign language tracking, such as epenthesis, assimilation, reduction, and hold deletion.
The principal difference between the presented video data and other datasets developed for similar purposes is the use of continuous sign utterances and elements of Russian sign language proper instead of the so-called “signed Russian” (that is, the visual form of the Russian spoken language), popular in deaf schooling.
One of the most challenging problems, dealt with in the paper, is an automatic segmentation of sign speech into separate meaningful units with clearly defined boundaries. Its solution is hampered by signs’ deformations in continuous speech. In this paper, the segmentation is carried out within the framework of the movement-hold model. This approach allows extraction of their functional core as well as annotation of possible changes likely to appear in different signs’ segments. Accordingly, each utterance was subdivided into segments of motion and hold, and the resulting sign schemes were then entered into a separate directory containing information about the main parameters of each hold, including the possibilities of change due to immediate environment. The result of this work is a full decomposition of the signs, forming the database which can find its application in different statistical models of Russian Sign Language.
Among other linguistic peculiarities of the database should be noted lexical variability of signs, mouthing and code switching between Russian sign language and “signed Russian”. Also, our signers learned Russian sign language in different regions of Russia, thus the collected data is potentially a source for research in dialectal variability of Russian sign language.

1. Brentari, D. A Prosodic Model of Sign Language Phonology. Cambridge, MA: MIT Press, 1998.

2. Burkova, S. I., Kimmelman, V. I., Filimonova, E. V., Kuseva, M. V., Varinova, O. A., Zavaritsky, D. A., Pristavko, K. V., Kadyrgulova, R. S. Introduction to the Linguistics of Sign Languages. Russian Sign Language. Textbook. Novosibirsk: NGTU, 2019, 356 p. (in Russ.)

3. Burkova, S. I., Varinova, O. A. On Territorial and Social Variation in Russian Sign Language. In Russian Sign Language: The First Linguistic Conference / Ed. O. V. Fedorova. Moscow: Buki Vedi, 2012, pp. 127–143. (in Russ.)

4. Camgöz, N. C., Kındıroğlu, A. A., Karabüklü, S., Kelepir, M., Özsoy, A. S., Akarun, L. Bosphorus Sign: A Turkish Sign Language Recognition Corpus in Health and Finance Domains. Proc. of the 10th International Conference on Language Resources and Evaluation (LREC’16), Portoroz, Slovenia, 2016, pp. 1383–1388. URL: https://aclanthology.org/L16-1220

5. De Souza, R. S., de Martino, J. M., Temoteo, J., Rodrigues, I. Automatic Recognition of Continuous Signing of Brazilian Sign Language for Medical Interview. Proc. of the Sixth International Conference on Informatics and Assistive Technologies for Health-Care, Medical Support and Wellbeing HEALTHINFO, Barcelona, Spain, 2021, pp. 41–46. URL: https://www.researchgate.net/publication/355208551_Automatic_Recognition_of_Continuous_Signing_of_Brazilian_Sign_Language_for_Medical_Interview

6. Drapkina, O. M. Brief Algorithms of Management of Patients at the Stage of Providing Primary Healthcare. Manual for therapists; Ed. O. M. Drapkina. Moscow: Vidox, 2019, 20 p. (in Russ.)

7. Geilman, I. F. Russian Sign Language Dictionary, in 2 Volumes. St. Petersburg: Prana, 2004, 363 p. (in Russ.)

8. Grenoble, L. An overviEw of Russian Sign Language. Sign Language Studies, 1992, no. 77, pp. 321– 338.

9. Grif, M. G., Kugaevskikh, A. V. Recognition of Deaf Gestures Based on a Bio-Inspired Neural Network. Journal of Physics: Conference Series, 2020, 1661 012038. DOI:10.1088/1742-6596/1661/1/012038

10. Grif, M., Korolkova, O., Manueva, Y. A New Algorithm and Other Software for Disambiguation of Polysemy and Homonymy for Computer Translation into Russian Sign Language Based on Semantic Principle. Vestnik NSU. Series: Linguistics and Intercultural Communication, 2018, vol. 16, no. 3, p. 32–44. (in Russ.)

11. Kagirov, I. A., Ryumin, D. A., Axyonov, A. A., Karpov, A. A. Multimedia Database of Russian Sign Language items in 3D. Voprosy Jazykoznaniya, 2020, no. 1, pp. 104–123. (in Russ.) DOI: 10.31857/S0373658X0008302-1

12. Kibrik, A. E. The Methodology of Field Investigations in Linguistics (Setting up the Problem). Moscow: Moscow University Press, 1972. (in Russ.)

13. Kimmelman, V. Reflexive Pronouns in Russian Sign Language and Sign Language of the Netherlands: MA Thesis in Linguistics. Amsterdam: Universiteit van Amsterdam, 2009.

14. Klezovich, A. Automatic Extraction of Handshapes Inventory in Russian Sign Language. Linguistics. WP BRP. NRU HSE, 2019, no. 86. Higher School of Economics Research Paper No. WP BRP 86/LNG/2019.

15. Korolkova, O. O. Functional Aspect of the Russian Sign Language: Defining a Research Approach. Vestnik NSU. Series: Linguistics and Intercultural Communication, 2017, vol. 15, no. 3, p. 67– 75. (in Russ.)

16. Kosmopoulos, D., Oikonomidis, I., Constantinopoulos, C., Arvanitis, N., Antzakas, K., Bifis, A., Lydakis, G., Roussos, A., Argyros, A. Towards a Visual Sign Language Dataset for Home Care Services. Proc. of the 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), 2020, pp. 520–524. DOI: 10.1109/FG47880.2020.00099

17. Leech, G. Introducing Corpus Annotation. In Corpus Annotation / Ed. by R. Garside, G. Leech, and A. McEnery. London, U.K.: Longman, 1997, pp. 1-18.

18. Liddell, S., Johnson, R. E. American Sign Language: The Phonological Base. Washington, D.C.: Ms. Gallaudet University, 1989.

19. Perlmutter, D. M. Sonority and Syllable Structure in American Sign Language. Linguistic inquiry, 1992, no. 23(3), pp. 407-442.

20. Rozelle, L. The structure of sign language lexicons: inventory and distribution of handshape and location: PhD Dissertation. Washington: University of Washington, 2003.

21. Sandler, W. The Spreading Hand Autosegment of American Sign Language. Sign Language Studies, 1986, no. 50, pp. 1–28.

22. Sandler, W., Lillo-Martin, D. Sign Language and Linguistic Universals. Cambridge: Cambridge University Press, 2006.

23. Stokoe, W. C. Sign Language Structure: An Outline of the Visual Communication Systems of the American Deaf. Buffalo: Dept. of Anthropology and Linguistics, University of Buffalo. 1960.

24. Vidalón, Y. J. E., de Martino, J. M. Continuous Sign Recognition of Brazilian Sign Language in a Healthcare Setting. Journal of Communication and Information Systems, 2015, no. 30(1), pp. 82–89. DOI:10.14209/jcis.2015.10

25. Zaytseva, G. L. Sign Speech. Dactilology: A Textbook for Higher School Students. Moscow: VLADOS, 2000, 192 p. (in Russ.)