First-person expository discourse in multimodal perspective: complementary and redundant co-speech gesture patterns

The article addresses the problem of co-speech gesture alignment in the multimodal construal of first-person expository discourse. Adopting a cognitive semiotic framework, the study reveals the key semiotic dimensions of first-person expositive displaying its metaphoricity, entrenchment, indexicality, pragmaticity, agency, and transitivity in referent and event construal. It hypothesizes that despite the variance in co-speech gesture distribution mediated by different semiotic dimensions, there exist stable alignment patterns of complementary and redundant nature acting as their regulations. The research data are 20 recorded expository monologues containing 5603 co-speech gestures, with 725 first-person perspective utterances containing 1959 cases of co-speech gesture use in four groups, deictic, representational, pragmatic and adapters. The utterances were further explored to identify the complementary and redundant alignment patterns in the distribution of co-speech gestures aligned with metaphors, spatial deictic words and discourse markers, agentive and transitive verbs within the first-person expository discourse profile. The results show that while gesture iconicity displayed in representational and pragmatic gestures in co-speech gesture alignment is displayed in both complementary and redundant patterns, gesture indexicality displayed in deictic gestures is predominantly complementary. Additionally, the explored semiotic dimensions mediate the overall distribution of gestures within the utterance despite the amalgams of these dimensions, which alleviates the negative effects of their identification as synchronized with single cues in speech. Overall, the study paves the way to establishing the common cognitive semiotic grounds of multimodal alignment in discourse and to prognosticating this alignment as mediating by the distribution of semiotic dimensions.

1. Bressem J., Müller C. The family of away gestures. Body – Language – Communication: An International Handbook on Multimodality in Human Interaction, vol. 2. Ed. by C. Müller, A. Cienki, E. Fricke, S. H. Ladewig, D. McNeill, and S. Teßendorf. Berlin, De Gruyter Mouton), 2014, pp. 1592–1604.

2. Chan D. M., Kelly S. Construing events first-hand: Gesture viewpoints interact with speech to shape the attribution and memory of agency. Memory and Cognition, 2021, no. 49, pp. 884–894.

3. Chen Y., Adolphs S., Knight D. Towards a speech-gesture profile of discourse markers: The case of ‘I mean’. Lingua, 2024, no. 312, 2024, Article number 103836.

4. Cienki A., Müller C. (eds.) Metaphor and gesture. Amsterdam and Philadelphia, John Benjamins Publishing Company, 2008, 306 p.

5. de Ruiter J. P. The Production of gesture and speech. Language and Gesture. Ed. by D. McNeill. Cambridge, Cambridge University Press, 2000, pp. 284–311.

6. Denisova V. A., Cienki A., Iriskhanova O. K. Boundary expression in verbs and gesture: Differences between l1 and l2 speakers. Computational Linguistics and Intellectual Technologies: Papers from the Annual International Conference “Dialogue”, 2018, no. 17 (4), pp. 163–171.

7. Grishina Е. А. Manual pointing as a system (based on Multimedia Russian Corpus). Voprosy Yazykoznanija, 2012, no. 3, pp. 3–50. (in Russ.)

8. Hinnell J., Parrill F. Gesture influences resolution of ambiguous statements of neutral and moral Preferences. Frontiers of Psychology, 10 December 2020, vol. 11. URL: https://doi.org/10.3389/fpsyg.2020.587129 (accessed 15.08.2025).

9. Iriskhanova O. K., Cienki A. The semiotics of gestures in cognitive linguistics: Contribution and challenges. Voprosy Kognitivnoy Lingvistiki, 2018, no. 4, pp. 25–36.

10. Iriskhanova O. K., Kiose M. I., Leonteva A. V., Agafonova O. V. Multimodal languaging: Reification profiles in language and gesture. Linguistic Frontiers, 2023, no. 6 (2), pp. 78–91.

11. Kelly S., Ozyürek A., Maris E. Two sides of the same coin: speech and gesture mutually interact to enhance comprehension. Psychological Science, 2010, no. 21, pp. 260–267.

12. Kendon A. Gesture: Visible action as utterance. Cambridge, Cambridge University Press, 2004, 410 p.

13. Leonteva A. V., Cienki A., Agafonova O. V. Metaphoric gestures in simultaneous interpreting. Russian Journal of Linguistics, 2023, no. 27 (4), pp. 820–842.

14. Litvinenko A. O., Kibrik A. A., Fedorova O. V., Nikolaeva J. V. Annotating hand movements in multichannel discourse: Gestures, adaptors and manual postures. The Russian Journal of Cognitive Science, 2018, no. 5 (2), pp. 4–17.

15. Mittelberg I., Waugh L. R. 2009. Metonymy first, metaphor second: A cognitive semiotic approach to multimodal figures of thought in co-speech gesture. Ed. by Ch. J. Forceville, E. Urios-Aparisi. Multimodal metaphor. Berlin, De Gruyter Mouton, 2009, pp. 329–358.

16. Müller C. Gestural modes of representation as techniques of depiction. Ed. by C. Müller, A. Cienki, E. Fricke, S. H. Ladewig, D. McNeill, J. Bressem. Body – language – communication. Berlin, De Gruyter Mouton, 2014, pp. 1687–1702.

17. Wu Suwei. Multimodality of transitive, intransitive and copular constructions in spoken language. Linguistics and Literature Studies, 2019, no. 7, pp. 241–249.

18. Wu Suwei, Cienki A. Transitivity, events, and gesture: The Case of the causative-inchoative alternation. Open Linguistics, 2019, no. 5, pp. 311–331.