Topic Modeling of Scientific Texts Using BERTopic (Based on Scientific Abstracts on the Great Patriotic War published in 2014–2023)

Amid the continuous growth of the volume of scientific literature, the development of efficient tools for systematizing and automating the analysis of scientific texts has become a critical challenge. These tools are essential to ensure the accessibility, organization, and information retrieval across various scientific domains. Topic modeling is one of the most effective approaches for automated text analysis since it enables the identification of key themes within large collections of texts, facilitating classification and speeding up the search for relevant information. This is particularly important for analyzing scientific literature and tracking research trends. In this study, the authors consider the capabilities of the BERTopic algorithm for topic modeling of Russian-language scientific texts. The dataset consists of scientific articles on the Great Patriotic War, published in Russian academic journals over the past decade (2014–2023). The relevance of this work arises from the limited research on the application of topic modeling algorithms to scientific texts, especially those that utilize large language models. There is a notable gap in the studies focusing on this approach. Additionally, there is an urgent need in more advanced methods for searching and selecting scholarly sources. This further highlights the importance of the current study. The paper provides a detailed description of the data collection, preprocessing, and topic modeling processes. The results reveal the most prevalent topics in academic works, offering valuable insights for researchers in computational linguistics, as well as for historians and scholars of Russian historiography. The approach to corpus building, combining the use of keyword markers, automated data collection tools, and expert validation, can be applied to other subject areas in similar studies. Topic modeling allowed for the identification of key themes within the corpus, along with the keywords that characterize them, and facilitated the creation of both hierarchical and dynamic representations of these topics. BERTopic’s strengths lie in its use of vector-based text representations from large language models, as well as its extensive tools for visualizing the resulting topics.

1. Abdelrazek, A., Eid, Y., Gawish, E., Medhat, W., Hassan, A. Topic modeling algorithms and applications: A survey. Information Systems, 2023, no. 112, p. 102131. DOI 10.1016/j.is.2022.102131

2. Babina O. I. Topic Modeling for Mining Opinion Aspects from a Customer Feedback Corpus. Automatic Documentation and Mathematical Linguistics, 2024, vol. 58, no. 1, pp. 63–79. DOI 10.3103/S0005105524010060

3. Bird S. NLTK: the natural language toolkit. Proceedings of the COLING/ACL, 2006, pp. 69–72.

4. Bodrunova S. S., Blekanov I. S., Kukarkin M. Topic modeling for Twitter discussions: Model selection and quality assessment. 6th SWS International Scientific Conferences on social sciences, 2019, pp. 207–214. DOI 10.5593/SWS.ISCSS.2019.5/S16.026

5. Chizhik A. V., Sergeyev D. A. Exploring the Parliamentary Discourse of the Russian Federation Using Topic Modeling Approach. Digital Transformation and Global Society (DTGS 2021), 2021, pp. 403–416. DOI 10.1007/978-3-030-93715-7_29

6. Devlin J., Chang M.-W., Lee K., Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of NAACL, 2019, pp. 4171–4186. DOI 10.18653/v1/N19-1423

7. Epifanov А. Е., Krasnozhenova Е. Е. Organization and holding sanitary and epidemiological measures in Stalingrad and separate districts of the Stalingrad region (1942-1943). Science journal of Volgograd state university. History. Area Studies. International Relations, 2023, iss. 1, pp. 162– 174. (in Russ.) DOI 10.15688/jvolsu4.2023.1.15

8. Galushko I. N. The use of topic modeling to optimize the process of searching for relevant historical documents (on the example of the stock exchange press of the early 20th century). Historical informatics, 2023, iss. 2, pp. 129–144. (in Russ.) DOI 10.7256/2585-7797.2023.2.43466

9. Glazkova A. V., Kruzhinov V. M., Sokova Z. N. Dynamic Topic Models for Retrospective Event Detection: A Study on Soviet Opposition-Leaning Media. Lecture Notes in Computer Science, 2019, no. 11832, pp. 145–154. DOI 10.1007/978-3-030-37334-4_13

10. Grootendorst M. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. ArXiv preprint arXiv:2203.05794, 2022.

11. Kirina М. А. A comparison of topic models based on LDA, STM and NMF for qualitative studies of Russian short prose. NSU Vestnik. Series: Linguistics and Intercultural Communication, 2022, iss. 2, pp. 84–101. (in Russ.) DOI 10.25205/1818-7935-2022-20-2-93-109

12. Kolmogorova A. V., Kolmogorova P. A., Kulikova E. R. About the past, but at different times: computer analysis of texts in textbooks on the history of the USSR/Russia for six generations of students. Tomsk State University Journal of Philology, 2024, iss. 89, pp. 73–103. (in Russ.) DOI 10.17223/19986645/89/4

13. Kolmogorova A. V., Zalevskaya E. D. Computer-assisted modeling as an instrument for fiction text analysis. Philological class, 2023, iss. 2, pp. 22–33. (in Russ.)

14. Korobov M. Morphological analyzer and generator for Russian and Ukrainian languages. AIST: 4th International Conference, Revised Selected Papers 4, 2015, pp. 320–332. DOI 10.1007/978-3- 319-26123-2_31

15. Korshunov А., Gomzin А. Topic modeling of texts in natural language. Proceedings of ISP RAS, 2012, iss. 23, pp. 215–244. (in Russ.)

16. Krasnov F. V., Dimentov A. V., Shvartsman M. E. Using topic models for pairwise comparison of collections of scientific papers. Informatics and Applications, 2020, vol. 14, iss. 3, pp. 129–135. (in Russ.) DOI 10.14357/19922264200318

17. Krinko Е. F., Medvedev М. V. Demographic consequences of the Stalingrad battle. Science journal of Volgograd state university. History. Area Studies. International Relations, 2018, iss. 1, pp. 91–104. (in Russ.) DOI 10.15688/jvolsu4.2018.1.9

18. Kuznetsov A. V. Topic modeling in historical sciences. Sociosfera [Sociosphere], 2022, iss. 4, pp. 9–12. (in Russ.)

19. Maltseva A., Shilkina N., Evseev E., Matveev M., Makhnytkin O. Topic modeling of Russian-language texts using the parts-of-speech composition of topics (on the example of volunteer movement semantics in social media). FRUCT, 2021, pp. 247–253. DOI 10.23919/FRUCT52173.2021.9435475

20. Mitrofanova О. А., Athogodage М. М. Dynamic topic modelling of the Russian legal text corpus. Terra Linguistica. 2023, iss. 1, pp. 70–87. (in Russ.) DOI 10.18721/JHSS.14107

21. Nokel M. A., Loukachevitch N. V. Topic Models: Adding Bigrams and Taking Account of the Similarity between Unigrams and Bigrams. Numerical Methods and Programming, 2015, vol. 16, pp. 215–234. (in Russ.) DOI 10.26089/NumMet.v16r222

22. Owa D. L. M. Identification of topics from scientific papers through topic modeling. Open Journal of Applied Sciences, 2021, vol. 10, no. 04, p. 541. DOI 10.4236/ojapps.2021.104038

23. Popova О. V., Grishin N. V., Pogodina М. Ya. Communication of youth with the highest officials of the subjects of the Russian Federation on the social network “VKontakte” in 2022. Polis. Political Studies, 2023, iss. 4, pp. 122–137. (in Russ.) DOI 10.17976/jpps/2023.04.09

24. Reimers N., Gurevych I. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of EMNLP-IJCNLP, 2019, pp. 3982–3992. DOI 10.18653/v1/D19-1410

25. Robledo S., Zuluaga M. Topic modeling: Perspectives from a literature review. IEEE Access, 2022, no. 11, рр. 4066–4078. DOI 10.1109/ACCESS.2022.3232939

26. Romanov R. E. Social and labor relocations of evacuated people in west Siberian rear (1941-1945): constraints and incentives. Vestnik NSU. Series: History and Philology, 2016, iss. 8, pp. 146–156. (in Russ.)

27. Sakhovskiy A., Solovyev V., Solnyshkina M. Topic modeling for assessment of text complexity in Russian textbooks. 2020 Ivannikov Ispras Open Conference (ISPRAS), 2020, pp. 102–108. DOI 10.1109/ISPRAS51486.2020.00022

28. Solovyev V., Solnyshkina M., Tutubalina E. Topic Modeling for Text Structure Assessment: The case of Russian Academic Texts. Journal of Language and Education, 2023, vol. 9, no. 3 (35), pp. 143–158. DOI 10.17323/jle.2023.16604

29. Snegireva L. I. Re-evacuation of children and children institutions form West-Siberian region (1942- 1945). Vestnik NSU. Series: History and Philology, 2016, iss. 8, pp. 157–169. (in Russ.)

30. Syed S., Spruit M. Full-text or abstract? examining topic coherence scores using latent dirichlet allocation. 2017 IEEE International conference on data science and advanced analytics (DSAA), 2017, pp. 165–174. DOI 10.1109/DSAA.2017.61

31. Ten L. V. Topic modeling in automatic categorization of news texts. Terra Linguistica, 2023, iss. 2, pp. 77–91. (in Russ.) DOI 10.18721/JHSS.14207

32. Tverskoi О. I., Kaneva О. N. Application of topic modeling for the decomposition of the textual formulation of educational competence. Applied mathematics and fundamental computer science, 2023, iss. 1, pp. 33–46. (in Russ.) DOI 10.25206/2311-4908-2023-10-1-33-43

33. Vorontsov K. V., Karabulatova I. S., Kattsina T. A., Borodina N. V., Sinitsyn M. V. Problems of developing an analytical information system on the history of social assistance in the Yenisei province (19th - first quarter of the 20th century). Issues of history, 2022, iss. 12-3, pp. 150–161. (in Russ.) DOI 10.31166/VoprosyIstorii202212Statyi91

34. Zamiraylova E., Mitrofanova O. Dynamic topic modeling of Russian prose of the first third of the XXth century by means of non-negative matrix factorization. Proceedings of the III International Conference on Language Engineering and Applied Linguistics (PRLEAL-2019), 2020, pp. 321–339.

35. Zemlyansky S. A., Axyonov S. V., Lyzin I. A., Berestneva O. G. Topic modeling in the context of medical texts. Proceedings of TUSUR University, 2021, iss. 4, pp. 58–64. (in Russ.) DOI 10.21293/1818-0442-2021-24-4-58-64