Methods for Improving Inter-Annotator Agreement in Modelling Argumentation Structure of a Text

The efficiency of machine learning algorithms applied to various NLP tasks, particularly the automatic extraction of argumentation structures from texts, heavily depends on datasets annotation consistency. As a rule, annotation of large corpora for algorithms training relies on a joint effort of several annotators, detrimental to its consistency. The present article describes methods of modifying argumentation annotations for improving their consistency. Annotations in the study take form of argumentation graphs constructed in accordance with the Argument Interchange Format standard, using ArgNetBank Studio tools. These graphs contain nodes with statements text (for premises, conclusions) aggregated into arguments with edges through scheme nodes (which correspond to argument types from Walton’s compendium). The study proposes methods for automatic modification of graphs in case of annotators’ disagreement in identifying argumentative statements and arguments of specific types. At the level of statements, consistency improvement relies on two procedures: 1) removal of leaf nodes present only in one of the graphs; 2) transformation of two-argument sequences into one-argument upon fulfillment of specific conditions. For argument types, agreement increases through replacing infrequent narrow-focused schemes with more common general ones, as well as by applying a hierarchical system of schemes substitution rules based on their functional classification (a functional group unites schemes similar in semantic and textual expression properties). A typical quantitative way of measuring annotation consistency is employing inter-annotator agreement coefficients. Calculation of the Krippendorff α agreement coefficient demonstrates a considerable increase of consistency upon modifying a corpus of 160 annotations for 80 short scientific articles in Russian: the increase equals 25 % for argumentative statements and 19 % for argument types. However, an experiment in identifying argumentative sentences with an MLP classifier shows only a 5 % increase of F-measure after modifying the corpus. The increase of F-measure for identifying four argumentation types under analysis (Cause to Effect, Verbal Classification, Example, Practical Reasoning) is even less: no more than 2 %. We arrive at a conclusion that an improvement of the inter-annotator agreement coefficient is by itself insufficient for a considerable increase in identification efficiency values in practice.

1. Castro S. Fast Krippendorff: Fast computation of Krippendorff’s alpha agreement measure, 2017. https://github.com/pln-fing-udelar/fast-krippendorff.

2. Fishcheva I., Kotelnikov E. Cross-Lingual Argumentation Mining for Russian Texts. Proc. Of the 8th International Conference “Analysis of Images, Social Networks and Texts” (Kazan), 2019, pp. 134–144.

3. Fishcheva I., Goloviznina V., and Kotelnikov E. Traditional machine learning and deep learning models for argumentation mining in russian texts. Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialog-2021”, 2021, pp. 246–258.

4. Kotelnikov E., Loukachevitch N., Nikishina I., Panchenko A. RuArg-2022: argument mining evaluation. Proceedings of the International Conference “Dialogue 2022”, 2022, pp. 1–16.

5. Krippendorff K. Content analysis: An introduction to its methodology, 3rd edition. Thousand Oaks, CA: Sage, 2013.

6. Pimenov I. S. Compatibility of Arguments from Different Functional Groups in Scientific Texts. Philology. Theory & Practice. Tambov, Gramota, 2022, vol. 11. pp. 3672−3680. (in Russ.)

7. Pimenov I. S. Analyzing Disagreements in Argumentation Annotation of Scientific Texts in Russian Language. Vestnik NSU. Series: Linguistics and Intercultural Communications, 2023, vol. 21, no. 2, pp. 89–104. (in Russ.) DOI 10.25205/1818-7935-2023-21-2-89-104.

8. Pimenov I. S., Salomatina N. V. An Automatic Method for Standartizing Argumentative Annotations across Annotators. Proc. 2024 IEEE 25th International Conference of Young Professionals in Electron Devices and Materials (EDM), 28 June 2024 – 02 July 2024. DOI: 10.1109/EDM61683.2024.10615176.

9. Rahwan I., Reed C. The argument interchange format. Argumentation in artificial intelligence / ed. Rahwan I. and Simari G., Springer, 2009, pp. 383–402.

10. Sidorova Е. А., Akhmadeeva I. R., Zagorulko Yu. A., Sery A. S., Shestakov V. K. Research platform for the study of argumentation in popular science discourse. Ontology design, 2020, vol. 10, no. 4 (38), pp. 489–502. (in Russ.)

11. Skeppstedt M., Peldzus A., Stede M. More or less controlled elicitation of argumentative text: Enlarging a microtext corpus via crowdsourcing. Proceedings of the 5th Workshop on Argument Mining, Brussels, Belgium, 2018, pp. 155–163.

12. Stab C. and Gurevych I. Annotating argument components and relations in persuasive essays. Proceedings of COLING 2014а, the 25th International Conference on Computational Linguistics: Technical Papers, 2014a, pp. 1501–1510.

13. Stab C. and Gurevych I. Identifying argumentative discourse structures in persuasive essays. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014b, pp. 46–56.

14. Stab C. and Gurevych I. Parsing Argumentation Structures in Persuasive Essays. Computational Linguistics, 2017, vol. 43, no. 3, pp. 619–659.

15. Teruel M., Cardellino C., Cardellino F., Alemany L., Villata S. Increasing Argument Annotation Reproducibility by Using Inter-annotator Agreement to Improve Guidelines. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). Miyazaki, Japan, 2018.

16. Walton D., Reed C., Macagno F. Argumentation schemes Fundamentals of critical argumentation. New York, Cambridge University Press, 2008, 443 p.