Observing bots in the wild: A quantitative analysis of a large open source ecosystem

[en] The GitHub platform allows repository maintainers and contributors to automate their activities either through GitHub Actions workﬂows or by making use of bots. The second category concerns automated actors that interact with regular users in GitHub repositories, and includes GitHub’s internal automation services (e.g., Dependabot), GitHub Apps, and bot accounts. There has been little to no empirical research on the presence and use of such bots in large software ecosystems. Also, no former studies have compared how bot accounts and GitHub Apps are used within and across GitHub repositories. This paper addresses this gap, through an empirical analysis of the use of bot accounts and GitHub Apps in NumFOCUS, a large open source software ecosystem for data science. We analyse, over a three-month period, the activity sequences of 853 contributors across 59 GitHub organisations hosting 1,169 repositories. Using state-of-the-art bot identiﬁcation approaches we identify activities of 802 humans, 34 bot accounts, 13 GitHub Apps and 4 internal automation services. Based on this dataset we reveal behavioural differences in NumFOCUS between bots and humans on the one hand, and between different bot categories on the other hand.

Disciplines :

Computer science

DOI :

10.1109/BotSE67031.2025.00008

Author, co-author :

Chidambaram, Natarajan ; Université de Mons - UMONS > Faculté des Sciences > Service de Génie Logiciel

Mens, Tom ; Université de Mons - UMONS > Faculté des Sciences > Service de Génie Logiciel

Language :

English

Title :

Observing bots in the wild: A quantitative analysis of a large open source ecosystem

Publication date :

26 April 2025

Event name :

6th International Workshop on Bots in Software Engineering

Event organizer :

Ahmad Abdellatif

Event place :

Canada

Event date :

27 April 2025

Audience :

International

Main work title :

International Workshop on Bots in Software Engineering

Publisher :

IEEE, United States

Peer reviewed :

Peer reviewed

Research unit :

S852 - Génie Logiciel

Research institute :

R300 - Institut de Recherche en Technologies de l'Information et Sciences de l'Informatique

Funders :

Wallonia
F.R.S.-FNRS - Fonds de la Recherche Scientifique

Funding number :

DigitalWallonia4.AI research project ARIAC grant 2010235; T.0149.22; J.0147.24

Funding text :

This work is supported by DigitalWallonia4.AI research project ARIAC grant 2010235, and the Fonds de la Recherche Scientifique - FNRS under grants T.0149.22 and J.0147.24.

Data Set :

A Dataset of bots and human activities in NumFOCUS

Dataset accompanying the paper

Available on ORBi UMONS :

since 12 December 2024

Statistics

Number of views

150 (22 by UMONS)

Number of downloads

27 (6 by UMONS)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenAlex citations

Bibliography

L. Dabbish, C. Stuart, J. Tsay, and J. Herbsleb, “Social coding in GitHub: Transparency and collaboration in an open software repository,” in International Conference on Computer Supported Cooperative Work (CSCW). ACM, 2012, pp. 1277–1286.
A. Decan, T. Mens, P. R. Mazrae, and M. Golzadeh, “On the use of GitHub Actions in software development repositories,” in International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2022, pp. 235–245.
M. Golzadeh, A. Decan, and T. Mens, “On the rise and fall of CI services in GitHub,” in International Conference on Software Analysis, Evolution and Reengineering (SANER), 2022, pp. 662–672.
A. Decan, T. Mens, and H. Onsori Delicheh, “On the outdatedness of workflows in the GitHub Actions ecosystem,” Journal of Systems and Software, vol. 206, p. 111827, 2023.
P. Rostami Mazrae, T. Mens, M. Golzadeh, and A. Decan, “On the usage, co-usage and migration of CI/CD tools: A qualitative analysis,” vol. 28, no. 2, p. 52.
M. Golzadeh, T. Mens, A. Decan, E. Constantinou, and N. Chidambaram, “Recognizing bot activity in collaborative software development,” IEEE Software, vol. 39, no. 5, pp. 56–61, 2022.
N. Cassee, C. Kitsanelis, E. Constantinou, and A. Serebrenik, “Human, bot or both? A study on the capabilities of classification models on mixed accounts,” in International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2021.
T. Dey, S. Mousavi, E. Ponce, T. Fry, B. Vasilescu, A. Filippova, and A. Mockus, “Detecting and characterizing bots that commit code,” in International Conference on Mining Software Repositories (MSR). ACM, 2020, pp. 209–219.
M. Golzadeh, A. Decan, and T. Mens, “Evaluating a bot detection model on git commit messages,” in Belgium-Netherlands Software Evolution Workshop (BENEVOL), vol. 2912. CEUR Workshop Proceedings, 2020.
M. Golzadeh, A. Decan, D. Legay, and T. Mens, “A ground-truth dataset and classification model for detecting bots in GitHub issue and PR comments,” Journal of Systems and Software, vol. 175, 2021.
A. Abdellatif, M. Wessel, I. Steinmacher, M. A. Gerosa, and E. Shihab, “BotHunter: An approach to detect software bots in GitHub,” in International Conference on Mining Software Repositories (MSR), 2022, pp. 6–17.
N. Chidambaram, T. Mens, and A. Decan, “RABBIT: A tool for identifying bot accounts based on their recent GitHub event history,” in International Conference on Mining Software Repositories (MSR). ACM, 2024.
N. Chidambaram, A. Decan, and T. Mens, “A dataset of bot and human activities in GitHub,” in International Conference on Mining Software Repositories (MSR). IEEE/ACM, 2023, pp. 465–469.
N. Chidambaram, T. Mens, and A. Decan, “A bot identification model and tool based on GitHub activity sequences,” Journal of Systems and Software, vol. 221, March 2025.
M. Golzadeh, A. Decan, and N. Chidambaram, “On the accuracy of bot detection techniques,” in International Workshop on Bots in Software Engineering (BotSE). IEEE, 2022.
Z. Wang, Y. Wang, and D. Redmiles, “From specialized mechanics to project butlers: The usage of bots in open source software development,” IEEE Software, vol. 39, no. 5, pp. 38–43, 2022.