[en] Continuous integration, delivery and deployment (CI/CD) is used to support the collaborative software development process. CI/CD tools automate a wide range of activities in the development workflow such as testing, linting, updating dependencies, creating and deploying releases, and so on. Previous quantitative studies have revealed important changes in the landscape of CI/CD usage, with the increasing popularity of cloud-based services, and many software projects migrating to other CI/CD tools. In order to understand the reasons behind these changes in CI/CD usage, this paper presents a qualitative study based on in-depth interviews with 22 experienced software practitioners reporting on their usage, co-usage and migration of 31 different CI/CD tools. Following an inductive and deductive coding process, we analyse the interviews and found a high amount of competition between CI/CD tools. We observe multiple reasons for co-using different CI/CD tools within the same project, and we identify the main reasons and detractors for migrating to different alternatives. Among all reported migrations, we observe a clear trend of migrations away from Travis and migrations towards GitHub Actions and we identify the main reasons behind them.
Disciplines :
Sciences informatiques
Auteur, co-auteur :
Rostami mazrae, Pooya ; Université de Mons - UMONS > Faculté des Sciences > Service de Génie Logiciel
Mens, Tom ; Université de Mons - UMONS > Faculté des Sciences > Service de Génie Logiciel
Golzadeh, Mehdi ; Université de Mons - UMONS > Faculté des Sciences > Service de Génie Logiciel
Decan, Alexandre ; Université de Mons - UMONS > Faculté des Sciences > Service de Génie Logiciel
Langue du document :
Anglais
Titre :
On the usage, co-usage and migration of CI/CD tools: A qualitative analysis
Abdalkareem R, Mujahid S, Shihab E (2020) A machine learning approach to improve the detection of CI skip commits. IEEE Trans Softw Eng
Abdalkareem R, Mujahid S, Shihab E, Rilling J (2019) Which commits can be CI skipped? IEEE Trans Softw Eng 47(3):448–463 DOI: 10.1109/TSE.2019.2897300
Ampatzoglou A, Bibi S, Avgeriou P, Verbeek M, Chatzigeorgiou A (2019) Identifying, categorizing and mitigating threats to validity in software engineering secondary studies. Inf Softw Technol 106:201–230 DOI: 10.1016/j.infsof.2018.10.006
Beck K (2000) Extreme programming explained: embrace change. Addison-Wesley Professional
Beller M, Gousios G, Zaidman A (2017) Oops, my tests broke the build an explorative analysis of Travis CI with GitHub. In: International conference on mining software repositories (MSR), pp 356–367
Bernardo JH, da Costa DA, Kulesza U (2018) Studying the impact of adopting continuous integration on the delivery time of pull requests. In: International conference on mining software repositories (MSR), pp 131–141. IEEE
Betz RM, Walker RC (2013) Implementing continuous integration software in an established computational chemistry software package. In: International workshop on software engineering for computational science and engineering (SE-CSE), pp 68–74. IEEE
Chen L (2015) Continuous delivery: Huge benefits, but challengs too. IEEE Software - Special Issue on Release Engineering 32(2):50–54
Chen T, Zhang Y, Chen S, Wang T (2021) Let’s supercharge the workflows: an empirical study of GitHub actions. In: International conference on software quality, reliability and security companion (QRS-C), pp 01–10. IEEE
Decan A, Mens T, Mazrae PR, Golzadeh M (2022) On the use of GitHub actions in software development repositories. In: International conference on software maintenance and evolution (ICSME)
Duvall PM, Matyas S, Glover A (2007) Continuous integration: improving software quality and reducing risk. Addison-Wesley Professional
Elazhary O, Werner C, Li ZS, Lowlind D, Ernst NA, Storey M-A (2022) Uncovering the benefits and challenges of continuous integration practices. IEEE Trans Softw Eng 48(7):2570–2583 DOI: 10.1109/TSE.2021.3064953
Esfahani H, Fietz J, Ke Q, Kolomiets A, Lan E, Mavrinac E, Schulte W, Sanches N, Kandula S (2016) CloudBuild: Microsoft’s distributed and caching build service. In: International conference on software engineering (ICSE), pp 11–20
Foundjem A, Constantinou E, Mens T, Adams B (2022) A mixed-methods analysis of micro-collaborative coding practices in OpenStack. Empir Softw Eng 27(5):120 DOI: 10.1007/s10664-022-10167-w
Fowler M, Foemmel M (2006) Continuous integration
Fusch PI, Ness LR (2015) Are we there yet? Data saturation in qualitative research. The Qualitative Report, vol 20(9)
Gallaba K, Junqueira Y, Ewart J, Mcintosh S (2022) Accelerating continuous integration by caching environments and inferring dependencies. IEEE Trans Softw Eng 48(6):2040–2052 DOI: 10.1109/TSE.2020.3048335
Ghaleb TA, Da Costa DA, Zou Y (2019) An empirical study of the long duration of continuous integration builds. Empir Softw Eng 24(4):2102–2139 DOI: 10.1007/s10664-019-09695-9
Gmeiner J, Ramler R, Haslinger J (2015) Automated testing in the continuous delivery pipeline: a case study of an online company. In: International conference on software testing, verification and validation workshops (ICSTW), pp 1–6. IEEE
Golzadeh M, Decan A, Mens T (2022) On the rise and fall of CI services in GitHub. In: International conference on software analysis, evolution and Reengineering (SANER)
Guest G, Bunce A, Johnson L (2006) How many interviews are enough? An experiment with data saturation and variability. Field Methods 18 (1):59–82 DOI: 10.1177/1525822X05279903
Gupta Y, Khan Y, Gallaba K, McIntosh S (2017) The impact of the adoption of continuous integration on developer attraction and retention. In: International Conference on Mining Software Repositories (MSR), pages 491–494. IEEE
Hilton M, Nelson N, Tunnell T, Marinov D, Dig D (2017) Trade-offs in continuous integration: assurance, security, and flexibility. In: Joint meeting on foundations of software engineering (FSE), pp 197–207
Hilton M, Tunnell T, Huang K, Marinov D, Dig D (2016) Usage, costs, and benefits of continuous integration in open-source projects. In: International conference on automated software engineering (ASE), pp 426–437. IEEE
Holmstrom H, Conchúir EÓ, Agerfalk J, Fitzgerald B (2006) Global software development challenges: A case study on temporal, geographical and socio-cultural distance. In: International conference on global software engineering (ICGSE), pp 3–11. IEEE
Jin X, Servant F (2020) A cost-efficient approach to building in continuous integration. In: International conference on software engineering (ICSE), pp 13–25. IEEE
Jin X, Servant F (2021) What helped, and what did not? An evaluation of the strategies to improve continuous integration. In: International conference on software engineering (ICSE), pp 213–225. IEEE
Jin X, Servant F (2022) Which builds are really safe to skip? Maximizing failure observation for build selection in continuous integration. J Syst Softw 188:111292 DOI: 10.1016/j.jss.2022.111292
Kim M, Zimmermann T, DeLine R, Begel A (2016) The emerging role of data scientists on software development teams. In: International conference on software engineering (ICSE), pp 96–107. IEEE
Kinsman T, Wessel M, Gerosa MA, Treude C (2021) How do software developers use GitHub actions to automate their workflows?. In: International conference on mining software repositories (MSR)
Kulas M, Borelli JL, Gässler W, Peter D, Rabien S, de Xivry GO, Busoni L, Bonaglia M, Mazzoni T, Rahmer G (2014) Practical experience with test-driven development during commissioning of the multi-star AO system ARGOS. In: Software and Cyberinfrastructure for Astronomy III, vol 9152, pp 110–119. SPIE
Leppänen M, Mäkinen S, Pagels M, Eloranta V-P, Itkonen J, Mäntylä MV, Männistö T (2015) The highways and country roads to continuous deployment. IEEE Softw 32(2):64–72 DOI: 10.1109/MS.2015.50
Li Z, Liu W, Chen H, Wang X, Liao X, Xing L, Zha M, Jin H, Zou D (2022) Robbery on: DevOps understanding and mitigating illicit cryptomining on continuous integration service platforms. In: IEEE symposium on security and privacy (SP), pp 2397–2412. IEEE
Lu J, Yang Z, Qian J (2014) Implementation of continuous integration and automated testing in software development of smart grid scheduling support system. In: International conference on power system technology, pp 2441–2446. IEEE
Machiraju V (2021) Change in Azure pipelines grant for public projects. https://devblogs.microsoft.com/devops/change-in-azure-pipelines-grant-for-public-projects/. Accessed Oct 14 2022
Maxwell J (1992) Understanding and validity in qualitative research. Harvard educational review 62(3):279–301 DOI: 10.17763/haer.62.3.8323320856251826
Mendy M, Rios N, Rybinski M (2020) The new pricing model for travis-ci.com. https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing. Accessed Oct 14 2022
Meyer AN, Barr ET, Bird C, Zimmermann T (2019) Today was a good day: the daily life of software developers. IEEE Trans Softw Eng 47(5):863–880 DOI: 10.1109/TSE.2019.2904957
Rahman A, Agrawal A, Krishna R, Sobran A (2018) Characterizing the influence of continuous integration: empirical results from 250+ open source and proprietary projects. In: ACM SIGSOFT international workshop on software Analytics, pp 8–14
Ralph P, Tempero E (2018) Construct validity in software engineering research and software metrics. In: International conference on evaluation and assessment in software engineering, pp 13–23
Rausch T, Hummer W, Leitner P, Schulte S (2017) An empirical analysis of build failures in the continuous integration workflows of Java-based open-source software. In: International conference on mining software repositories (MSR), pp 345–355. IEEE
Russel Bernard H, Wutich A, Gery WR (2016) Analyzing qualitative data: systematic approaches. SAGE Publications 2nd edn.
Savor T, Douglas M, Gentili M, Williams L, Beck K, Stumm M (2016) Continuous deployment at Facebook and OANDA. In: International conference on software engineering (ICSE), pp 21–30. IEEE
Shahin A, Babar MA, Zhu L (2017) Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices. IEEE Access 5:3909–3943 DOI: 10.1109/ACCESS.2017.2685629
Soares E, Sizilio G, Santos J, da Costa DA, Kulesza U (2022) The effects of continuous integration on software development: a systematic literature review. Empir Softw Eng 27(3):1–61 DOI: 10.1007/s10664-021-10114-1
Ståhl D, Bosch J (2013) Experienced benefits of continuous integration in industry software product development: a case study. In: IASTED international conference on software engineering, pp 736–743
Valenzuela-Toledo P, Bergel A (2022) Evolution of GitHub Action workflows. In: International conference on software analysis, evolution and Reengineering (SANER). IEEE
Vasilescu B, Yu Y, Wang H, Devanbu P, Filkov V (2015) Quality and productivity outcomes relating to continuous integration in GitHub. In: Joint meeting on foundations of software engineering (FSE), pp 805–816
Vassallo C, Palomba F (2018) Continuous refactoring in CI: a preliminary study on the perceived advantages and barriers. In: International conference on software maintenance and evolution (ICSME), pp 564–568. IEEE
Vassallo C, Proksch S, Gall HC, Di Penta M (2019) Automated reporting of anti-patterns and decay in continuous integration. In: International conference on software engineering (ICSE), pp 105–115. IEEE
Widder DG, Hilton M, Kästner C, Vasilescu B (2019) A conceptual replication of continuous integration pain points in the context of Travis CI. In: Joint meeting on european software engineering conference and symposium on the foundations of software engineering (ESEC/FSE), pp 647–658
Widder D, Vasilescu B, Hilton Ml, Kästner C (2018) I’m leaving you, Travis: a continuous integration breakup story. In: International conference on mining software repositories (MSR), pp 165–169. IEEE
Zampetti F, Scalabrino S, Oliveto R, Canfora G, Di Penta M (2017) How open source projects use static code analysis tools in continuous integration pipelines. In: International conference on mining software repositories (MSR), pp 334–344. IEEE