Exploring Developer Perceptions on the Potential of CI Recommendation Systems

Our paper, "Exploring Developer Perceptions on the Potential of CI Recommendation Systems," has been accepted to the Registered Reports Track at ESEM 2026 in Munich. The work is co-authored with Professor Taher A. Ghaleb at Trent University.

This is the empirical follow-up to the vision paper we presented at FSE Companion 2026. That paper argued that CI adoption should be a deliberate decision rather than a default. This one goes looking for the evidence.

Why a registered report

In a registered report, the research questions and the full methodology are peer-reviewed before any data is collected. Acceptance is based on whether the question matters and whether the design can actually answer it — not on whether the results turn out to be interesting.

For a survey study, that ordering matters. It commits us to our analysis plan in advance, so the findings cannot be quietly reshaped after the responses arrive.

The gap we are trying to close

Continuous Integration is now routine infrastructure. Adoption has climbed from roughly 40% of open-source projects to over 50% in the GitHub Actions era. But ease of adoption has not resolved the harder question of fit:

• 18.8% of Java projects abandon CI entirely after adopting it.
• 52% of developers say they want easier configuration.
• Choosing between services still means weighing tangled trade-offs in features, pricing, and workflow.

Prior work has documented these barriers thoroughly — through systematic reviews, pain-point taxonomies, and analyses of technical and social factors. Three gaps remain, and they are the ones we target:

1. Whether adoption decisions are driven by genuine project needs or by social influence.
2. Whether developers see CI as universally necessary or context-dependent, and what stops those who see it as inappropriate.
3. Whether developers would actually accept an automated system that advises on CI suitability, services, and configuration.

There is a structural reason these gaps persist. Most prior surveys sampled only CI users, or only experienced adopters. If you only ask people who adopted CI why they adopted it, you cannot learn anything about the people who looked at it and declined.

Three research questions

RQ1 — What drives CI adoption and service selection? We separate need-driven motivations from socially-influenced ones, and ask which criteria developers prioritise when choosing a specific service, and how they switch between services or run several at once.

RQ2 — Is CI universally necessary, or context-dependent? We ask whether developers consider CI essential for every project or valuable only in particular contexts, what reasoning guides selective adoption, and which barriers hold back non-adopters.

This question directly shapes what a recommendation system should even do: suggest CI for everything, or first assess whether a given project is a good candidate.

RQ3 — Would developers find a CI recommendation system useful? We measure perceived usefulness, trust, and likelihood of use on Likert scales, and collect desired capabilities — service comparison, configuration templates, troubleshooting guides, integration examples — through multi-select and open-ended responses.

RQ3 is grounded in the Technology Acceptance Model, using perceived usefulness and behavioural intention, with trust added explicitly. A recommendation developers do not trust is a recommendation they will not act on.

How the study is designed

We will sample approximately 5,000 active open-source developers through the GitHub REST API, drawn from repositories with at least 10 stars and at least 10 commits in the past 12 months. For each repository we identify developers who committed within the last year and randomly select one, excluding bots and non-contactable accounts, and de-duplicating email addresses so nobody is invited twice.

That 5,000 is the number of invitations, not the expected yield. We anticipate 250–500 completed responses.

The survey is deliberately split so that non-adopters are first-class participants rather than an afterthought:

• Part A — CI users (18 questions): adoption motivations, service selection, switching behaviour, views on CI universality, and perceived value of recommendation systems.
• Part B — non-CI users (14 questions): views on CI universality, adoption barriers, and perceived value of recommendation systems.

Both parts converge on RQ3, which lets us compare how adopters and non-adopters judge the same proposed tooling.

For analysis, we will compute descriptive statistics, compare need-driven against socially-influenced motivation distributions using Chi-square tests, and apply Mann-Whitney U tests with effect size r for ordinal Likert outcomes.

Why this matters now

AI agents are increasingly being pointed at CI/CD configuration. That makes the underlying question more urgent, not less: a system that generates a pipeline faster is not helpful if the project never needed that pipeline, or if it needed a different service entirely.

Before building tools that recommend, we should understand how developers currently decide — and whether they would trust a recommendation at all. That is what this study is for.

Acknowledgment

My thanks again to Professor Taher A. Ghaleb, whose guidance shaped both the framing of this study and the discipline of committing to a method before seeing the data.