After evaluating over 300 user stories from multiple stakeholders, the Community Platform Engineering (CPE) team have aligned on a decision for the git forge that CPE will operate for the coming years. We are opting for GitLab for our dist git and project hosting and will continue to run pagure.io with community assistance.

A lot of comments and concerns were raised about the suitability of GitHub as a forge of choice. The preference from all stakeholders (Fedora, CentOS, RHEL, CPE) is that GitHub is not a contender and not a preference. With that in mind, we have decided to not analyse it as an option and respect the wider wishes of our stakeholders. Therefore the rest of this analysis focuses on Pagure versus GitLab as our choice.

Looking at the user story list, we have a picture of a standard set of practices that users expect to have from a git forge. The basics of storing code, accessing it, merging, forking and the traditional git workflow are satisfied by both gorges under investigation. 

A key requirement coming to us is security. The need for HTTPS pushes and the need for more stringent branch control via protected & private branches are key operating requirements of the CentOS stakeholders. The need to interface with internal and external users in a private capacity whereby embargoed content can be worked on in private is a necessary requirement. 

Another key requirement is usability and accessibility. It is clear that our current forge solution is used as a mixture of ticket tracker, work planning, code repository, and storage of documents and other artifacts. The barrier to usage needs to be low to attract drive by users and a strong representation was made for the need to have more accessible ways to interface with the system from a GUI to a command line client.

Developer-centric needs came from multiple sources. Integrations with daily workflow, integrations within the IDE, integrations in an always-ready and always-on approach (SLA requirements were high) as well as the ability to use the forge as a means to improve the code base (auto notifications of issues, interactive PR reviews etc.) and way of working by providing analytical output was also raised.

A big factor in a decision here needs to be both the immediate usability to meet stakeholder needs that includes an immovable deliverable for CentOS Stream which CPE must deliver by the end of the year. 

Another major factor is the stability, availability and responsiveness of the platform chosen. While no forge meets the full suite of requirements, the issue of stability, availability and some of the richer features that were requested are currently not available in Pagure. GitLab provides the most feature-rich experience out of the box and the recommendation of the CPE management is to opt for GitLab as our chosen forge for dist-git and general project hosting. For pagure.io, we want to offer it to the community to maintain. CPE would provide “power and ping” and the rest of it will be up to the community willing to do the work. If no one steps up to pick the maintenance of pagure.io, it will be a candidate application to sunset. Some top level requirements which helped us arrive at this decision:

• There is a need for CentOS Stream to integrate with a kernel workflow that is an automated bot driven merging solution (merge trains). This allows for richer CI capabilities and minimises the need for human interaction
• GitLab provides subgroups allowing for more granular permissions for repos
• GitLab allows for project planning capability which could make multiple trackers such as Taiga redundant, allowing for the planning and tracking to reside within the repo. It would enrich the current ticket based solution that Pagure has evolved into for some groups
• 24/7 availability in an SLA model and not hosted by the CPE team freeing up resourcing and removing the need to staff a dedicated team for a git forge SLA which would necessitate a follow-the-sun ops model and a heavy investment in stability and observability of the Pagure solution.

The opportunity cost to invest our finite resources into bringing Pagure up to the minimum standard that we require by the end of the year would mean feature starving both Fedora and CentOS for the next 18-24 months as we strive for the optimal standard. As a team, we spend 40% of our available resources on keeping the lights on day to day with a very small amount of that improving our technical debt situation. We are spending 30% of our team on delivering CentOS Stream. The available bandwidth for the team is not at a point that we could safely and with confidence deliver the required features to make Pagure work as our forge of choice.

It additionally would have a longer term impact with our lights on work needing to expand to move Pagure to an SLA, tilting our resourcing plan for that body of work towards 60% of our capacity. We feel this is not a responsible decision that we can make as the inward investment in a forge is not something that we can do at the expense of planned initiatives that are on our backlog. Some of them include a better packager workflow, more investment in CI/CD to remove CPE from manual work and empower the community to do more things in our infrastructure, more observability and monitoring of our infra and services, movement of services towards the Cloud to make use of a modern tech stack and that’s before we consider immovable service progression that we simply have to undertake, for example, the new AAA system.

However, we do not want to abandon Pagure and our plan going forward is thus:

• Offer the maintenance of pagure.io to anyone in the community interested in leading it.
• Engage with GitLab on the possibility of a SaaS offering so that CPE can attain key requirements of uptime, availability and throughput as well as ensuring tooling integrations (such as Fedora Messaging among others) are preserved. Legal considerations with respect to control of code will be our first discussion point with them enabling us to make a SaaS versus self-hosted decision.
• Keep Pagure running with our oversight while we analyse a sunset timeline which will give a minimum of 12 months notice once we have a plan firmed up. We will fix blocker bugs, address critical vulnerabilities and keep the lights on in the same manner that we have committed to over the last 14 months where Pagure has not been a staffed and supported initiative.
• Where possible, when we have to update our tooling, we will attempt to refactor our tooling to be forge-agnostic, allowing our communities the choice of storing their code on GitLab or continuing to use pagure.io
• Watch closely for collaboration with other Communities on Pagure and provide them with guidance and oversight to help the Pagure community grow. We recognise that this is a growing and unique ecosystem and we genuinely want to see it succeed and will do our best to support it in that capacity. To that end we will publish the roadmap difference between Pagure and GitLab to allow the Community to focus on feature enhancements to bridge that gap.
• Facilitate our Communities and assist them in standing up a version of Pagure that can be driven and maintained by the community allowing a pure open source principles approach for those who seek it.

We recognise how difficult a decision this is and we empathise with the emotional attachment to Pagure. It is why we want to have a mutually-beneficial approach to ultimately allow Pagure to grow and flourish and allow our community members to setup and work with any Forge they wish. This ultimately allows the CPE team to focus on adding value to a greater scale of initiatives. This approach allows us to focus on value added services and initiatives that will benefit a large percentage of our communities instead of focusing on a singular foundational service which would ultimately consume our finite resourcing and limit our impact on both Communities.

— Jim and Leigh