This is a summary of the work done on initiatives by the CPE Team. Every quarter, the team together with CentOS and Fedora community representatives, choose initiatives to work on. The CPE Team is then split into multiple smaller sub-teams for initiatives and a dedicated team for day to day infrastructure and release engineering work.

Following is the list of sub-teams in this quarter:

  • Infra & Releng
  • CentOS Stream
  • OSCI – Distrobaker monitoring
  • EPEL
  • CentOS Duffy CI

Infra & Releng


This team takes care of day to day business regarding CentOS and Fedora Infrastructure and Fedora release engineering work. It’s responsible for services running in Fedora and CentOS infrastructure and preparing things for the new Fedora release (mirrors, mass branching, new namespaces etc.). This sub-team also investigates possible future initiatives.

Issue trackers: Fedora Infrastructure | CentOS Infrastructure | Fedora Release Engineering

Documentation: Fedora Infrastructure | CentOS Infrastructure | Fedora Release Engineering

Members of sub-team for Q4 2021

  • Mark O’Brien (Team Lead) (Fedora Operations, CentOS Operations) (mobrien)
  • Kevin Fenzi (Team Lead) (Fedora Operations) (kevin)
  • Michal Konecny (Agile Practitioner) (Developer) (mkonecny)
  • Patrik Polakovic (Agile Practitioner) (Developer) (ppolakov)
  • Fabian Arrotin (CentOS Operations) (arrfab)
  • Tomas Hrcka (Release Engineering) (humaton)
  • Adam Saleh (Developer) (asaleh)
  • Aurelien Bompard (Developer) (abompard)
  • Emma Kidney (Developer) (ekidney)
  • Pedro Moura (Developer) (pmoura)
  • Ryan Lerch (Developer) (rlerch)

What the sub-team did in Q4 2021

Fedora Infrastructure

  • Fedora Infra moved their documentation to
  • Migrated discourse2fedmsg from fedmsg to fedora messaging and deployed the app in production. 
  • Migrated most koji builders to Fedora 35 (finished in Q1 2022)
  • Got CentOS stream 9 using mirrormanager (
  • Helped release Fedora 35 Beta and then Final
  • Kinoite website published.
  • Fedoraproject dnssec keys moved to sha384 from sha1
  • All wiki talk pages have been disabled. We don’t use them or read them.
  • S390x builders moved to the new z15 mainframe. Additional resources allowed us to increase kvm builders from 10 to 20.

CentOS Infrastructure

  • Upgraded openshift for CI tenants
  • Migrated the cico-workspace to CentOS 8-stream instead of CentOS 7
  • Onboarding new SIGs and adapted workflow
  • Migrated sig guide to and hosting SIGs docs (opt-in)
  • Prepared the CentOS Linux 8 EOL/decommissioning steps
  • Migrated several services in infrastructure due to some sponsors leaving the project (willing to sponsor other rebuilds now instead)
  • Rolled out (with Artwork SIG) new visual theme across all centos infra for stream 9 visual style (website, mirrors, mailing-list, etc)
  • Implemented the new mirror pool for Stream 9 (that is also used with mirrormanager)

Fedora Release Engineering


The ARC Team was looking at Bodhi and Image Builder in Q4.

  • Bodhi: 
    • Investigated doing an initiative on Bodhi
    • Looked at splitting Bodhi up into separate packages
    • Investigated decoupling Bodhi from PDC
    • Looked at dependency management
    • Concluded PDC functionality should move to dist-git instead
    • Not suitable for an initiative 
    • Package separation & dependency management work to go ahead outside of initiative work
  • Image Factory
    • Possible replacement for OZ and Image factory
    • Could be used as a service from Red Hat internal team
    • Would likely need to use our own builders for Fedora
    • Fedora IOT moving to image builder could use builders provided by Image builder as it does not support ppc or s390x
    • Initiative going ahead in Q1 2022 to use image builder for Fedora IOT
    • Potentially used for Fedora/CentOS Stream in the future

CentOS Stream


This initiative is working on CentOS Stream/Emerging RHEL to make this new distribution a reality. The goal of this initiative is to prepare the ecosystem for the new CentOS Stream. 

Issue trackers: Bugzilla

Documentation: CentOS documentation

Application URLs:

Members of sub-team for Q4 2021

  • Brian Stinson (Team Lead) (bstinson)
  • Adam Samalik (Agile Practitioner) (asamalik)
  • James Antill (jantill)
  • Johnny Hughes
  • Merlin Mathesius
  • Mohan Boddu (mboddu)
  • Petr Bokoc (pbokoc)
  • Stephen Gallagher (sgallagher)
  • Troy Dawson (tdawson)

What the sub-team did in Q4 2021

  • 1/3 of all srpms built in Stream were modules in Nov (fun/interesting fact!)
  • Automated compose checks for c9s and added repoclosure check for baseOS and app stream
  • We now report on differences between RHEL 9 and CentOS Stream 9 composes
  • Added c9s links on mirror network for downloading!
  • CI testing for SIGs enabled for c9s
  • Started work on bringing c8s and c9s closer
  • Updated the ELN Extras docs
  • Got ELN side tag builds working
  • Started work on Content Resolver buildroot integration

OSCI – Distrobaker monitoring


In Q4 some of the CPE team were able to assist the OSCI team with some open issues they had that they were finding hard to get to before the end of the year. Our team worked on a way to improve the Distrobaker monitoring to monitor side-tags and have the code update prometheus for metrics on the side-tags. Distrobaker itself is a service which rebuilds the CentOS 9 Stream Koji builds for RHEL 9 in Brew and having good metrics on the application provides useful insights as to how the service is operating.

Issue trackers: Gitlab

Documentation: README

Members of sub-team for Q4 2021

  • David Kirwan (dkirwan)
  • Lenka Segura (lenka)
  • Leonardo Rossetti (lrossett)

What the sub-team did in Q4 2021

This team managed to do everything that is described in the ‘about’ section.



Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special Interest Group that creates, maintains, and manages a high quality set of additional packages for Enterprise Linux, including, but not limited to, Red Hat Enterprise Linux (RHEL), CentOS and Scientific Linux (SL), Oracle Linux (OL).

EPEL packages are usually based on their Fedora counterparts and will never conflict with or replace packages in the base Enterprise Linux distributions. EPEL uses much of the same infrastructure as Fedora, including buildsystem, bugzilla instance, updates manager, mirror manager and more.   

Issue trackers: Pagure

Documentation: EPEL documentation

Members of sub-team for Q4 2021

  • Carl George (Team Lead) (carlwgeorge)

What the sub-team did in Q4 2021

CentOS Duffy CI


Duffy is a system within CentOS CI Infra which allows tenants to provision and access bare metal resources of multiple architectures for the purposes of CI testing.

We need to add the ability to checkout VMs in CentOS CI in Duffy. We have OpenNebula hypervisor available, and have started developing playbooks which can be used to create VMs using the OpenNebula API, but due to the current state of how Duffy is deployed, we are blocked with new dev work to add the VM checkout functionality. 

Issue trackers: Github


Members of sub-team for Q4 2021

  • Akashdeep Dhar (t0xic0der)
  • Ben Capper (bcapper)
  • Nils Philippsen (Team Lead) (nphilipp)
  • Vipul Siddharth (Agile Practitioner) (siddharthvipul1)

What the sub-team did in Q4 2021

Reimplement Duffy from the ground up (which is ongoing). It features a new, much cleaner API than the currently deployed version which allows users to allocate differently featured nodes for their CI workflows. It comes with a metaclient app which translates between users of the legacy API and the new one. The Duffy core is agnostic of the features of managed nodes (e.g. bare metal vs. VM, architecture, OS type & version) and shifts that knowledge into configurable node pools with corresponding Ansible playbooks used for provisioning and deprovisioning.

Datanommer/Datagrepper V.2


The datanommer and datagrepper stacks are currently relying on fedmsg which we want to deprecate. These two applications need to be ported off fedmsg to fedora-messaging. As these applications are ‘old-timers’ in the fedora infrastructure, we would also like to look at optimizing the database or potentially redesigning it to better suit the current infrastructure needs.

For phase two, we would like to focus on a DB overhaul.

Issue trackers: Github project

Documentation: Datagrepper

Application URLs: Datagrepper

Members of sub-team for Q4 2021

  • Aurelien Bompard (Team Lead) (abompard)
  • Ryan Lerch (ryanlerch)
  • Lenka Segura (lenkaseg)

What the sub-team did in Q4 2021

The team migrated the datanommer and datagrepper tools to use TimescaleDB as a backend, instead of plain PostgreSQL. This will greatly improve the scalability of the apps. As a reminder, datanommer stores all messages ever sent to our message bus (and that’s a lot of messages), and datagreppers is a web UI and API to query this database.

FCOS OpenShift migration


Enable the Fedora CoreOS to move their pipeline from the CentOS CI OCP4 cluster to the newly deployed Fedora infra OCP4 cluster.

Issue trackers: Fedora Infra tracker

Documentation: Ansible playbook

Application URLs: Openshift project (restricted access)

Members of sub-team for Q4 2021

  • David Kirwan (Team Lead) (dkirwan)
  • James Richardson (jrichardson)
  • Lenka Segura (lenkaseg)
  • Stephen Coady (scoady)

What the sub-team did in Q4 2021

Fedora CoreOS were making use of the CentOS CI OCP4 cluster to run some of their pipelines. We reused the playbooks and roles already developed in CentOS CI Infra, to recreate the project, service account and permissions required in order to deploy their pipeline on the new Fedora infra OCP4 cluster.


If you get here, thank you for reading this. If you want to contact us, feel free to do it in #redhat-cpe channel on