This is a weekly report from the CPE (Community Platform Engineering)
Team. If you have any questions or feedback, please respond to this
report or contact us on #redhat-cpe channel on

We (CPE team) will be joining Fedora Social Hour on Jan 27th.
Looking forward to seeing a lot of you!

Highlights of the week

Infrastructure & Release Engineering

Goal of this initiative

Purpose of this team is to take care of day to day business regarding
CentOS and Fedora Infrastructure and Fedora release engineering work.
It’s responsible for services running in Fedora and CentOS
infrastructure and preparing things for the new Fedora release
(mirrors, mass branching, new namespaces etc.). The ARC (which is a
subset of the team) investigates possible initiatives that CPE might
take on.


Fedora Infra

  • All koji builders/hubs upgraded to F35 and ready for mass rebuild ( 🤞 )
  • Additional s390x disk space appeared, so added 10 more s390x builders.
  • Fixed IPA issue with certs ( known upgrade bug)
  • Difficult container builds failing issue solved.

CentOS Infra including CentOS CI

  • CentOS Linux 8 EOL plan
  • Hardware issues (storage box, 64 compute nodes for CI infra)
  • Kmods SIG DuD discussion (koji plugin vs external script)
  • CI storage for ocp/openshift migration completed and working faster/better !
  • CentOS CI tenants Survey (for the upcoming DC move)

Release Engineering

  • Mass rebuild starts today
  • Several rawhide issues fixed and composes have been good the last few days.

CentOS Stream

Goal of this initiative

This initiative is working on CentOS Stream/Emerging RHEL to make this
new distribution a reality. The goal of this initiative is to prepare
the ecosystem for the new CentOS Stream.


  • The NFV repo was added to CentOS Stream 8, work is happening now on the repo files in centos-release
  • Module branching work is ongoing
  • Libffi is causing some interesting breakage in ELN
  • GCC bugs in ELN
  • Koji/brew Inheritance discussions are still happening
  • Testing Content Resolver with production data before deployment

Datanommer/Datagrepper V.2

Goal of this initiative

The datanommer and datagrepper stacks are currently relying on fedmsg which
we want to deprecate.
These two applications need to be ported off fedmsg to fedora-messaging.
As these applications are ‘old-timers’ in the fedora infrastructure, we would
also like to look at optimizing the database or potentially redesigning it to
better suit the current infrastructure needs.
For a phase two, we would like to focus on a DB overhaul.


  • It’s done! Data is migrated, the new code is now running in prod.

CentOS Duffy CI

Goal of this initiative

Duffy is a system within CentOS CI Infra which allows tenants to provision and
access bare metal resources of multiple architectures for the purposes of
CI testing.
We need to add the ability to checkout VMs in CentOS CI in Duffy. We have
OpenNebula hypervisor available, and have started developing playbooks which
can be used to create VMs using the OpenNebula API, but due to the current state
of how Duffy is deployed, we are blocked with new dev work to add the
VM checkout functionality.


  • Legacy API
  • Node Pools & Ansible Backend

Image builder for Fedora IoT

Goal of this initiative

Integration of Image builder as a service with Fedora infra to allow Fedora IoT
migrate their pipeline to Fedora infra.


  • Officially kicked off this week
  • Fact finding at the moment
  • Met with Peter Robinson and team from Fedora IoT
    • Need to figure out a way to run their pipeline
    • At least 1 koji plugin to be written + deployed
  • Meeting with Image Builder team tomorrow
    • They are currently blocked by auth, development underway
    • Need to get an idea of their API and what they expect from us


Goal of this initiative

This initiative is to separate Bodhi into multiple sub packages,
fix integration and unit tests in CI, fix dependency management,
and automate part of the release process.
Read ARC team findings in detail at:


  • splitting the codebase into separate python packages
  • migrating from CentOS CI to Zuul


Goal of this initiative

Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special Interest
Group that creates, maintains, and manages a high quality set of additional
packages for Enterprise Linux, including, but not limited to,
Red Hat Enterprise Linux (RHEL), CentOS and Scientific Linux (SL), Oracle Linux (OL).

EPEL packages are usually based on their Fedora counterparts and will never
conflict with or replace packages in the base Enterprise Linux distributions.
EPEL uses much of the same infrastructure as Fedora, including buildsystem,
bugzilla instance, updates manager, mirror manager and more.


  • epel9 up to 1346 source packages (increase of 188 from last week)
  • Two talks submitted and accepted for the February CentOS Dojo
    • State of EPEL
    • EPEL Packaging Hackfest

Kindest regards,
CPE Team