Category: Infrastructure (page 1 of 7)

All articles in this category are related to the Infrastructure team in the Fedora Project.

CPE Quarterly Update Q3 2021

This is a summary of the work done on initiatives by the Community Platform Engineering (CPE) Team in Red Hat. Each quarter, the CPE Team—together with CentOS and Fedora community representatives—chooses initiatives to work on in the quarter. The CPE Team is then split into multiple smaller sub-teams that will work on chosen initiatives, plus the day-to-day work that needs to be done.

Following is the list of sub-teams in this quarter:

  • Infra & Releng
  • CentOS Stream/Emerging RHEL
  • Datanommer/Datagrepper
  • DNF Counting
  • Metrics for Apps on OpenShift

Infra & Releng


The purpose of this team is to take care of day-to-day business regarding CentOS and Fedora Infrastructure and Fedora release engineering work. It’s responsible for services running in Fedora and CentOS infrastructure and preparing things for the new Fedora release (mirrors, mass branching, new namespaces etc.). This sub-team is also investigating possible initiatives. This is done by the Advance Reconnaissance Crew (ARC), which is formed from the Infra & Releng sub-team members based on the initiative that is being investigated.

Issue trackers


Members of sub-team for Q3 2021

  • Mark O’Brien (Team Lead) (Fedora Operations, CentOS Operations) (mobrien)
  • Michal Konecny (Agile Practitioner) (Developer) (zlopez)
  • Kevin Fenzi (Fedora Operations) (nirik)
  • Fabian Arrotin (CentOS Operations) (arrfab)
  • Tomas Hrcka (Fedora Release Engineering) (humaton)
  • Lenka Segura (Developer) (lenkaseg)
  • Emma Kidney (Developer) (ekidney)
  • Ben Capper (Developer) (bcapper)

What the sub-team did in Q3 2021

Fedora Infrastructure

In addition to the normal maintenance tasks (reboots, updates for security issues, creating groups/lists, fixing application issues) we worked on a number of items:

  • Cleaned up nagios checks to stop alerting on swap on hardware machines
  • Moved the vast majority of our instances to use linux-system-roles/networking to configure networking via ansible
  • Got broken openqa-p09-worker02 back up and working with a lot of firmware upgrades and help from IBM techs.
  • Archived off ~35TB of space from our netapp to a storinator
  • Got zodbot (our IRC bot) moved to python3 and pointed to the new account system
  • Upgraded the wiki to the latest stable version.
  • Fixed an issue with OSBS building 0ad, needed a larger than default container.
  • Setup new fedora matrix hosted server rooms/etc.
  • Started on EPEL9 setup, mirroring centos9stream buildroot content, etc
  • Got vmhost-x86-copr04’s motherboard replaced and back in service.
  • Kinoite website deployed

CentOS Stream

  • prepared the new mirror network to accept CentOS Stream 9
  • modified koji/ to allow building for CentOS Stream 9, including new tags
  • importing 9-stream content
  • modified SIG process to include/support stream 9 for modified requirements (directory layout, included sources and debuginfo vs what we had before )
  • prepare the needed infra for AWS for EC2 testing and replication across all regions for CentOS Stream 9 images

CentOS common/public infrastructure

  • converting all deployed CentOS Linux 8 to CentOS Stream 8
  • relocated the armhfp community builders to other DC/hardware
  • started investigation about migrating from Pagure 5.8 on CentOS 7 to Pagure 5.13 on CentOS Stream 8
  • created doc website, and working in pairing mode to share infra knowledge within the team
  • collaboration with artwork SIG to prepare some *.dev* variants of websites to have a “playground” to test Ansible role changes directly and then having corresponding PR for deployments in .stg. and then prod
  • Business As Usual (BAU)
    •  koji tags creation
    •  hardware issues to fix/follow

CentOS CI infrastructure

  • updated openshift to 4.8.x stable branch
  • moved/onboarded new tenants on CI infra
  • moved some workload in CI infra for better resiliency and backup plans
  • expanded the existing cloud.cico (opennebula) infra with new hypervisors (x86_64)
  • reorganized the slow nfs storage box (out of warranty) with raid10 layout to speed up/help with containers in openshift (for PersistentVolumes)

Fedora Release Engineering

While taking care of day to day business like nightly composes, package retirements and unretirements, new scm requests and occasional koji issues, we worked on new Fedora release.

  • Mass rebuild of rpms and modules in Fedora Rawhide
  • Branching of Fedora 35 from Rawhide
  • Fedora Linux 35 Beta release


Investigated upgrading the Frontend Web UI for the CentOS mailing list. The investigation came to the conclusion that Mailman3, Postorius and Hyperkitty would need to be packaged for EPEL8. A new server would need to be deployed with the current CentOS mailing list migrated to it.

CentOS Stream/Emerging RHEL


This initiative is working on CentOS Stream/Emerging RHEL to make this new distribution a reality. The goal of this initiative is to prepare the ecosystem for the new CentOS Stream.

Issue trackers


Members of sub-team for Q3 2021

  • Brian Stinson (Team Lead) (bstinson)
  • Adam Samalik (Agile Practitioner) (asamalik)
  • Aoife Moloney (Product Owner) (amoloney)
  • Carl George
  • James Antill
  • Johnny Hughes
  • Mohan Boddu (mboddu)
  • Merlin Mathesius
  • Stephen Gallagher (sgallagh)
  • Troy Dawson (tdawson)
  • Petr Bokoc (pbokoc)

What the sub-team did in Q3 2021

One thing we tackled was enabling side tag builds for Fedora ELN. Initially, we wanted to implement proper side tags for ELN, but we eventually settled for a simpler approach where we tag the Rawhide builds in, and then rebuild them in ELN. This ensures that we get all the packages built in ELN, with the Rawhide build as a backup should it fail in ELN. And we can even use this as a health metric for ELN — how many ELN packages are actually ELN builds.

For CentOS Stream 9, we have cloud images in AWS available. You can get it by searching for “centos stream 9” in AWS, and to make sure you get the latest you can add this month (so “202110” for October 2021).

Also, CentOS Stream 9 repositories are now available through mirrors using a meta link. Existing systems get this set up automatically with an update, as the centos-release package will include this metalink. This will take some load off the CentOS infra and potentially even make your updates faster.



Goal of this initiative is to update and enhance Datanommer and Datagrepper apps. Datanommer is the database that is used to store all of the fedora messages sent in the Fedora Infrastructure. Datagrepper is an API with web GUI that allows users to find messages stored in Datanommer database. Current solution is slow and the database data structure is not optimal for storing current amounts of data. And here is when this initiative comes into play.

Issue trackers

Application URLs

Members of sub-team for Q3 2021

  • Aurelien Bompard (Team Lead) (abompard)
  • Aoife Moloney (Product Owner) (amoloney)
  • Ellen O’Carroll (Product Owner)
  • Ryan Lerch (ryanlerch)
  • Lenka Segura (lsegura)
  • James Richardson (jrichardson)
  • Stephen Coady (scoady)

What the sub-team did in Q3 2021

Datanommer and Datagrepper have been upgraded to use TimescaleDB, an open-source relational database for time-series data. TimescaleDB is a PostgreSQL extension that takes care of sharding the large amount of data that we have (and keep generating!), and maintains an SQL-compatible interface for applications.

Datagrepper and the Datanommer consumer are now running in OpenShift instead of dedicated VMs.

DNF Counting


DNF Counting is used to obtain data on how Fedora is consumed by users. The current implementation experiences timeouts and crashes when the data are obtained. This initiative is trying to make the retrieval of counting data more reliable and efficient.

Issue trackers


Members of sub-team for Q3 2021

  • Nils Phillipsen (Team Lead) (nils)
  • Aoife Moloney (Product Owner) (amoloney)
  • Ellen O’Carroll (Product Owner)
  • Adam Saleh (asaleh)
  • Patrik Polakovic
  • With special shout-out to Stephen Smoogen that provided vital fixes even though he wasn’t officially part of the initiative 

What the sub-team did in Q3 2021

Scripts that create the statistics for were cleaned up and refactored, making them stable enough, so that they don’t require more manual intervention. 

The code on now has tests running in CI and is packaged as an rpm to avoid further mishaps in package installation. The deployment scripts were cleaned-up as well, alongside the actual deployment on log01 machine, with it’s hard-to-track manual interventions for last minute bug-fixes replaced by ansible-scripts.

Cron-jobs that run the batch-jobs now only send notification emails on failures and to see the overall health of the batch-process you can see the simple dashboard on –

Metrics for Apps on OpenShift


Goal of this initiative is to deploy OpenShift 4 in Fedora Infrastructure and start using Prometheus as a monitoring tool for apps deployed in OpenShift. This initiative should also define what metrics will be collected.

Issue trackers


Members of sub-team for Q3 2021

  • David Kirvan (Team Lead) (dkirwan)
  • Aoife Moloney (Product Owner) (amoloney)
  • Ellen O’Carroll (Product Owner)
  • Vipul Siddharth (siddharthvipul1)
  • Akashdeep Dhar (t0xic0der)

What the sub-team did in Q3 2021

  • Infrastructure prep work to install Red Hat CoreOS on nodes for OpenShift Container Platform (OCP)
  • Deployed OCP4.8 in staging and production
  • Configuring cluster with OAuth, OpenShift Container Storage (OCS) and other important needed operators/configs to support Fedora workloads
  • Automate the process of OCP deployment with Ansible
  • Deployed and configure the User Workload Monitoring stack
  • Investigate app migration from older cluster to new


If you get here, thank you for reading this. If you want to contact us, feel free to do it in #redhat-cpe channel on

CPE Weekly Update – Week of October 18th – 22nd

This is a weekly report from the CPE (Community Platform Engineering) Team. If you have any questions or feedback, please respond to this report or contact us on #redhat-cpe channel on (

Continue reading

GitLab available for Fedora

I’m excited to tell you we will have hosted GitLab available for the Fedora community. GitLab is providing this to Fedora as part of their GitLab for Open Source program. With this, our community has an additional option for code and project management.

The GitLab instance will available for any Fedora-related activities. After we complete connecting the authentication system, you will log in with your Fedora account and create projects as you need them. If your team wants a group within our namespace, we can create that for you. We will follow up with guidance on how to make those requests, but in the short term, you can open a ticket in the infrastructure tracker in Pagure.

Continue reading

Changes to Bugzilla queries

On 13 September 2021, Red Hat’s Bugzilla team released updates to Bugzilla that included new functionality for pagination. There is also a change to the default number of results with the bug search API to support this feature. The default is now 20 but can be adjusted to 1000 by using the limit/offset parameters. 

Continue reading

CPE to staff EPEL work

We are pleased to announce that Red Hat is establishing a small team directly responsible for participating in EPEL activities. Their job isn’t to displace the EPEL community, but rather to support it full-time. We expect many beneficial effects, among those better EPEL readiness for a RHEL major release. The EPEL team will be part of the wider Community Platform Engineering group, or CPE for short.

As a reminder, CPE is the Red Hat team combining IT and release engineering from Fedora and CentOS.
Right now we are staffing up the team and expect to see us begin this work from October 2021. Keep an eye on the EPEL mailing list and the associated tracker as we begin this exciting journey with the EPEL community.

Stories from the amazing world of #11

The sun is shining on the land as I walk through the streets of one of the cities on Anitya. My staff is doing clinking sounds on the stone road under my feet. Everything looks peacefully and seems to be in order. This is how I like it.

Not many citizens have noticed my humble presence. The life here is too busy and I’m a frequent guest in these parts. My steps are going towards the local tavern. I know who will wait for me there and this makes my heart happy. Traveler is always coming back to visit me and his visits are always good way to spent my time.

The tavern is a little dark like most of the taverns in these parts of the land. I looked around searching the faces sitting here, looking for the one that seems familiar to me. I found it! Traveler was sitting in corner sipping his pint of beer and looking at the fire crackling in the heart of the tavern. I came closer to his table: “Hello traveler, it’s nice to meet you again. It was few months till we saw each other last time. The times were busy, but I have plenty of good news for you. I know you want to know what is happening around. So let’s begin.” I sat on the stool beside his table and started talking.

Continue reading

Community Platform Engineering is hiring

The Community Platform Engineering (CPE) group is the Red Hat team combining IT and release engineering from Fedora and CentOS. Our goal is to keep core servers and services running and maintained, build releases, and other strategic tasks that need more dedicated time than volunteers can give. See our docs for more information.

We are hiring new talent to come work full time on Fedora and CentOS. The following positions are now open:

Continue reading

Git repo branch name changes

The Fedora Project envisions a world where everyone benefits from free and open source software built by inclusive, welcoming, and open-minded communities.

The Fedora Project Vision

In line with the Fedora vision, we just completed some changes to the git branch names used on and elsewhere. We removed the “master” branch for those repositories. For rpms and containers, the default branch is now named “rawhide”, with a symref (alias) of “main”. For flatpaks, “stable” is the default/only branch. The fedpkg tool is updated on all supported released to accommodate this change.

For now module repos are unchanged. We are awaiting improvements in the branch/repo requesting tool to allow module owners to request only those specific named branch streams, since “main” and “rawhide” don’t make sense in that context.

For a list of other impacted repositories, see the change proposal. Of course, other repos have been migrated by their owners independently.

If you have a repo checked out with the master branch still, you can run: git fetch && git switch main

This work is part of a larger effort across the technology industry to be more inclusive in the language we use. See Rich Bowen’s Nest With Fedora keynote, for example. If you encounter any trouble, please file a ticket in the infrastructure issue tracker.

GitLab AMA follow-up

Last month, we invited folks from GitLab to a public Ask Me Anything (AMA) session. We collected questions from the community in advance about the upcoming Fedora migration to GitLab. 

The Community Platform Engineering (CPE) team has been working with GitLab for the past few months on understanding and troubleshooting the technical challenges associated with the migration. This AMA was a natural next step to enable the community to participate and give the Fedora community a chance to get to know some of the GitLab team members who are supporting the migration process.

Continue reading

Copyright © 2021 Fedora Community Blog

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

Theme by Anders NorenUp ↑