This is a weekly report from the I&R (Infrastructure & Release Engineering) Team. We provide you both infographic and text version of the weekly report. If you just want to quickly look at what we did, just look at the infographic. If you are interested in more in depth details look below the infographic.
Week: 14th October – 18th October 2024
Infrastructure & Release Engineering
The purpose of this team is to take care of day to day business regarding CentOS and Fedora Infrastructure and Fedora release engineering work.
It’s responsible for services running in Fedora and CentOS infrastructure and preparing things for the new Fedora release (mirrors, mass branching, new namespaces etc.).
List of planned/in-progress issues
Fedora Infra
- In progress:
- vmhost-x86-cc02.rdu-cc.fedoraproject.org has failed disk
- Retirement of `monitor-dashboard` tracker
- CNAME record request for image builder community deployment
- OIDC client credentials for image builder
- Create a POC integration in Konflux for fedora-infra/webhook-to-fedora-messaging
- Retirement of `monitor-gating` tracker
- Need some OIDC credentials for com blog in STG
- setup ipa02.stg and ipa03.stg again as replicas
- RFE: fedoras container image register change
- [CommOps] Communishift – Datanommer
- The process to update the OpenH264 repos is broken
- httpd 2.4.61 causing issue in fedora infrastructure
- Support allocation dedicated hosts for Testing Farm
- anaconda-devel list archive prior to 2021 missing
- Mailman deployment memory spikes
- Searching mailing list archives does not work
- EPEL minor version archive repos in MirrorManager
- vmhost-x86-copr01.rdu-cc.fedoraproject.org DOWN
- Update DRAC Firmware on All Dell Servers
- Recurring issue that Bugzilla stops sending messages to fedora-messaging
- Cleaning script for communishift
- rhel7 eol
- move resultsdb-ci-listener deployment pipeline
- rhel7 EOL – github2fedmsg
- Setup RISC-V builder(s) VM in Fedora Infrastructure
- Move from iptables to firewalld
- fedmsg -> fedora-messaging migration tracker
- rhel9 adoption
- Replace Nagios with Zabbix in Fedora Infrastructure
- Migration of registry.fedoraproject.org to quay.io
- Commits don’t end up on the scm-commits list
- Done:
- mangohud 0.7.2 fc40, rawhide repository version mismatch
- Planning to switch the Copr community RHSM account to SCA
- Approve my user on GitLab test instance
- releng-bot failing to process branch request with an HTTP 400 error
- [IPv6] mirrors.fedoraproject.org & mirrormanager.fedoraproject.org return 503/500 HTTP errors
- Planned Outage – switch upgrade – 2024-10-10 15:00 UTC
- Koji staging `koji.AuthError: Invalid session or bad credentials`
- Need OIDC credentials for commblog in prod
- Planned Outage – koji upgrade – 2024-10-09 21:00 UTC
- Planned Outage – pagure.io network upgrades -2024-10-08 06:00 UTC
- Request to create a public mailing list for “bootloader-eng-team” in Fedora.
- bvmhost-x86-05.stg.iad2 failed HD
- Outage: Upgrade of Copr servers – 2024-10-03 11:00 UTC
- Outage of datagrepper messages ‘org.fedoraproject.prod.github.push’
- Requesting Matrix chat room for Perl SIG
- create copr group `artalk`
- Planned Outage – openqa / openqa-lab – 2024-10-08 21:00UTC
- buildvm-x86-05.iad2.fedoraproject.org: Read-only file system: ‘/var/tmp/koji/tasks/2642/123992642’
- xml-rpc down?
- bvmhost-x86-05.stg.iad2.fedoraproject.org has failed disk
- Create group for OpenStack Kolla Erlang RPMs builds
- buildvm-ppc64le-20.iad2.fedoraproject.org: [Errno 30] Read-only file system: ‘/var/tmp/koji/tasks/388/123690388’
- Retire ODCS and add redirects for ELN repos
- Not able to consume rabbitmq messages through openscanhub user
- Wiki can’t connect to database
CentOS Infra including CentOS CI
- In progress:
- Done:
- CentOS CI – OCP4 migration: NFS-Ganesha cluster access
- Communiy Cage network maintenance
- We no longer wish to be a mirror for CentOS
- Reconfigure CNAME for autosd.sig.centos.org
- [NFV SiG] Request of specific “bootstrap” tag with only s390x enabled
- Upgrade cbs infra to koji 1.34
- Maintainers lost the ability to push changes into c9s protected branches.
- Upgrade koji to 1.34.3
- Investigate how MBS relies on fedmsg and its implications on us when fedmsg disappears
Release Engineering
- In progress:
- retirement processing based on age of pushed commit that adds “dead.package” is too strict
- Please add permission for blocking for epel tags for toddlers
- Fedora-FEX-Rootfs is missing from koji tags
- Could we have fedoraproject-updates-archive.fedoraproject.org for Rawhide?
- Impact check for Changes/FEX
- Investigate and untag packages that failed gating but were merged in via mass rebuild
- a few mass rebuild bumps failed to git push – script should retry or error
- Drop modularity from EPEL8.
- Renaming distribution media for Fedora Server
- Package retirements are broken in rawhide
- Implement checks on package retirements
- Untag containers-common-0.57.1-6.fc40
- orphan-all-packages.py should remove bugzilla_contact entries from fedora-scm-requests as well
- Packages that fail to build SRPM are not reported during the mass rebuild bugzillas
- When orphaning packages, keep the original owner as co-maintainer
- Create an ansible playbook to do the mass-branching
- Cleaning old stuff from koji composes directories
- Fix tokens for ftbfs_weekly_reminder. script
- Update bootloader components assignee to “Bootloader Engineering Team”for Improved collaboration
- Done:
- Package rust-lv2-units not in list for tag epel10.0-testing-candidate
- koji 1.35: blocking a package from a tag does not seem to cause a newRepo
- Stalled EPEL package: libbs2b
- Add permissions needed for sync script to push to quay.io/fedora/fedora-iot
- Missing packages in f41-flatpak, f42-flatpak
- block retired packages in koji that were missed by automation
- Stalled EPEL package: xisxwayland
- BuildRequires environment-modules seems required for projects that build against openmpi/mpich
- FEDORA-2024-ddf075de0f ejected from the push because “Cannot find relevant tag for mesa-24.2.3-3.fc41
- New `epel10` branch creation failed with a traceback
- Stalled EPEL package: perl-NetAddr-IP
- Missing Rawhide Vagrant boxes
- httpd F-41 update in a weird state
If you have any questions or feedback, please respond to this report or contact us on #redhat-cpe channel on matrix.
Start the discussion by commenting on the auto-created topic at discussion.fedoraproject.org