Fedora 29 release retrospective

On November 14, we conducted a release retrospective for Fedora 29. While the general consensus was that we put out a quality release in accordance with the schedule, there are a few areas flagged by the community for discussion and possible improvement. Minutes and IRC logs are available from MeetBot.

Retrospective topics

Unannounced changes to core packages

This is primarily related to dnf. The dnf maintainers did not file a Change proposal for Fedora 29. However, some significant changes were made to the dnf package through the course of the development cycle. Some of these changes were made to support modularity and are ultimately beneficial to Fedora, however it resulted in late destabilization and several blocker bugs. Prior to the release, the FPL, FPgM, and dnf team met to discuss the issue. They had already begun doing bugfix-only instead of rebasing to new releases. This should not be an issue in future releases.

“Go” decision despite bugs with updates

Several bugs proposed as blockers were not accepted because they could not be reproduced or did not violate release criteria. Most notable was a bug (BZ 1642796) where Gnome Software was stuck at 97% during an upgrade. This was later identified to be caused by having multilib libraries installed. František Zatloukal will propose a change to the release criteria that would require testing multilib installs, as this is expected to be a common use case.

Missing spins at release time

Some of the Fedora Spins and Labs were not included in the compose that was declared gold. With a few notable exceptions, Spins intentionally do not block releases. However, from a user-friendliness standpoint, it is nice to have all of the Spins available. Some of the missing Spins failed to build due to broken packages, while others (e.g. the ppc64le Cloud image) happened to have a transient failure during the compose that eventually shipped.

We identified two separate avenues to address this issue. The first is raising the visibility of failed Spin builds prior to the release so that Spin maintainers have time to address this issue. František Zatloukal suggested having a member of the Spins SIG track Release Engineering Pague issues to that they will see build failures and can notify the appropriate Spin owner(s). However, the Spins SIG may not be active enough to support this.

The second is decoupling Spins from the main release process such that they can be published in a more-self service manner. This may take the form of Spin maintainers marking their spin “Go” or of providing a self-service build and publishing system for non-blocking deliverables.

In subsequent discussion with Release Engineering, we determined that the Fedora Program Manager will send a list of failing Spins and Labs to owners at the Freeze points for Fedora 30. Work to provide an automated notification system based on Pungi notifications will be considered for Fedora 31 and beyond. Similarly, Release Engineering is already planning how a self-service platform might be developed for Fedora 31.

SELinux denials the first few days after install

Visible SELinux denials violate the release criteria. No denials were observed during pre-release testing, however some community members observed it after release. The SELinux troubleshooting applet is no longer a part of the default Workstation install, which means the release criteria no longer match reality. Julen Landa Alustiza is going to propose a modification to the release criteria to reflect the new state — either to drop the SELinux denial criterion or to update it to match the default Workstation configuration.

Cisco openh264 repo

Updated openh264 packages provided to Cisco were not available on Cisco’s repo when Fedora 29 was released. This meant the packages could not be installed or updated initially. Release Engineering worked with Cisco to resolve the issue. For future releases, Release Engineering will verify that the packages are available before updating the repodata in Fedora’s repos.

Meta retrospective

The discussion was hindered by difficulties with the Jitsi conference. Not all participants were audible to all other participants. We eventually moved the conversation to IRC, where I had been logging notes. In the future, we may use BlueJeans or another conference platform.

In addition, we had several topics suggested by members of the community. In the future, it would help for the Program Manager to collect topics with some details during the weeks leading up to the release. This advanced preparedness would allow more time for the community to see some of the topics and ensure that relevant participants can be explicitly invited.

Categories: Program Management


  1. Will these problems be addressed in Fedora 29 patches? I have delayed my upgrade from 28 to 29.

    • @Chris, which problems specifically? Many bugs have been fixed in the last few weeks. If you hit the bug specifically mentioned in the post (where GNOME Software gets stuck at 97% during an upgrade), you can reboot and the upgrade will continue as expected.

  2. Akarshan Biswas

    November 21, 2018 — 12:45

    There were problems with dbus as well. Also I noticed gnome-software asking for rpm-ostree libs in a non atomic/silverblue install. Idk whether it was intended or not.

  3. Leslie Satenstein

    November 28, 2018 — 02:52

    I started with Fedora core. I have been happy with Fedora up to and including Fedora 28.
    I am disappointed with Fedora 29.

    I began testing the F29 betas in August. In August I identifed dnf groups that would crash anaconda. The one group that crashes the ISO’s was never ever corrected is “Authoring and Publishing”.
    If today you do a fresh F29 installation and include the mentioned A and P, anaconda will CRASH before installation has completed.

    Some of the groups in the Fedora workspace group are in conflict with each other. Anaconda would highlight the conflict as a red-warning, but not tell you which of the groups were in conflict. Only by trial and error to remove, test and re-add groups was that issue resolved.

    In those pre-gold release months, I also identified that the gnome application to install a printer (install, not configure) was broken. This is still broken after Fedora 29 went gold. Nothing in the release notes to indicate these issues.

    The release notes should be online and should have an associated alert system. If an issue is corrected, and post the info into the release note, a flag && date or user emails should be sent to indicate a document change.

    Fedora engineering for the new release takes the position that all dnf groups and *.rpm files are clean and are free of installation-bugs. Here is my take. QA and others assume the responsibility to create a minimal Fedora system that meets the blocker exceptions. The step to test that Gnome/KDE or other distributed groups do function is missing. Engineering needs support from the Application side. The Application installation testing and expected execution need to be done, with confirmation given to QA/Engineering. A handover process and procedure.

Comments are closed.

Copyright © 2019 Fedora Community Blog

Theme by Anders NorenUp ↑

%d bloggers like this: