On November 14, we conducted a release retrospective for Fedora 29. While the general consensus was that we put out a quality release in accordance with the schedule, there are a few areas flagged by the community for discussion and possible improvement. Minutes and IRC logs are available from MeetBot.

Retrospective topics

Unannounced changes to core packages

This is primarily related to dnf. The dnf maintainers did not file a Change proposal for Fedora 29. However, some significant changes were made to the dnf package through the course of the development cycle. Some of these changes were made to support modularity and are ultimately beneficial to Fedora, however it resulted in late destabilization and several blocker bugs. Prior to the release, the FPL, FPgM, and dnf team met to discuss the issue. They had already begun doing bugfix-only instead of rebasing to new releases. This should not be an issue in future releases.

“Go” decision despite bugs with updates

Several bugs proposed as blockers were not accepted because they could not be reproduced or did not violate release criteria. Most notable was a bug (BZ 1642796) where Gnome Software was stuck at 97% during an upgrade. This was later identified to be caused by having multilib libraries installed. František Zatloukal will propose a change to the release criteria that would require testing multilib installs, as this is expected to be a common use case.

Missing spins at release time

Some of the Fedora Spins and Labs were not included in the compose that was declared gold. With a few notable exceptions, Spins intentionally do not block releases. However, from a user-friendliness standpoint, it is nice to have all of the Spins available. Some of the missing Spins failed to build due to broken packages, while others (e.g. the ppc64le Cloud image) happened to have a transient failure during the compose that eventually shipped.

We identified two separate avenues to address this issue. The first is raising the visibility of failed Spin builds prior to the release so that Spin maintainers have time to address this issue. František Zatloukal suggested having a member of the Spins SIG track Release Engineering Pague issues to that they will see build failures and can notify the appropriate Spin owner(s). However, the Spins SIG may not be active enough to support this.

The second is decoupling Spins from the main release process such that they can be published in a more-self service manner. This may take the form of Spin maintainers marking their spin “Go” or of providing a self-service build and publishing system for non-blocking deliverables.

In subsequent discussion with Release Engineering, we determined that the Fedora Program Manager will send a list of failing Spins and Labs to owners at the Freeze points for Fedora 30. Work to provide an automated notification system based on Pungi notifications will be considered for Fedora 31 and beyond. Similarly, Release Engineering is already planning how a self-service platform might be developed for Fedora 31.

SELinux denials the first few days after install

Visible SELinux denials violate the release criteria. No denials were observed during pre-release testing, however some community members observed it after release. The SELinux troubleshooting applet is no longer a part of the default Workstation install, which means the release criteria no longer match reality. Julen Landa Alustiza is going to propose a modification to the release criteria to reflect the new state — either to drop the SELinux denial criterion or to update it to match the default Workstation configuration.

Cisco openh264 repo

Updated openh264 packages provided to Cisco were not available on Cisco’s repo when Fedora 29 was released. This meant the packages could not be installed or updated initially. Release Engineering worked with Cisco to resolve the issue. For future releases, Release Engineering will verify that the packages are available before updating the repodata in Fedora’s repos.

Meta retrospective

The discussion was hindered by difficulties with the Jitsi conference. Not all participants were audible to all other participants. We eventually moved the conversation to IRC, where I had been logging notes. In the future, we may use BlueJeans or another conference platform.

In addition, we had several topics suggested by members of the community. In the future, it would help for the Program Manager to collect topics with some details during the weeks leading up to the release. This advanced preparedness would allow more time for the community to see some of the topics and ensure that relevant participants can be explicitly invited.