Inspired by a similar report from the Copr team, I’ve decided to look back at 2020 from the perspective of Python in Fedora (and little bit in RHEL/CentOS+EPEL as well). Here are the things we have done in Fedora (and EL) in 2020. By we I usually mean the Python Maint team at Red Hat and/or the Fedora’s Python SIG.

The year 2020 was a special year for the Python community (not only because of the pandemic), as Python 2 has finally gone out of support at the very beginning of the year, with an ultimate (somehow celebratory) release of Python 2.7.18 in April.

Rollin’, rollin’, rollin’

During the year, we’ve kept several Python versions in Fedora always up to date, and Python developers were able to test their software with Python 2.7.18, 3.5.10, 3.6.10–12, 3.7.7–9, 3.8.2–7, 3.9.0 alphas, betas and release candidates as well as 3.9.1 and early alphas of 3.10.0 as soon as they were released upstream, usually within one business day + mirror delays (including the release candidates of the bugfix releases on Rawhide). Tomáš Hrnčiar created a nightly-updated calendar with upstream Python release schedules, which makes this much easier to track. We have also kept PyPy 2.7 and 3.6 as well as MicroPython up to date (although the response time is not that good there).

Since the process for keeping our Pythons up to date was only in my head, we have started to document most of the stuff to avoid problems if I’m hit by a bus. We now have a public forever-work-in-progress guide about our Python (interpreters) maintenance in Fedora. When documenting, we also identified stuff to automate and we managed to actually automate most of it, such as importing our patches from GitHub, cherry-picking dist-git commits between different similar components, automatically resolving simple git merge conflicts in %changelogs and more.

When EPEL 6 went end of life, we have retired Python 2.6 and 3.4 from Fedora 33+.

At the beginning of 2020, we have just successfully finished migrating Fedora 32 to Python 3.8 only to immediately start working on Python 3.9 for Fedora 33. This close followup was caused by originally postponing Python 3.8 from Fedora 31 to 32 in 2019 because the release schedules of Python and Fedora were a tad too misaligned to allow a safe upgrade so soon. We have managed to align them better late in 2019 which allowed us work on 3.9 starting in early 2020.

Due to the end of life of Python 2 in 2020, many Python 2 backwards-compatibility shims were removed from Python 3.9, causing a lot of new breakages in projects that have not yet managed to respond by removing Python 2 support, not because they would not want to, but simply because it happened too early. We’ve detected and measured the impact on Fedora packages and managed to postpone the most breaking changes to Python 3.10, giving the projects one more year to adapt.

Speaking of Python 3.10, we have started to work on the Python 3.10 change for Fedora 35 late in 2020, making it one of our primary tasks for 2021. If you maintain a Python package, you might have seen a bug report from us already. We test early and we test often to avoid sudden Rawhide breakage. As always, we don’t only report, but we try to help as much as we can, but we cannot single-handedly fix hundreds of packages. Package maintainers are crucial to make this all work, so the thanks goes out to you!

During one year, we’ve obviously updated many crucial Python packages as well. I’d like to highlight a complicated pipenv upgrade after two years of upstream development.

Continuous integration

As most of the Python interpreters (all but MicroPython) need setuptools and pip for the venv tool (that creates Python virtual environments) as well as for virtualenv and tox, we’ve made sure the new Python versions have an up to date version of setuptools and pip built in Fedora and the older ones use the bundled versions provided by upstream. In order to make sure everything works as expected, we were constantly improving our CI coverage: During an update of Python, setuptools, pip, virtualenv, tox and/or other important packages, automated tests ensure that everything keeps working as expected for Python developers running Fedora.

Apart form testing Python integration in Fedora, we have also broadened the testing matrix of Python’s upstream CI, adding Fedora and RHEL buildbot workers for various architectures. The s390x machines were provided in cooperation with IBM Research, ppc64le machines were provided by OpenPower hub from Brno’s Faculty of Information Technology, ARM provided an aarch64 bare metal server, and x86_64 machines were already in place. Currently the upstream CI tests CPython with various configurations: Fedora Rawhide, the latest stable Fedora, RHEL 7 and RHEL 8, and on all the architectures we support.

Being able to test upstream changes early on Fedora not only avoids potential problems in Python itself, but helped us to identify unrelated problems in the distro as well. Thanks to the upstream CI, we have discovered:

We have also worked on extending the Fedora Python experience beyond developers’ machines. It is now easier than ever to run Python tests on Fedora using GitHub Actions. Give it a go, instead of the default Ubuntu, when setting an upstream CI.

In 2020, we have also significantly improved the automated testing of Python packaging RPM macros and dependency generators. Speaking of which…

Python packaging improvements

Many interesting changes happened during 2020, changes not only targeted to developers and users, but also to Fedora’s Python package maintainers: The overall number of Python packages in Fedora has grown from ~3150 to ~3900 since Fedora 31 was released late in 2019. They currently constitute ~17% of all Fedora packages. Over 700 different people (unique by email addresses, obvious duplicates and bots removed) have committed to those packages in 2020. Thank you all!

We have significantly improved the Python RPM dependency generators, we have introduced the %py_provide, %pytest, %python3_shebang_fix macros, we have introduced a way to create more reproducible Python bytecode with a new tool written just for that purpose. But the biggest new development in Python RPM packaging was finally adding the ability to package Python extras. Most of this was backported to EPEL (or at least we made sure the usage of these new features in an EPEL specfile does not fail the build).

Several small changes were made to the Fedora’s Python packaging guidelines in 2020, while we have continuously worked to present a completely revised guidelines and macros in 2021. We have also significantly improved the pyproject-rpm-macros, adding support for more complex cases and even new features, most importantly the automatic %files section. Give it a try in 2021 if you haven’t yet. Close to one hundred packages already use them, most of them started in 2020.

Improving Python for everybody

In April, Phoronix reported Python is a lot faster in Fedora 32 – this was achieved by using --no-semantic-interposition. The flag is now the default for Python in upstream, thanks to integration testing in Fedora.

For 16 years, Fedora was patching Python to install to /usr/lib64 on 64-bit systems. Similar patch was present in openSUSE, Debian, Gentoo and many other distros. In 2020, we have finally managed to merge it upstream! (Though credit where credit’s due, the code was contributed both by Fedora and openSUSE folks.)

Early in 2020, we have explored some options to minimize the filesystem footprint of Python, to match with the Fedora’s Minimization objective. Unfortunately, there has not been much response from upstream and we have not finished much on this front yet. However, some small adjustments were made in downstream packaging, for example only shipping some of the large automatically generated modules as bytecode, saving 2+ MiB. We’ve also added a hardlink deduplication flag for the bytecode compiling tool, mimicking the already existent buildroot policy script that originated in Fedora 13 years ago.

On the RHEL (and CentOS) side

The Python 3.8 module has been added to RHEL 8 and we are working on Python 3.9 as well. If all goes well, you should be able to check it out in the new CentOS Stream even before it’s released in RHEL 8. We have also been preparing Fedora to allow packaging parallel-installable Python stacks with non-modular RPMs trying to provide an even better experience in RHEL 9. The Python 3.8 Software Collection was added for RHEL 7.

The --no-semantic-interposition speedup was backported to the (Platform) Python 3.6 interpreter as well as Python 3.8 in RHEL 8. FIPS support has been overhauled and implemented on RHEL 8 for Python 3.6 and 3.8, various related fixes have been merged upstream. Many CVEs affecting Python and relevant components have been fixed across various RHEL products.

Honorable mentions

GNOME Activities Overview with the IDLE 3 icon.

What’s next?

In 2021, we’ll try to keep up with 2020 and ensure Fedora is the best operating system for a Python developer.

The priorities of our team for the upcoming months (apart from constant backporting of CVE fixes to an n-dimensional matrix of components) are the Python 3.9 module for RHEL 8, Python 3.10 in Fedora 35 (and 3.11 for 37, huh, that sounds very far away, but fits into 2021), more RHEL 9 work, finally proposing the new Python guidelines and possibly creating a new spec file generator that leverages the pyproject macros. We might end up working on the filesystem footprint minimization.

I’d like to add even more complex CI tests to Fedora (e.g. when I upgrade pytest, what’s the impact on all packages that use it). If there is high enough demand, I might look into adding Python 3.9 to EPEL 7 (only the interpreter, not a parallel stack) to supplement the RHEL 8 module and the main Python version in ELN.

However, most of the cool stuff from 2020 happened without a year-long planning. So as always, I expect we’ll just have to wait and see.

(Big thanks to the team for helping me write this article.)