This is a part of the FESCo Elections Interviews series. Voting is open to all Fedora contributors. The voting period starts on Thursday, December 6th and closes promptly at 23:59:59 UTC on Thursday, December 20th, 2018.
The Fedora project has been moving decisively in the direction of increased flexibility and diversity – with Modularity, with new distribution mechanisms and formats like ostree and containers. One of our big challenges is to able to take advantage of all this flexibility and still provide a stable, easy to understand experience to our users. I’ve been involved in desktop Linux from the days of editing your window manager and XFree86 configuration manually to the current point where have slick desktop environments that just work. I’m familiar not just with the technology, but also the design processes we’ve developed, and the tricky balancing act between keeping things streamlined and accommodating the needs of advanced users.
There is a lot of work currently going on in Fedora to separate out the operating system layer from the application layer – Fedora CoreOS, Fedora Silverblue, and our container and Flatpaks efforts all represent aspecs of this. FESCo should be continually keeping these ways of consuming Fedora in mind.
A personal interest of mine is how developers work in these new models – how do Fedora users develop their applications, and how do Fedora developers develop Fedora. Our traditional model, where the operating system and applications emerge from a great big sea of packages gives a ton of flexibility, and once you move away from it, it feels a bit like putting handcuffs on, but I think there’s a lot of value to be found for developers as well – to be able to have different development setups for different applications, and to be able to try out operating system changes in ways that are well contained and easily reverted.
At times, it seems like FESCo can get a bit bogged down in the nuts-and-bolts of fixing the operating system – and while that’s essential work, I’d hope that some of that work can be delegated and FESCo can concentrate a bit more on high-level issues and what Fedora needs to do as a project to enable a better operating-system / application split.
I think it’s generally agreed that the weakest part of our process is the “compose” step – taking the bits we create and making an operating system image out of them, reliably and quickly. Not only does this hamper our ability to fix bugs, it is a huge limitation in our ability to do integration testing – any packager who wants to update a system image or component should be able to get tests run on the entire operating system with their component included. In fact they should be required to get such tests run. I think there are some good efforts out their to improve things – FESCO can help out by directing focus towards those efforts and by continually pushing back when a “broken operating system” problem comes up – not just helping out with an immediate fix but figuring out why it wasn’t caught by automated testing.
This is a part of the FESCo Elections Interviews series. Voting is open to all Fedora contributors. The voting period starts on Thursday, December 6th and closes promptly at 23:59:59 UTC on Thursday, December 20th, 2018.
This is a part of the FESCo Elections Interviews series. Voting is open to all Fedora contributors. The voting period starts on Thursday, December 6th and closes promptly at 23:59:59 UTC on Thursday, December 20th, 2018.
We are moving towards more flexible and varied ways of delivering the software. And this flexibility is going to become an issue on its own.
On a technical side, extended list of deliverables requires extended testing effort which is hard to achieve via semi-manual workflows. As a CI Engineer and member of Fedora CI Working Group I want to make sure that CI effort is aligned with current engineering goals. And while we work on improving the CI infrastructure and user experience we also find the ways to incorporate CI in the development process.
Now from the process and policy point of view, as a former DevOps Engineer, I have worked on the “other” side – in cloudy environments flexible to the point of becoming chaotic. And while I believe that there are things Fedora needs to catch up with in terms of modern development practices, there are a lot of things “modern practices” need to catch up with in terms of processes and workflows, which are widely known and established in the Linux distributions world.
I hope that bringing this perspective to FESCo would help us find the right balance between providing the flexible tooling and keeping the solid foundation for it to be usable
It may sound odd, but I would say that one of the big goals of FESCo and Fedora community in general is to prevent Fedora from becoming stable. In this case by stable I mean “freezed” not “free of bugs”.
We need it to be easy to change things, even most core features. We need certain loose ends hanging for everyone to come and take care of.
One of the objectives to achieve that big goal is to provide people with the toolchain and services to make their own unique flavors of Fedora or on top of Fedora. Custom repositories, modules, images, flatpaks… you name it. And we need it to be a self-service available for any contributor to use. We should provide pieces for community to play, build and create.
The other complementary thing is actually CI. While many believe that Continuous Integration is there to prevent people from changing things (it does prevent you from merging changes sometimes), the actual reason why we need CI is to allow more changes coming in. CI (and test automation) gives you the freedom to take in changes which otherwise look unusual, or risky. It allows you to focus on whether or not the change is reasonable and brings any value, rather than worry if it might break something.
Additionally to the topics above one of the troubling issues in Fedora is the lack of open collaboration on the package maintenance side. Traditionally, changes in package specs happen locally in a private environment, and community can only interact with the outcome of the change via karma or bug reports. It is very hard for people to contribute and actively help the maintainer especially when they don’t have the packager status themselves.
We have already done a lot to improve this situation. The open pull-requests workflow, which is available on Pagure already, being one of those big steps. But there is more to it. CI, again, could help here as via early integration we encourage maintainers of dependent packages to work closely and discuss the changes happening in interacting packages early in the development phase, and to actually contribute into each other’s test suites.
This is a part of the FESCo Elections Interviews series. Voting is open to all Fedora contributors. The voting period starts on Thursday, December 6th and closes promptly at 23:59:59 UTC on Thursday, December 20th, 2018.
There’s been a lot of discussion about the compose (and release) process recently. While I don’t necessarily always agree with the motivation, I think improving the compose process would be crucial for the future. Speed is one measure, but it’s not the most important one in my opinion. I’d rather focus on making the process more reliable (and possibly even reproducible) to avoid issues like deliverables (possibly randomly) missing from composes (even the GOLD one, like what happened for fedora 29). I’d push for making the processes more reliable first, and optimize later (“premature optimization is the root of all evil”).
Another focus should be placed on how the Modularity effort is moving forward. While I see big benefits in some areas, especially server environments (I’ve read about efforts to make the different NodeJS versions supported by upstream concurrently available as modules), I don’t think it is suited for solving problems in the “desktop” use case, since there are a lot more (indepentently) moving parts involved.
With respect to the “release cadence” discussion that has erupted recently, I’d rather support a tick-tock cycle with annual “major” releases (e.g. fedora 2019.0) and a “minor” release (e.g. fedora 2019.1), where effort for “major” changes isn’t constrained to 5 months, but rather 11. Another possibility would be to just treat GNOME like other packages and do updates during releases (like KDE), which would make the “tock” minor releases unnecessary (many packagers already do updates like this, GNOME is rather an exception here).
By extension, a modified release cadence like this would result in a almost double the support period for major fedora releases (if we keep 2 major supported releases around, just like now), or reduce the number of supported fedora releases by one (if we keep the support period the same) – which other, recent discussions about the fedora release cadence have mentioned.
Since I don’t think that we can (or should!) “grow” fedora by fundamentally changing what it means to be the linux distribution “fedora”, I’d rather focus on improving upon the strengths fedora already has:
We all know that the development and release model of fedora has its drawbacks, too. Improving upon the “pain points” would be my preferred way forward, instead of throwing everything that alread works out the window. After all, the current model has served us well for many ears.
This could possibly mean moving to ONE annual release (I’d only agree with this if more components would be updated mid-release, for example GNOME), or moving to a tick-tock “major”-“minor” release cycle with one annual “major” release – to pick up updated language toolchains and make infrastructure changes – and one (or more?) “minor” release(s) to update things atop this stable “core”. I don’t think we need modularity for this, since only one set of packages would be supported to run on a “major release” at a time.
I think an area that could really be improved is the way some changes (be it “Changes” or major package updates being pushed) are introduced to fedora. For example, I find it hilarous that every release there’s a need for a GNOME mega-update and a corresponding freeze exception, just because the release schedules don’t match up. Moving the target date for a fedora release to avoid situations like this could even reduce release date postponements (which has improved over the years, but it’s still kind of a meme that fedora never releases on time).
I’d also suggest being more conservative with accepting changes – I’d like to see more testing before something big lands in rawhide (e.g. dbus-broker transition, redhat-rpm-macros changes for build flags, macros, etc.). Update gating can help with that, doing test rebuilds in koji side tags (and maybe even test composes for the biggest changes) could help too. Related to this, I’d also promote the usage of koschei, and just add all existing packages to be tracked automatically. I’ve already caught multiple issues by relying on koschei for my packages, and I’m always surprised by how small the number of tracked “core” packages is.
I also increasingly see some packages (mostly podman/buildah/runc/etc.) basically using fedora rawhide as their CI system with bot-created snapshot builds being submitted to koji. While I didn’t find a policy that explicitly disallows this, I think it’s against the spirit of the Updates Policy, even for rawhide – since there’s no way that automatically submitted builds can be verified to conform the Updates Policy criteria. For example, “don’t push clearly broken builds”, or “announce API/ABI changes a week in advance” just isn’t satisfiable with bot-submitted snapshot builds. To reduce the likelihood of broken packages in fedora, I’d amend the Updates Policy to explicitly state that – basically – “rawhide isn’t a CI system”. (This point doesn’t apply to GCC, glibc, and kernel snapshots though – they are already explicitly allowed, and they are submitted by a human who knows very well what they are doing.)
This is a part of the FESCo Elections Interviews series. Voting is open to all Fedora contributors. The voting period starts on Thursday, December 6th and closes promptly at 23:59:59 UTC on Thursday, December 20th, 2018.
The biggest problem I see currently is the introduction of Modularity and the related changes causing a slowdown of development in other areas. There are three important aspects: first, it is a complicated topic that brings a whole set of new concepts, tools, techniques, and bugs, and packagers and users both have to get acquainted with modules, which takes time and effort. Second, the roll-out of modularity, e.g. integration in dnf and other package managers, is still far from great. Third, many smart people at the core of the distribution are diverting from other activities to work on this. I hope that with F30 most of those issues will be figured out and we can reap the benefits.
The way we do packaging is evolving all the time. We have Modularity, and we are right now facing questions how to make it generally usable, without abandoning external users and redistributors of our packages. There are various proposals how to evolve packaging to cater to go and rust and modern python packaging, to allow maintainers to manage packages semi-automatically, more in line with native package management in those languages. I hope FESCo can coordinate these efforts to allow things to “happen”, but in a way that the needs of all stakeholders are taken into account.
I’m not convinced by the recent proposals to delay F31 to work out the releng and pipeline and tooling challenges. I do see many things that need to be improved, but most of them are independent of the release cycle, and can be worked at any time. It seems that currently too much depends on the core releng team, and I hope we some of those responsibilities can be devolved, and others automatized. In this and other areas, I would love to see more freedom for “fly-by” contributions, like proven packagers do for spec file maintenance and anyone can get involved in the freeze testing and bug zapping process.
Communication between different teams is, as always, an issue. I think all our teams are doing an excellent job, but there’s always more need for transparency and exchange of information. I hope FESCo can improve it’s procedures to invite stakeholders earlier and more consistently.
There’s a number of processes that don’t seem to be happening or are only happening through manual efforts, without automatization. Cleanup of FTBFS, orphaned, and insecure packages, package ownership reassignments, notifications about missing dependencies. We are still recovering from pkgdb removal. We need to automatize those things again to make everything run a little smoother, and to avoid burnout from releng members.
This is a part of the FESCo Elections Interviews series. Voting is open to all Fedora contributors. The voting period starts on Thursday, December 6th and closes promptly at 23:59:59 UTC on Thursday, December 20th, 2018.
I believe Fedora is going to have to face issues in areas like Silverblue, Modularity, wider Flatpak adoption and so on. All this is it’s kind of “reinventing” some core concepts of traditional Linux Distribution. However, I see this as an amazing opportunity for Fedora … to distinguish itself more from others.
The Idea behind Silverblue is a true revolutionary concept in Linux Desktop area and Fedora is leading the way while leaving other Distributions far behind. Light core systems, applications contained in flatpaks, “de-facto” instant upgrades and updates, being able to boot an older tree if something goes south… I am really excited for this.
Not to speak just about Silverblue, one more area that we should pay attention is proposed Fedora LTS. This, if done correctly, could bring Fedora to many more people as pre-installed OS. I see this as a huge challenge, not to overwhelm package maintainers and not to slow Fedora’s leading edge pace.
I see some areas which FESCO should focus on in following months. Recently opened up discussions around delaying Fedora 31 in order to refine and improve the way we release Fedora is definitely one of them. The vision of fast composing, decoupled spins so anything new can be tested right away instead of waiting 8 hours or more sounds great to my Fedora QE ears 🙂 .
Apart from Silverblue, I believe Modularity is also a way forward. Something that, if spread across more packages, could make Fedora an obvious choice for Server deployment.
I’ve mentioned one of these in previous answer. How could FESCO help here? One of the ways is to improve and accelerate the communication between teams. From top of my head, i see some opportunities in tighter collaboration between QE and Marketing/Websites. I believe there are many more areas which Fedora could do better and it might make sense to invest some time and investigate it more.
This is a part of the FESCo Elections Interviews series. Voting is open to all Fedora contributors. The voting period starts on Thursday, December 6th and closes promptly at 23:59:59 UTC on Thursday, December 20th, 2018.
The advent of Fedora CoreOS in our community gives us a good opportunity to re-evaluate our build architecture and infrastrucure. The packaging process, for RPMs as well as Containers, still has lots of room for improvement, ranging from easier on-boarding to improved developer experience and documentation, up to technical improvements and a higher degree of automation. As packager and member of the CoreOS Working Group (previously Project Atomic WG), the Container SIG and the IoT WG I have gained insights into all kinds of packaging- and container-related things in Fedora. I now currently intern on the Fedora CoreOS team.
I believe with well-debated technical decisions as well as efforts to document and communicate best practices, FESCo can have a great impact and make for huge improvements in all of these spots.
Copyright © 2019 Fedora Community Blog
This work is licensed under a Creative Commons Attribution 4.0 International License.
Theme by Anders Noren — Up ↑
Recent Comments