Summary
Fedora’s Modularity initiative aims to make it easy for packagers to create alternative versions of software and for users to consume those streams simply. We’ve been working on this for several years, resulting in the “Boltron” prototype this summer and the recent Fedora Modular Server beta. Feedback shows that these test releases didn’t meet the goal, and we’re incorporating that in a modified design which we think will. We plan to demo the new approach by DevConf.cz and FOSDEM.
Retrospective
As you have probably read, the members of the Modularity Working Group and the Server Working Group decided that the current implementation of Modularity would not meet Fedora’s standard of excellence in time for a release in Fedora 27 — even with the extra time that the Fedora Council granted to the project. In particular, feedback received from the Fedora 27 Modular Server Beta release alerted us to several serious problems in the design.
The most common feedback from users was: “How do I install package foo
in Modular Server?”. In this case, foo
ranged across a wide variety of software, including everything from the screen
package to complex third-party applications. In that version of Modularity, a system was either all Modular or none of it was. To make software available in the fully-modular system, its packages needed to be part of a module — and unfortunately, we didn’t succeed in making many of those.
The lack of available modules was a symptom of a larger problem with packager involvement. Some of this problem was due to disagreements with the implementation or design, and some of it was because creating modules turned out to be more time-consuming and had a steeper learning curve than we wanted. Additionally, the whole process was not obvious or well-socialized. The team produced a lot of documentation, but packagers needed to understand a whole lot of it before even getting started. As a result, all of the modules slated for delivery in F27 Modular Server were being developed by the Modularity team, a group that had limited expertise around many of the packages we desired to ship.
There were other problems, of course. We didn’t really have a strong plan for how to handle upgrades from traditional deployments. We lacked a story for how third-party software should be loaded onto the system. And, we never got away from needing a “bootstrap” module (as explained below). So, we decided not to ship Modular Server in Fedora 27, and we activated our fallback plan of releasing Fedora 27 Server Edition in its classic incarnation.
Moving Forward
With all of this in mind, we took Modularity back to the drawing board (or perhaps “chopping board”). We analyzed the various functionality that the design offered us and made some trade-offs.
First and foremost, we gave up on the idea of a strictly-maintained, stable buildroot. Traditional Fedora builds are performed by building an RPM in a buildroot containing the latest packages available in the stable updates repository for a release (plus the overrides repo when needed to allow building against pre-release packages). With modularity, we hoped that we could define a small and specific buildroot which would be stable and maintained for the life of a release. We would then build every module against it. It turned out when we tried to implement this that it was basically impossible. It required us to find a point at which the entire buildroot could be used to build itself, a feat which has not been accomplished in Fedora for many years (if ever). Fedora packages are really only guaranteed to build successfully against a buildroot for which they have already been built. If it builds against any other buildroot, this is a happy accident.
There are many reasons for this; something in the dependency chain for one of the build dependencies may have broken, a bug may have been introduced in the compiler or linker resulting in an inability to continue, new default compiler flags may have been added that result in warnings becoming errors and so on. This was the easiest requirement to drop as it had never worked. We used a kludgy “bootstrap” module to work around this which provided a set of packages copied from non-Modular Fedora to be able to build the platform. In its place, we will now build modules against the standard Fedora buildroot.
Since our buildroot now has access to everything that has been built for Fedora, we reconsidered another of our original goals that of providing a module to be the base platform upon which all of the other modules would depend. Originally, this was intended to define the provided API of the Base and indicate which portions of it should be considered stable. However, Fedora already has a stable updates policy for how things must act within a release. This policy has only a few outliers (Kernel, KDE, etc.), but for the most part, once we release the Beta, the platform API is stable for the life of the Fedora release. By relaxing this requirement, we realized that we could eliminate (or at least postpone) the requirement for having a defined platform module.
What we decided instead is to treat Fedora’s “Everything” repository (essentially, the complete set of software available within a Fedora release) as the “platform module”, though the tooling will not report this content as a module. In practical terms, this means that creators of modules will no longer need to go through the very painful process of tracking down which modules provide a dependency that they need. Instead, they will be able to depend on the system version available in the Everything repo. If that version does not meet their needs, they will have several options described later.
This provides us with several advantages, including a straightforward upgrade path from a traditional deployment, because we will retain the traditional repositories as well as a set of default modules. This means it will be possible to upgrade from a current Fedora 27 system to a modular Fedora 28 system without any special steps. In fact, this approach will also mean that modularity need not be limited to the Server Edition.
Another practical advantage to this change is that module-creation will become significantly simpler. Instead of a complex collection of a package and all of its dependencies, modules will now only need to describe the parts that differ from the base repository. For example, Fedora 28 will ship with the Node.js 8.x LTS release in the standard repository, and a module could be built to provide the 9.x experimental release as an option. We could also easily provide the older 6.x LTS release to support older applications. In these cases, we can ship very simple module definitions which just lists the dist-git branches matching the desired upstream releases.
In fact, this will now be so simple that we plan to provide tools to automate the creation of these modules. Since most modules will now only require a single source package be in the components list, we plan to enable support for automatically building a single branch in dist-git for all active Fedora (and EPEL) releases. Even for more complex multi-package modules, the automatically-created module definitions provide an easy and obvious starting point.
What Will It Look Like?
From an end-user’s perspective, Fedora will ship with two sets of repositories. One will be the traditional Fedora repositories (fedora, updates, and updates-testing) and the other will be a new set of repositories providing alternative and supplementary modules. We haven’t decided on a final name for these yet, so we will use the placeholder terms modular, modular-updates, and modular-updates-testing.
With this design, anyone who does not wish to access the additional versions of software provided by modules can disable the modular and modular-updates repositories and their system will function exactly as it does today. Packages built with Fedora’s traditional process will be installed and managed from the regular fedora repository, as will default versions of packages which use the new process behind the scenes.
For anyone who wants access to additional versions of packages, these new module repositories will make them available. Users will be able to interact with these new repositories by taking advantage of some new syntax in DNF, the same as was used in the Modular Server Beta. If a user wants to install a particular module stream, they can do dnf install module[:stream[/profile]]
(e.g. dnf install @nodejs:6
which will install the default profile containing the nodejs and npm packages). If the user just wants to install whatever version is the default for this system, a dnf install package
will continue to work as it always has.
Conclusion
This refined plan offers an understandable, approachable, and deliverable future for Modularity. Packagers who don’t want to produce modules will be able to continue packaging exactly as they always have with no modification to their workflows. Those who want to provide alternative versions of software in a single release or to easily provide the same version across multiple releases will have new tools to simplify this.
As the number of available modules grows, users of Fedora will have a much easier access to the exact version of software they want to accomplish their tasks. People doing rapid-prototyping can more easily access newer versions of packages and at the same time people running older applications can continue to access the older streams that they need.
December 20, 2017 — 10:40
Thanks for the write-up. Some questions:
1. Can two different versions of the same package (for example nodejs 6 and 8) be installed in parallel on the same system or in the same container? Or must the different versions be installed in separate containers?
2. How are the library dependencies handled for different versions of the same package? Especially for libraries which don’t provide API stability. For example nodejs 6 depends on libfoo 0.2, but nodejs 8 depends on libfoo 0.4, with API breaks between libfoo 0.2 and 0.4. Is libfoo part of the f28/f29 buildroot? I guess that in those cases, a choice must be made and for example the f28 buildroot contains libfoo 0.2 and the f29 buildroot contains libfoo 0.4, and thus nodejs 6 can be installed only on F28 and nodejs 8 only on F29. Usually libraries provide a stable API, so I suppose that this won’t be a big problem.
December 20, 2017 — 17:54
1. In general, we’re expecting people to use containers (or entirely separate VMs, in the last decade’s model) to solve that. Some things _could_ be packaged for parallel install, though, but it’d take per-package work (like SCLs or similar).
2. In that case, ideally, we’d move libfoo out of the base and into a module itself. If that’s not possible, a nodejs-libfoo-compat (or whatever name) package could be introduced.
December 20, 2017 — 18:12
1. As Matthew says, the goal here is parallel-*availability*, not necessarily parallel-installability. All available evidence suggests that parallel-installability is not nearly as important because most deployments of software happens in dedicated VMs or containers. Since parallel installation is a much harder problem to solve, we’re not focusing on it at this point.
2. This is absolutely a big problem, and it’s one of the trade-offs. What we’re going to continue to guarantee is that Fedora’s main repository is internally consistent. So in theory you can choose not to enable any non-default module stream and continue on just as you are today. However, if you choose to enable a module, you may in some circumstances be making a conscious decision that you are locking yourself out of part of the available ecosystem.
From the packager perspective, it looks more like this (I’ll use libuv and nodejs as contrived examples, pretending that Node.js 8.x relies on libuv 1.16.x and Node.js 10.x relies on libuv 2.0.x which is a backwards-incompatible version).
In Fedora, there are several packages that depend on libuv. Let’s say that in F28, we carry libuv 1.16 which can support Node.js 8.x, but we also want to have the Node.js 10.x available in a stream. There are several ways that a packager can handle this:
a) Include both the `nodejs` and the `libuv` package in the 10.x module stream of Node.js. Then, when switching to Node.js 10.x, you accept that this means libuv on your system has switched to 2.0.x and you may have locked yourself out of being able to run software relying on 1.16.x on this system.
b) The nodejs module can carry a private copy of libuv located in a non-standard location or statically linked and teach the nodejs package to link against this copy rather than the system copy.
c) For runtimes that support it (such as C with a SOname bump), you can package the 2.0.x version as libuv20 (or similar name) and depend on that version of the library for the new module.
December 20, 2017 — 20:01
Thanks for the detailed explanation, I now better understand.
It looks like a promising future for Fedora, good luck with your new plan 😉
December 20, 2017 — 13:36
What happens if a module stream provides some version of an RPM that is also available in the Everything repo? I assume dnf will be smart enough to keep the version from the stream.
And what happens if an update of some RPM in “Everything” repo requires an updated version of some dependency, which happens to be installed from a stream, and the version in the stream is too old. Does this block the update?
December 20, 2017 — 18:14
Yeah, the plan uses DNF’s existing repo priorities to keep streams straight. Stephen can fill in more details.
In the second situation, is the RPM with the update getting a *new* dependency? Previously it was good with the old version but now it needs a new one? Yeah, if the new dependency isn’t also available, that would block. I think I might need a more concrete example 🙂
December 20, 2017 — 18:16
DNF is being taught to prioritize the version from a user-selected stream. This work is not yet complete, but can be simulated with repo priorities. (We want to make this an internal feature not relying on the repo priorities simply because we want our users to be able to use that feature without having to know the internals of how we use them for modules).
For the second question, yes. If you have installed a non-default stream of a module, our only reasonable action is to assume that this software is important to you and an update must not break it. See also my comment to the question above, particularly section 2a.
December 28, 2017 — 21:52
So, I installed the Fedora 27 Modularity server beta as a new install (and encountered frustrations in not being able to install packages). Is there an easy way to get my system upgraded to the full Fedora 27 Server package?
January 20, 2018 — 12:25
In the end this sounds like additional repo definitions created by special dnf commands (for the ease of use) which will ensure correct versions will be used when installing and updating packages without module specifier.
I personally don’t see much use for this in Fedora, because the lifecycle is fast. But with RHEL I really like to see this feature. The SCL is nice, but almost all of the time replacing the base-version of the package would be better.
Looking forward to see how dependencies between the module streams will work out.
February 5, 2018 — 18:05
It seems like Copr might be a solution to a similar problem. How are these solutions different? One is more official and tested for upgrade paths?
April 6, 2018 — 08:45
Then how about using the GoboLinux approach for this? it lets you have all the multiple versions of a package that you may want: https://en.wikipedia.org/wiki/GoboLinux