A Quick Primer on Modularity
Modularity (formerly, Modularization) is an ongoing initiative in Fedora to resolve the issue of divergent, occasionally conflicting lifecycles of different components. A module provides functionality (such as a web server) and includes well-integrated and well-tested components (such as Apache httpd
and the libraries on which it depends). It can be deployed into production in various ways: as “classic” RPM packages or a container image, and is updated as a whole. Different modules can emphasize new features, stability, security, etc. differently.
Modules differ from traditional packaging in certain important ways. Perhaps most importantly, they allow us to separate internal implementation details from the exposed interfaces of the module. Historically in Fedora, if a packager wanted to deliver a new web application, that would also often mean that they needed to package and carry the framework or other libraries used by that application. This tended to be a double-edged sword: on the one hand, those libraries were now available for anyone to pick up and use in Fedora. However, in many cases, this meant that the primary maintainer of that package might actually have no specific knowledge or understanding of it except that its lack would mean their application didn’t work. This can be a problem if a person is carrying around a library for the use of a single helper function and don’t want to be responsible for issues in the rest of the library.
With a modular approach, the module itself will provide a definition of what public interfaces are stable for use by other projects. This way, they can opt to contain an internal-only implementation of some libraries.
A good metaphor for Modularity might be urban planning: sections of the Earth are selected for establishing housing, businesses and other construction projects. Each of these projects would be effectively a module.
Base Runtime: The Bedrock
The first thing that a construction company would look at when establishing a new building site would be the ground on which it is to be constructed. It is essential that this location be sturdy, reliable and capable of supporting the weight of the projects being built atop it.
In the Fedora Project, the bedrock upon which the other modules will be built is called the Base Runtime. The definition of this project has gone through a number of revisions over the last few months, but at this point it is fairly settled down to this:
“The Base Runtime contains the software necessary to boot the system to a running kernel and the runtime libraries for the most rudimentary operation of the system.”
In practical terms, this means that the Base Runtime is:
- Not installable by itself. It can only boot to a kernel; it has no init system or running applications.
- Not self-hosting. It does not contain the packages necessary to rebuild itself from source. Other modules will need to provide this.
- Limited to packages with an extremely stable public API. The Base Runtime needs to be swappable at any time out from under the running system without impacting the operation of applications running atop it.
System Runtime: Urban Infrastructure
Once you have a location chosen and have built some homes and some businesses, you need a certain amount of infrastructure to connect it all together: roads, plumbing, electricity, etc.
In the case of a computer operating system, this means things like service control and monitoring, filesystem operations, command shells and other basic tools for operating on and maintaining the system. In Fedora, this means essentially the bash shell environment, login services and the standard POSIX command utilities.
The primary reason for separating the System Runtime from the Base Runtime is to allow these two modules to carry different API lifecycle guarantees. While the Base Runtime will need to remain backwards compatible for an extended period, it may be permissible for the System Runtime to see revisions at a higher rate (where it makes sense to provide new functionality faster). For example, Fedora may wish to update the systemd project (used for control, monitoring and management of system services) at a much higher rate than the low-level C runtime library.
Shared Components: Building Materials
In order to build up your city, you naturally need a set of raw materials. Wood, stone, metal and essential workers.
The third and final piece of the puzzle is the Shared Components module. This special module is comprised of the set of low-level libraries common to both the Base Runtime and the System Runtime. Some of these libraries may be made available for other services to consume, but the majority of them will be held privately within the modules.
Generational Core: Local Government
After building up the town, it is important to have a mechanism in place for maintaining it, improving it and making sure to adapt to changing conditions in the world around it. In the urban planning world, this would mean that the local government would establish policies and groups capable of performing these tasks. They would be given access to all of the tools used to create the city in the first place and would continue to monitor things and address issues as they arise. This is effectively what the Generational Core is: the tying together of all those disparate components as a single entity.
While the Base Runtime, System Runtime and Shared Components modules will be built separately and maintained for independent lifecycles, they will be delivered to end-users as part of a single combined module stack called the Generational Core. (Defining “Generational” in the sense of “genealogy” as opposed to “creation”).
Unlike the Base Runtime and System Runtime, the Generational Core will be installable and very similar to the “minimal install” of previous releases of Fedora. It will be somewhat more stripped down even than those. For example, the Generational Core does not need to provide network management services, remote login capabilities or network storage connectivity. These features will be provided by additional modules and module stacks built atop the Generational Core.
November 19, 2016 — 06:25
This modularity thing is a huge step backwards. Instead of one integrated Fedora that is known to work together, we will now get a hodgepodge of repositories that may or may not be compatible with each other, bringing us back to the repository hell that ruined RPM’s reputation in the old days (because people were blaming “RPM hell”, as they called it, on RPM). Using isolation techniques such as SCLs or containers can mitigate this to some extent (though compatibility issues may still occur in APIs intended to be public), but those techniques are themselves a problem rather than a solution, as I am going to elaborate below.
While allowing a packager to declare a library private and enforce this through isolation techniques may sound attractive to that packager for the reasons you mention (not having to support the library for other users), that is a very egoistical point of view and severely hurts the project overall. Making a library private to a module means that all other modules that also need the library now need to carry their own copy of it! This results in all the well-known issues with bundled libraries: duplication of efforts (Why should I have to package a library again if somebody else already packaged it for Fedora?), difficulties with deploying bug and security fixes (You have to track down each and every copy.), wasted space (bandwidth, disk space, RAM), etc. Allowing the bundling to happen at distribution/module level rather than at tarball level does not solve these issues at all, it only leads to more bundling, amplifying the issues. A module-private .so library defeats all the purposes of building the library as a shared library to begin with (by unsharing it).
I think it is not unreasonable at all to expect a packager of a library to make that library usable for everyone, not just the one application he or she cares about. If there really are different applications needing different small subsets of a library’s functionality, as in your example, which is quite an edge case in practice, then that is a perfect use case for comaintainership. The people who care about different parts of the library should just comaintain the library together. What makes a GNU/Linux distribution work so well is the community that works together on packaging software. It should not degenerate into a bunch of private sandboxes.
Even just the idea of having a separate Core and Extras has already been tried in Fedora and rejected. It was just not practical. Many core packages have optional subpackages depending on stuff you don’t want in the core. Kicking those out of the core requires you to build separate SRPMs out of the same code just to build the subpackages separately. (That, or you lose the affected features entirely, which I fear is going to happen more often than not.) The Core-Extras Merge was a huge relief for Fedora, this issue just went away overnight. Now you want to bring it back, multiplied by the number of modules, even.