The Community Platform Engineering (CPE) team recently wrote about our face-to-face meeting where we developed a team mission statement and developed a framework for making our workload more manageable. Having more focus will allow us to progress higher priority work for multiple stakeholders and move the needle on more initiatives in a more efficient manner than how we are working right now.
During the F2F we walked through the process of how to gracefully remove ourselves from applications that are not fitting our mission statement. The next couple of months will be a transition phase as we want to ensure continuity and cause minimum disruption to the community. To assist in that strategy, we analysed our applications and came up with four classifications to which they could belong.
Application service categories
1. We maintain it, we run it
This refers to apps that are in our mission and we need to both actively maintain and host it. CPE will be responsible for all development and lifecycle work on those apps, but we do welcome contributors. This is business as usual for CPE and it has a predictable cost associated with it from a planning and maintenance perspective.
2. We don’t maintain it, we run it
This represents apps that are in our infrastructure but we are not responsible for their overall maintenance. We provide power and ping at a server level and will attempt to restart apps that have encountered an issue . We are happy to host them but the maintenance of them, which includes development of new features and bug fixes, are no longer in our day to day remit. This represents light work for us, as the actual applications ownership resides outside of CPE, with our responsibility exclusively on lifecycle management of the app.
3. We don’t maintain it, we don’t run it
This represents an application that we need to move into a mode whereby somebody other than CPE needs to own it. This represents some work on CPE’s side to ensure continuity of service and to ensure that an owner is found for it. Our Community OpenShift instance will be offered to host services here. Apps that fall into this category have mostly been in maintenance mode on the CPE side, but they keep “head space”. So we want for them to live and evolve exclusively outside of CPE on a hosting environment that we can provide as a service. Here, we will provide the means to host an application and will fully support the Community PaaS but any app maintenance or lifecycle events will be in the hands of the people running the app, not the CPE team.
These are apps for which we are a main contributor and which drain time and effort. In turn, this is causing us difficulty in planning wider initiatives because of the unpredictable nature of the requests. Category 3 apps are where our ongoing work in this area is more historical than strategic.
Winding down apps
Category 3 apps ultimately do not fit within CPE’s mission statement and our intention here is to have a maintenance wind-down period. That period will be decided on an app-by-app basis, with a typical wind down period being of the order of 1-6 months. For exceptional circumstances we may extend this out to a maximum of 12 months. That time frame will be decided in consultation with the next maintainer and the community at large to allow for continuity to occur. For apps that find themselves here, we are providing a community service in the form of Community OpenShift (“Communishift”) that could become a home for those apps. However, the CPE team won’t maintain a Service Level Expectation (SLE) for availability or fixes. Our SLE is a best effort to keep services and hardware on the air during working hours while being creative with staff schedules and time zones. We have this documented here for further information. Ideally they would have owners outside the team to which requests could be referred to, but would not be a CPE responsibility.
We are working on formalising the process of winding down an application by creating a Standard Operating Procedure (SOP). At a high level, that should include a project plan derived from consultation with the community. That may see work on the CPE team to get the application to a state of maintenance. That work could be on documentation, training, development of critical fixes / features or help with porting it to another location. Ultimately, the time spent on that kind of work is a fraction of the longer term maintenance cost. Our intention is to run all of the apps through the Fedora Council first, in case the Council prefers any specific alternatives to a particular service or app.
4. We turn it off
This represents applications that are no longer used or have been superseded by another application. This may also represent applications that were not picked up by the other members of the community. Turning off does not equate to a hard removal of the app and if an owner can be found or a case made as to why CPE should own it, we can revisit it.
Initial app analysis
To help us identify that path, at our F2F we have evaluated a first round of apps.
For completeness we are highlighting one example of a Category 1 application that we will always aim to maintain and keep on the air. Bodhi is one such example as it is one of the core services used to build and deliver Fedora and was developed specifically around the needs of the Fedora project. This makes it one of a kind, there are no application out there that could be leveraged to replace it and any attempts to replace it with something else would have repercussions into the entire build process and likely the entire community.
Wiki x 2 (This may be a Category 3 after further analysis) — CPE maintains two wiki instances, one for Fedora and one for CentOS. Both of them are used by the communities in ways that makes it currently impossible to remove. In Fedora’s case it is also used by QA (Quality Assurance), making it an integral part of the Fedora release process and thus not something that can be handed to the community to maintain.
Overall, the trend for these tools will be to move them to a steady-state of no more fixes/enhancements. The community will be welcome to maintain, and/or locate a replacement service that satisfies their requirements. Replacements can be considered by the Council for funding as appropriate.
Mailman/Hyperkitty/postorious — Maintaining this stack has cost the equivalent of an entire developer’s time long-term. However, we recognize the imperative that projects have mailing lists for discussion and collaboration. No further features will be added here and based on the community needs an outside mailing list service can be contracted.
Elections — This application has been in maintenance mode for some time now. We recently invested some time in it to port it to python3, use a newer authentication protocol (OpenID Connect) and move it to openshift, while integrating Ben Cotton’s work to add support for badges to elections. We believe elections is in a technical state that is compatible with a low-maintenance model for a community member who would like to take it over. As a matter of fact, we have already found said community member in the person of Ben Cotton (thank you Ben!).
Fedocal — This application has been in maintenance mode for some time. It has been ported to python3 (but hasn’t had a release with python3 support yet). There is work in progress to port it to OpenID Connect and have it run in OpenShift. It still needs to be ported to fedora-messaging
Nuancier — This application has been in maintenance mode as well. It has been ported to python3 but needs to be ported to OpenID Connect, fedora-messaging and moved to OpenShift.
Badges — This application has been in maintenance mode for a while now. The work to port it to python3 has been started but it still needs to be ported to OpenID Connect, fedora-messaging and moved to openshift. We invested some time recently to identify the highest pain point of the application, log them in the issue tracker as user stories and start prioritizing them. We however, cannot commit to fixing them.
For Fedocal and Nuancier, we are thinking of holding virtual hackfest on Fridays for as long as there is work to do on them, advertise this to the community to try to spark interest in these applications in the hope that we find someone interested enough (and after these hackfest knowledgeable enough) to take over their maintenance.
Pastebin — fpaste.org is a well known and used service in Fedora, however it has been a pain point for the CPE team for few years. The pastebin software that exist are most often not maintained, finding and replacing one is a full-time work for a few weeks. Finally, this type of service also comes with high legal costs as we are often asked to remove content from it, despite the limited time this content remains available. CentOS is also running a pastebin service but it has the same long term costs and a similar conversation will need to occur there
Apps.fp.o — This is the landing page available at https://apps.fedoraproject.org/. Its content has not been kept up to date and it overall needs some redesign. We may be open to give it up to a community member, but we do not believe that the gain is worth the time investment in finding that person
Ipsilon — Ipsilon is our identity provider. It supports multiple authentication protocol (OpenID 2.0, OpenID Connect, SAML 2.0, …) and multiple backends (FAS, LDAP/FreeIPA, htpasswd, system accounts…). While it was originally shipped as a tech preview in RHEL it no longer is and the team working on this application has also been refocused on other projects. We would like to move all our applications to use OpenID Connect or SAML 2.0 (instead of OpenID 2.0 with (custom) extensions) and replace FAS with an IPA-based solution, which in turn allows us to replace ipsilon by a more maintained solution, likely Red Hat Single Sign On. The dependencies are making this a long term effort. We will need to announce to the community that this means we will shut down the public OpenID 2.0 endpoints, which means that any community services that use this protocol need to be moved to OpenID Connect as well.
Over the coming weeks we will setup our process to begin the formal window of the items listed above that are in a 3 or a 4 state and will share that process and plan with the Fedora Council.