Flock is behind us, so it will be good to do some recapitulation what happened and how it was. Like every year it was awesome to meet people in person and add a face to a name. There was plenty of talks about IoT, Modularity and Fedora CI. Our team attended most of these talks and learned so many new things, heard so many new ideas and initiatives going on.

Here are the events we were part of and how they went.

Future of release-monitoring.org

Presenter: Michal Konečný
Duration: 25 minutes
Schedule: Thursday, August 8 • 16:30 – 16:55

I started in the same spirit as in my blog posts with my wizard hat on, but it was hard to stay in role when talking about technical terms. There were about 20 people in the room. 

First I showed them how I see the distant bright future of release-monitoring.org, where everything is covered by light and working as it should.

Then I got to darker part of my presentation: what can we do with release-monitoring.org right now. I presented two choices: fixing the most problematic issues and replacing Anitya with libraries.io. I was surprised that nobody actually protested against the optional future replacing Anitya directly with libraries.io.

Few questions were asked by the audience during this talk, most of them were only asking about how things are and how far are we with the proposed solutions. But one question stays in my mind. I’m not in the project from the beginning, so I didn’t know about this, but in the past the project was automatically created in release-monitoring.org, when new package was added in Fedora and the question was: “Do we plan to have this feature again?” I think this will be really nice feature, that will automate packager workflow, so I will certainly look at this. 

Fedora Compose Tracker

Presenter: Mohan Boddu
Duration: 25 minutes [video]
Schedule: Friday, August 9 • 11:00 – 11:25

I was so excited to give this talk, but not many people showed up due to python 2 removal talk at the same time. But there were some interesting questions and I realized that not many people know how composes work. I spent more time answering questions of how RelEng does composes rather than how we track the failures. But, at the end, they understood the pain points we are having right now and how the compose tracker would help us in tracking the failures easily. One thing we want to do asap is to add the ability to ping the maintainer of the images if there is an image build failure which will solve a major problem of missing images in the release and releng doesn’t have to rebuild them and release them separately.

Open Source Agile in Red Hat’s Community Platform Engineering Team

Presenters: Leigh Griffin, Pierre-Yves Chibon
Duration: 50 minutes [video]
Schedule: Friday, August 9 • 14:00 – 14:50

I wanted to give a strong overview that Agile =/= Scrum. That seems to be the trend in industry because Scrum has won the Agile framework battle and has become ubiquitous in it’s adoption. However Scrum and Agile Conversions in general are difficult and I spoke about the need to be wary of success stories. That it is a long and difficult journey but we can start small.

Showing the audience the range of Agile tools and approaches that fall under the umbrella term Agile was an eye opener for a lot of people. Those approaches are really best practices in our way of working with many teams operating in an Agile manner, they just haven’t self-labelled it as such.

The latter half of the presentation I went and showed some of the changes that we as a team are about to make. I talked about the rationale and the approach and had some really good audience interaction and for a period of time I stepped off of the stage and engaged in a conversation among the audience. This was really helping to explain our various blog posts and comments in the weeks running up to it and it was a wonderful sidebar conversation that helped reassure the approach taken is valid and strengthened some of the slides I had coming up.

The talk finished on a cautionary note that we will get some things wrong and to have patience. That we are going to try our best and we welcome feedback. All in all, I felt the message got across and that the community now understand the reason behind our approach and I left the stage with an air of optimism. I blogged about my overall Flock experience on Medium.

Mass Rebuild and Mass Branching of modules in Fedora

Presenter: Mohan Boddu
Duration: 25 minutes
Schedule: Thursday, August 8 • 14:30 – 14:50

It was quite an interesting talk. I started with how we do mass rebuilds and mass branching in rpms and expanded it to modules. Lot of people had questions of why we are doing it the way we are doing it, especially why build for every release, instead of a particular release and how we can improve the process. Currently there are some tooling limitations which I hope to see resolved soon.

Gating rawhide packages: things just got real!

Presenter: Pierre-Yves Chibon
Duration: 50 minutes [video]
Schedule: Friday, August 9 • 13:00 – 13:50

We had a full room to discuss the rawhide package gating initiative. We worked through why we want to gate rawhide packages, the current state of the work as well as the upcoming changes. We had a lively discussion with a number of questions from the audience and overall everyone seemed positive about the change. There are still some gaps we have identified, and there may be some we’ve missed but we believe that with everyone’s help we will be able to find and fix them.

Building CentOS with Familiar Tools

Presenter: Brian Stinson
Duration: 25 minutes [video]
Schedule: Friday, August 9 • 16:30 – 16:55

This was a very small session. We covered some of the history of how we built CentOS Linux in the past, and how we retired some of our old tooling to adopt some of the same as Fedora and RHEL. We steered mostly away from the uncertainties around CentOS 8 at the time, and focused on some targeted questions about the content of BaseOS/AppStream/CRB(PowerTools) and discussed some of the decisions we made when choosing our tools.

The goal here was to introduce mbox, and discuss some ways forward but this happened to be the wrong room for that conversation so I mostly let the audience lead where they wanted to go. In general, having an “Intro to CentOS” talk including the history of the project and its relation to Fedora seems to be a good story to continue telling at Flock. 

communishift: A OpenShift cluster for Fedora community managed applications

Presenter: Kevin Fenzi
Duration: 25 minutes [video]
Schedule: Friday, August 9 • 16:30 – 16:55

The session was packed, which was great for the last session of the day, and we didn’t have a lot of time, but we did well. I introduced our communishift setup, how it was structured, what uses there were for it and how to get access to it. There were a number of lively questions (luckily most of which I was ready to answer). People seemed eager to try it out.

Community Platform Engineering hackfest

Organized by: CPE Team
Duration: half-day
Schedule: Sunday, August 11 • 09:00 – 11:50

We split into two groups for the hackfest, because there were plenty of people in the room. One was talking about packaging workflow and the second was going through CPE backlog.

In the first group we brainstormed about what the ideal, long term vision, packaging workflow should be. The discussion came from the realization during flock that a lot of different initiatives are going on around packaging in Fedora to try to make the packagers’ life easier but there is no long-term vision of what we want the workflow to be. This leads to a potential risk of having initiatives compete against each other because of this lack of shared vision. We thus brainstormed about what the long-term vision of the packaging workflow should/could be. Pingou has agreed to send a summary of this brainstorming session to the devel list to gather more inputs, so stay in touch as this discussion should be kicked off soon.

In the second group we talked about ResultsDB, Monitoring of infrastructure and FAS replacement.

ResultsDB was on the backlog because there are plenty of applications using it and the Taskotron, which ResultsDB is part of, will be retired. Fortunately ResultsDB is separate codebase, so we only need to find a new maintainer and be sure that the ResultsDB will continue working. We also need to identify applications that could be tied to taskotron, so we could make changes in advance.

There was a long discussion around Monitoring and Metrics in our infrastructure, both on app level and infra level. We talked about the current situation, where we are using Nagios for monitoring nodes and Kibana, Prometheus for monitoring openshift apps. For metrics collection we are using collectd.

Then we talked about what we want from the Monitoring and Metrics collection in the future. This discussion was long with plenty of ideas from the audience on both app and infra level. We ended up with a nice list:

EFK stack capabilitiesTraffic
Dashboard viewAlerting
Self serviceClear escalation path
Service discoveryAvoid alert fatigue
Avoid maintenance of our own monitoring stackSLE (Service Level Expectation)
SLE – with presets (SLE should be visible directly in monitoring)Aggregation of states
Link to SOP (Standard Operation Procedures)Multiple monitoring points
Periodic reviews

The last thing we discussed was replacement of FAS which is currently used for authentication in Fedora infrastructure. We decided to ask FreeIPA for help, but first we need to work on our requirements.

Whole hackfest time was really productive and helped us to move forward on the things we discussed. We want to thank everybody who attended this hackfest for their help.

Meet your FESCo

Organized by: FESCo
Duration: 50 minutes [video]
Schedule: Sunday, August 11 • 13:00 – 13:50

With some short introductions, we went directly to questions from the audience. There were good questions about who and how to approach about big sweeping changes (like a new arch or recompiling everything for a new cpu). There were introspective questions about how FESCo thought it was doing (over all pretty well). There were questions on modularity and changes process along with questions about when a change was ready to submit. Overall a lively set of questions and we ran out of time before we ran out of questions.

Read more

You can read more about Flock in individual blogs from our team members. Here are links to some of them: