As you’re probably aware, Fedora 29 reached End-of-Life (EOL) status yesterday. The Fedora Program Manager (that’s me!) is responsible for closing any bugs that are still open against that version. Typically, several thousand bugs remain open, so there is a script to do this. This morning, I accidentally closed bugs as EOL that should not have been closed. In the interests of community transparency, I want to share what happened.
How does the script work?
The script is fairly simple: given a comma-separated values (CSV) file with a list of bugs, it iterates over the rows and closes the bugs. So the key input is the CSV file, which is created in the Bugzilla web interface and downloaded for the script to parse.
What went wrong?
Simply put: I messed up. When I created the CSV file, I neglected to specify the version in the Bugzilla search. As a result, I had a CSV file with 20,000 bugs. I started the script and it processed approximately 150 bugs before the community noticed that bugs were being closed inappropriately.
I immediately stopped the script and manually audited the bugs that had been closed. Almost all of them should not have been closed. Some community members reopened bugs themselves and I reopened the rest.
The bug closing script prints a count of bugs before it begins working. I noticed that it said “20,000” and I thought to myself “that seems like a big number”, but I did not think to check previous releases to see how close it was.
How will I prevent this in the future?
For the Fedora 28 EOL, I modified the scripts to take the version and date as arguments. Previously it was hard-coded in a heredoc. Since the script already requires a version, it was easy to add a check to ensure the CSV file does not include bugs from the wrong version. While it’s entirely possible for me to find new ways to mis-close bugs, today’s error should not be repeated.
I apologize to the Fedora community for the confusion and extra work this created. At publication time, the script is running with the correct input.