In the dungeons bellow the-new-hotness island was impenetrable darkness. It looks like somebody tried to destroy every source of light. Only my own levitating fireball was shedding some light around. Damage was still visible on walls and furniture, but most of it is now repaired to function properly. I’m glad that you are here with me, otherwise it will be a scary experience. But you probably want to hear what happened.

Chasing the Gremlin

Everything started when conclave of mages (Community Platform Engineering Team) decided to move the latest version from Purgatory (staging) to Heaven (production), where the latest features could be used by outside entities (users). Few other mages were helping me with this work, mage from the department for Ethical Use of Magic (security expert) Patrick and mage from the department for Experimental Forms of Communication in Magic (fedora-messaging developer) Aurélien. The move itself was pretty easy even with the amount of new changes. My thanks to other mages for their help.

But this was only the beginning. I hadn’t noticed the gremlin that is hiding in the dungeons bellow the new version of the-new-hotness and let him go free. As they say: “Never let gremlins do what they want to do,” and now I think this is a good saying.

First the gremlin started to ambush messengers to Bugzilla realm and then he sent his own messengers to incorrect destinations (the-new-hotness literally hijacked the existing issues and changed the title). This caused havoc in the realm of Bugzilla, which was quickly reported to me. I figured it out and forced the gremlin to go back to dungeons (The search algorithm for bugzilla only took the component, platform and state of the issue into consideration, not issuer, so I added the issuer to prevent the hijacking). I did this pretty fast and only a few fake messengers were sent by gremlin.

Next thing gremlin did was removing one of the head workers (spectool wants to know username of the current user, even if we don’t use it for anything and OpenShift user has only user id). This was fixed rather quickly, by appointing another head worker to this position (added parameter –user to spectool). Another battle against gremlin was won, but it was not the last.

Gremlin stuck again, this time messing with the messages sent back from various tools that the-new-hotness is using (Output was expected to be string and after porting to python3 it started to return binary string and this wasn’t caught by unit tests, because of mocking 🙁 ). This completely stopped our communication with realm of Koji. I went to the dungeons and chased the gremlin from one tool working space to another and tried to fix everything he did. The gremlin is a cunning beast and always fled before I arrived. After half a day I finally fixed everything gremlin did regarding the tools.

Last battle with gremlin was fought in Kerberos lair. Gremlin convinced the Kerberos to do his evil deeds and so Kerberos decided to block any messenger going to realm of Koji (authentication with Koji through kerberos didn’t work). It took me some time to convince Kerberos that what he is doing is not helpful (change of the configuration to make Kerberos work as I wanted in OpenShift), but I finally caught the gremlin and bring end to his evil doings. Right now he is under the influence of charm spell and cleaning the mess he did.

The whole chase of the gremlin took me few days. So I apologize for any inconvenience this gremlin caused and I hope I will be able to prevent this in the future.

Learning from experience

What did I learnt from this experience? First thing is to not let gremlin do what it wants to do. Second thing is that you should do more testing in Purgatory (Koji builds were not tested on staging and most of the issues above were related to Koji scratch builds). And finally even the tested parts could still have issues.

Post scriptum

This is all for now from the world of release-monitoring.org. Do you like this world and want to join our conclave of mages? Seek me (mkonecny) in the magical yellow pages (IRC freenode -apps) and ask how can you help. Or visit the Bugcronomicon (GitHub issues on Anitya or the-new-hotness) directly and pick something to work on.