Caju Pereira#SRE Story: A team was releasing a new feature to production, which had been scheduled with customers for a long time. The code was already released to production, but it was toggled off behind a feature flag. When they toggled it on, nothing happened. (1)
The line of code that wasn't working was a single "if" statement. At one point, I was paged and we had 4 engineers in the call, some with masters in computer science, debugging the issue. We got to the point of adding a breakpoint in production! (2)
All of our monitors and SLOs were healthy, the feature flag system was behaving exactly like we expected and hadn't been changed in a long time. Now, take a guess! What do you think was the underlying cause of this incident? (3)
The issue: The feature flag system expected a string to be exactly equals to "on", and we had mistyped a line-break at the end of this string, which made the feature flag be interpreted as "off"🤦 (4)