Unraveling the Friday Incident: Lessons from Infrastructure as Code

Play this article

Introduction: Entering the Tech Story

Hey there, curious minds of the coding world! Today, we're diving into a tale that's all about learning. Imagine a world where computer spells bring things to life, and clouds aren't just fluffy but also powerful. We're talking about Pulumi, Google Cloud Platform (GCP), and Cloud Run. These are our digital tools, and our story is about a little misstep that happened while using them. So, get ready to peek into the world of Infrastructure as Code (IaC) and discover its wonders and worries.

The Incident: When a Code Move Made Noise

In each project, the Cloud Run service was turned on independently. But, here's the catch: it was enabled in each project without protection from accidental removal. Then, a project that wasn't needed anymore prompted the team to remove its infrastructure code. However, this action unintentionally led to the removal of the Cloud Run service too. The result? The whole company's setup was affected. Cloud Run stopped working and all the services that were running on it disappeared.

Conclusion: Wisdom Found in Code Mistakes

Missteps are like teachers in disguise, and boy, did we learn a thing or two. Lesson one: Keep your treasures safe. Always, always protect important stuff from being accidentally erased. Deletion protection is like a shield in a digital world.

Lesson two is about being extra careful with code. It's like when you're building a tower of blocks – one wrong move and it all falls down. That's how coding works. But here's the good news: tools like Pulumi's "preview" button let you peek into the future. You can see what might happen before you press the big "go" button.

This incident is a fascinating exploration that uncovers glitches within the code and configuration. The team embraced the lesson it offered, using it as a stepping stone to rectify and improve.