Blog

On a Sunday afternoon in June, Google Cloud experienced a major five-hour outage.  Besides taking down many Google products, it took down huge swaths of the Internet partially or completely.  These services included Shopify, Snapchat, Discord, Vimeo, Apple iCloud, Nest AC controllers, and many other services.

Why did Google's failure affect so many other services?  Because they depend upon Google Cloud.

In software development, we often use the term "Dependencies."

A dependency is a program or service upon which another program or service relies.  For example, our FASO image uploading system has several dependencies:  We use a service called FileStack to manage the uploader itself.  We then process the images with a program called ImageMagick.  We then store the images in the cloud using Amazon's Simple Storage Service.  Lastly, we copy the images over to Google's Storage Cloud.

I have quickly named four dependencies of our image upload system: Filestack, ImageMagick, Amazon S3, Google Storage.


And, dependencies don't just apply to software.  For example, if you're responsible for a project, you might need to enlist the help of others.  Perhaps you've hired someone to write or proofread your copy.  That person has become a dependency of your project.

Let's say you commit to finish a project by the end of this week.  And maybe you're not comfortable handling writing marketing copy.  So you turn that piece of the project over to someone else and they agree to do it for you.  What do you do if they don't finish writing your copy in time?

If I ask you (if you were a BoldBrush employee) this Friday: "how's that project going?"  Are you going to say, "I'm still waiting for to get that copy written?"  Does that let you off the hook for not finishing the project?  Do you think our customers would let us off the hook for their sites being down by simply saying "It's not us, we're waiting on Google?"

Let me answer that for you:  At BoldBrush, it's not okay to fail because of a failure in your dependencies.   You are always dependent upon others – other teams, partners, vendors, products, services, software packages. 

Developing a mentality of managing your dependencies is key to being a leader at BoldBrush. 

So how do you do this?  One main way is this: don’t just trust. Instead, as Reagan said, trust and verify. This "verify" step should not be seen as a challenge to each other, but should be modus operandi for BoldBrush.  When it comes to partners, we attempt to negotiate a managed service level on commitments and capabilities – and we hold them accountable. The other way to manage your dependencies is to develop a plan B.  Develop a backup or fail-over solution.  Perhaps you know two copy writers and could call the 2nd one if the first one can't finish your needed copy in time.

Here's a good example of how to structure your backup/fail-over solutions:  I mentioned above that our FASO image services DO actually depend on Google Cloud.  Why didn't Google's failure take us down?  In fact, we did get an alert on that Sunday afternoon regarding our images.  We investigated and realized that the problem was on Google's side and wasn't causing any customer facing issues, because our systems simply switched over to Amazon S3.  So we simply waited for Google to fix their issues and then took the few steps needed on our end to "repair the damage" so to speak.  Not a single FASO customer saw an issue with their site.

This is because, at BoldBrush, we actively manage our dependencies.  We have built our image storage system to use both Amazon and Google.  And it can switch between the two as needed.  So while Shopify was busy the following Monday explaining to their customers why their stores were down on Sunday afernoon, that Monday was business as usual at BoldBrush.

That's the power of managing your dependencies.