Like many other internet orgs this morning, ours started with an outage.

As I’m sure you can guess, this was due to the massive DDoS Attack Against Dyn DNS.

If, like us, you use Mongo’s MMS cloud management service, then your site may still be down. This is because MMS uses Dyn for DNS resolution internally within the cluster. As a result our entire cluster could not talk to itself.

(fake data, but you’d see something like this)
fake-mms

Our solution was simple: we did an endrun around the DNS DDOS attack by compiling and appending our internal IP to hostname mappings directly into the “/etc/hosts” file throughout our cluster. A few minutes later our Mongo cluster brought itself back online. While this was an old-school approach, it was simple and it worked. Hopefully this strategy will help others that remain impacted.