How well are data centers within the same region isolated for high availability?

Question

As our SaaS solution grows I want to invest more in high availability.

My first thought was to use vpc peering to tie 3 data centers together, put a database server in each data center and then use them as a logical cluster (MaxScale or Galera). But, I noticed that DigitalOcean does not recommend running their own Managed Database solution across data centers for performance reasons.

As I considered the performance implications I also started to wonder if I was making the HA gain I had hoped for. I remember a few years ago a networking issue (configuration error I think) actually affected both NYC3 and NYC2.

So here is the real question. I haven’t had an outage in NYC3 in over 9 years. If there is an issue big enough to bring down NYC3, would it likely impact NYC1 and NYC2 as well? If that is the case, I might as well deploy my HA cluster in one datacenter, avoid the performance issue, and have disaster recover automation that could relaunch the solution in another data center in any available region.

I still like my idea of a cluster across NYC1, NYC2 and NYC3 but maybe that just isn’t realistic as a HA solution.

Bobby Iliev · Answer

Hey David 👋

I believe that you’re right that DigitalOcean offers managed databases with standby nodes and automated failover, I think that these standby nodes are placed in the same data center as the primary node. If I had to guess, this sounds like by design to keep replication latency low and performance consistent.

While the docs mention multi-region support, that mainly refers to choosing a deployment region, not running a cluster across multiple data centers within the same region. In other words, you can deploy a managed database in a specific regions (e.g. NYC1 or NYC2), but you cannot deploy a single cluster across multiple data centers within that region (e.g. across multiple NYC1 data centers).

So while your idea of setting up MaxScale or Galera across NYC1, NYC2, and NYC3 using VPC peering is technically doable, the real HA gain might not be as strong as it seems. There have been rare cases, like the one you mentioned, where an incident affected more than one NYC DC, so it’s not guaranteed isolation. Plus, the performance hit from synchronous replication across data centers can be a challenge to manage as I mentioned in the other question you had here: https://www.digitalocean.com/community/questions/has-anybody-deployed-their-own-database-cluster-across-data-centers-in-a-region

I would probably stick to a single-DC HA setup for low latency and predictable failover, and then build disaster recovery automation to spin things up in a different region if needed.

That said, I think your idea highlights a real need. You might want to post it on the DigitalOcean Ideas Board, it’s a great place to share feedback and potentially influence future improvements in multi-DC or multi-region architecture options.

Would love to hear what direction you go!

- Bobby