Replication Crisis


Jul 18, 13:20 UTC
Resolved - Our Engineering team has successfully resolved the issue that was affecting Reserved IPs and Load Balancers. Between 10:45 UTC and 12:10 UTC today, users may have encountered errors while assigning, reserving, or releasing Reserved IPs, or provisioning Load Balancers. All services have now been fully restored and are functioning normally. If you continue to experience any issues, please open a ticket with our support team. We apologize for any inconvenience this may have caused.
Jul 18, 12:36 UTC
Monitoring - Our Engineering team has implemented a fix to address the issue affecting Reserved IPs and Load Balancers and is actively monitoring the situation. We will provide an update once the issue is fully resolved.
Jul 18, 12:17 UTC
Investigating - Our Engineering team is currently investigating an issue affecting Reserved IPs and Load Balancers. During this time, users may be unable to assign, reserve, or release Reserved IPs, or provision Load Balancers. This impact is limited to IPv4; IPv6 remains unaffected. We apologize for the inconvenience and will share an update as soon as we have more information

© Tom Saater/The New York Times/Redux
Jul 17, 18:10 UTC
Resolved - From 13:12 UTC to 17:18 UTC, users may have experienced errors with App deployment via App Platform. The deployments particularly failed for the Apps with Databases linked to them.
Our Engineering team has now confirmed the resolution of the issue, and App deployment should be working as expected.
If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

© Joanne Lee/Rest of World/iStock
Jul 17, 15:17 UTC
Resolved - Our Engineering team has implemented a fix for the issue that was preventing tickets submitted via the Support Contact Form from reaching our Support team. The form is now working as expected, and all new submissions are being properly routed to our Support team.
Additionally, previously submitted tickets that were impacted by this issue have been successfully routed to our Support Team. The team will address them as soon as possible.
We apologize for any inconvenience caused and appreciate your patience while we worked to resolve this issue.
Jul 17, 11:35 UTC
Identified - Our Engineering team has identified the root cause of the issue with submissions from our Support Contact Form and backlogged submissions are now flowing into our Support queues to be addressed.
Our Support team is now working to address the backlog of submissions as quickly as possible.
Jul 17, 09:46 UTC
Investigating - As of 9:23 UTC, our Engineering team is currently investigating an issue with our Support Contact Form, which users use when they are unable to log into their accounts to submit support tickets, located here: https://www.digitalocean.com/company/contact/support
At this time, users are still able to submit the form, but the submissions are not being received by our Support Team.
We have confirmed the submissions were received by our systems and we're working to get these submissions over to our Support Team, as well as remediate the issue.
If you are able to log into your account, please submit any Support tickets via https://cloudsupport.digitalocean.com/s/ rather than the contact form.
We will post another update as soon as we have further information. In the meantime, we appreciate your patience as we work through this.


© Dhiraj Singh/Bloomberg/Getty Images

© Andrea Mongia for Rest of World


© Rest of World/iStock


© Lauren DeCicca/Getty Images
Jul 10, 23:13 UTC
Resolved - Our Engineering team has fully resolved the issue with new Kubernetes cluster GPU creations. Users with new clusters should be able to create GPU nodes without issue.
If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.
Jul 10, 21:19 UTC
Identified - Our Engineering team has identified an issue impacting creations of GPU nodes on new Kubernetes clusters. New clusters on the latest version will not be able to launch a new GPU node. Existing workloads are not impacted.
Our team is working on deploying a fix for this issue, We will post an update once we have more information.

© A J Ghani for Rest of World


© Rest of World/iStock
Jul 8, 17:01 UTC
Resolved - Our Engineering team has resolved the issue affecting all control plane operations (Create, Read, Update, Delete) for non-MongoDB Managed Database clusters. All services and App Platform deployments with Managed Databases should now be operating normally.
If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.
Jul 8, 16:01 UTC
Monitoring - Our Engineering team has deployed a fix to address the issue affecting all control plane operations (Create, Read, Update, Delete) for non-MongoDB Managed Database clusters. Users should now be able to perform standard database operations successfully. App Platform deployments with Managed Databases should also be succeeding now.
We are now monitoring the situation in order to ensure stability and will post an update once the issue is resolved completely.
Jul 8, 14:57 UTC
Investigating - Our Engineering team is investigating an issue that is impacting all control plane operations (Create, Read, Update, Delete) for all non-Mongo Managed Database clusters. During this time, users may encounter long running queries when attempting those operations, including provisioning new clusters, listing clusters, etc.
In addition, App Platform deploys with Managed Databases are also affected, which may result in delayed or failed deployments for some applications.
We will provide updates as more information becomes available.

© Zinyange Auntony for Rest of World
