Jul 10, 20:26 UTC
Resolved - Our Engineering team has completed remediation of the previously stalled Mongo clusters and those clusters are now online.

This incident is now fully resolved. If you have any questions or continue to experience issues, please reach out to Support from within your account.

Thank you for your patience throughout this incident.

Jul 10, 19:50 UTC
Update - From 16:24 - 18:44 UTC, Load Balancers and Mongo Managed Databases failed to create successfully. Users experienced success messages, but the create processes did not complete as expected.

As of 18:44 UTC, users are able to create Load Balancers and Mongo clusters normally.

As of 19:00 UTC, all stalled Load Balancer creation operations were completed.

Our Engineering team is working to remediate a small number of stalled Mongo cluster creation attempts from the incident impact period. Users may await these remediation efforts, or simply create a new cluster, which will be successful.

We will post a final update once all stalled clusters have been remediated.

Jul 10, 19:02 UTC
Monitoring - The potential fix has completed deployment and has been confirmed to address the root cause of this incident. Users should now be able to create Load Balancers and Mongo Managed Database. Our Engineering team is monitoring the situation and we will post an update as soon as we confirm the issue is fully resolved.

Jul 10, 18:14 UTC
Identified - After further investigation, our Engineering team has confirmed that only Mongo Managed Database creates are impacted, along with Load Balancers. Managed Kubernetes is not impacted by this incident.
A potential fix is currently in progress. We will provide another update as soon as possible.

Jul 10, 18:02 UTC
Investigating - Our Engineering team is investigating an issue with creation of Load Balancers, Managed Databases, and Managed Kubernetes clusters in all regions, via both the Cloud Control Panel and API. During this time, users may see success messages when initially creating resources, but those requests will not complete. We apologize for the inconvenience and will share an update once we have more information.

Jul 11, 19:44 UTC
Resolved - Our Engineering team has confirmed the issue with SMS delivery report delays when sending messages has been fully resolved.

We appreciate your patience throughout this process and if you continue to experience problems, please open a ticket with our support team for further review.

Jul 10, 16:43 UTC
Update - We are continuing to monitor the issue of 2FA SMS codes failing to deliver. Since our last update, we have observed higher success rates in terms of SMS deliverability for two-factor authentication. Our Engineering team has been in close contact with the upstream provider, who has also noted recovery for their services. We will continue to monitor the situation.

In the meantime, if you are experiencing issues receiving 2FA codes, please reach out to our support team for assistance in accessing your account via the link here: https://www.digitalocean.com/company/contact/support

We apologize for the inconvenience.

Jul 10, 06:32 UTC
Update - We are continuing to investigate the issue of 2FA SMS codes failing to deliver. We are closely working with our upstream provider to resolve the issue, we will provide an update when we have more significant information to share.

In the meantime, if you are experiencing issues receiving 2FA codes, please reach out to our support team for assistance in accessing your account via the link here: https://www.digitalocean.com/company/contact/support

We apologize for the inconvenience and thank you for your patience and continued support.

Jul 9, 22:34 UTC
Identified - Our Engineering team has been investigating customer reports of 2FA SMS codes failing to deliver, beginning on July 8. After an extensive review, our team has identified that there is an issue with the message deliverability of an upstream provider. We are closely monitoring the situation and as soon as we have more information to share we will provide further updates.

In the meantime, customers are suggested to reach out to our support team for assistance in accessing their account if they are encountering issues receiving 2FA codes, via the link here:
https://www.digitalocean.com/company/contact/support