Jul 10, 20:26 UTC
Resolved - Our Engineering team has completed remediation of the previously stalled Mongo clusters and those clusters are now online.

This incident is now fully resolved. If you have any questions or continue to experience issues, please reach out to Support from within your account.

Thank you for your patience throughout this incident.

Jul 10, 19:50 UTC
Update - From 16:24 - 18:44 UTC, Load Balancers and Mongo Managed Databases failed to create successfully. Users experienced success messages, but the create processes did not complete as expected.

As of 18:44 UTC, users are able to create Load Balancers and Mongo clusters normally.

As of 19:00 UTC, all stalled Load Balancer creation operations were completed.

Our Engineering team is working to remediate a small number of stalled Mongo cluster creation attempts from the incident impact period. Users may await these remediation efforts, or simply create a new cluster, which will be successful.

We will post a final update once all stalled clusters have been remediated.

Jul 10, 19:02 UTC
Monitoring - The potential fix has completed deployment and has been confirmed to address the root cause of this incident. Users should now be able to create Load Balancers and Mongo Managed Database. Our Engineering team is monitoring the situation and we will post an update as soon as we confirm the issue is fully resolved.

Jul 10, 18:14 UTC
Identified - After further investigation, our Engineering team has confirmed that only Mongo Managed Database creates are impacted, along with Load Balancers. Managed Kubernetes is not impacted by this incident.
A potential fix is currently in progress. We will provide another update as soon as possible.

Jul 10, 18:02 UTC
Investigating - Our Engineering team is investigating an issue with creation of Load Balancers, Managed Databases, and Managed Kubernetes clusters in all regions, via both the Cloud Control Panel and API. During this time, users may see success messages when initially creating resources, but those requests will not complete. We apologize for the inconvenience and will share an update once we have more information.

Jul 11, 19:44 UTC
Resolved - Our Engineering team has confirmed the issue with SMS delivery report delays when sending messages has been fully resolved.

We appreciate your patience throughout this process and if you continue to experience problems, please open a ticket with our support team for further review.

Jul 10, 16:43 UTC
Update - We are continuing to monitor the issue of 2FA SMS codes failing to deliver. Since our last update, we have observed higher success rates in terms of SMS deliverability for two-factor authentication. Our Engineering team has been in close contact with the upstream provider, who has also noted recovery for their services. We will continue to monitor the situation.

In the meantime, if you are experiencing issues receiving 2FA codes, please reach out to our support team for assistance in accessing your account via the link here: https://www.digitalocean.com/company/contact/support

We apologize for the inconvenience.

Jul 10, 06:32 UTC
Update - We are continuing to investigate the issue of 2FA SMS codes failing to deliver. We are closely working with our upstream provider to resolve the issue, we will provide an update when we have more significant information to share.

In the meantime, if you are experiencing issues receiving 2FA codes, please reach out to our support team for assistance in accessing your account via the link here: https://www.digitalocean.com/company/contact/support

We apologize for the inconvenience and thank you for your patience and continued support.

Jul 9, 22:34 UTC
Identified - Our Engineering team has been investigating customer reports of 2FA SMS codes failing to deliver, beginning on July 8. After an extensive review, our team has identified that there is an issue with the message deliverability of an upstream provider. We are closely monitoring the situation and as soon as we have more information to share we will provide further updates.

In the meantime, customers are suggested to reach out to our support team for assistance in accessing their account if they are encountering issues receiving 2FA codes, via the link here:
https://www.digitalocean.com/company/contact/support

Jul 5, 16:43 UTC
Resolved - Our Engineering team identified and resolved an issue with IPv6 networking in the BLR1 region.

From 12:15 UTC - 15:32 UTC, users may have experienced issues connecting to Droplets and Droplet-based services in the BLR1 region using IPv6 addresses.

Our Engineering team quickly identified the root cause of the incident to be related to a recent maintenance in that region and implemented a fix.

We apologize for the disruption. If you continue to experience any issues, please open a Support ticket from within your account.

Jul 3, 21:36 UTC
Resolved - Our Engineering team identified and resolved an issue with creation of Snapshots and Backups in the NYC3 region.

From 20:11 UTC to 21:04 UTC, users may have experienced errors while taking Snapshots of Droplets in NYC3. Backup creation was also failing, however, Backups will be retried automatically.

Our Engineering team quickly identified the root cause of the incident to be related to capacity on internal storage clusters and were able to rebalance capacity, allowing creations to succeed.

We apologize for the disruption. If you continue to experience any issues, please open a Support ticket from within your account.

Jul 3, 21:16 UTC
Resolved - Beginning July 2nd, 20:55 UTC, team account owners may have seen an issue with removing other users from their team accounts. As of July 3rd 20:47 UTC, a fix was deployed and our Engineering team has confirmed full resolution of the issue. Team owners should be able to remove other users from their teams without issue.

Thank you for your patience. If you continue to experience any problems, please open a support ticket from within your account.

Jul 3, 20:53 UTC
Monitoring - Our Engineering team has completed the rollout of the fix to resolve the issue with team owners being unable to remove users from their team accounts.

We are currently monitoring the situation and will post an update as soon as the issue is fully resolved.

Jul 3, 20:34 UTC
Identified - Our Engineering team has identified an issue with removing team members from active DigitalOcean accounts and is currently working on a fix. During this time, users (team owners) will see an error when attempting to remove team members from their accounts.

We apologize for the inconvenience and will post an update as soon as the fix is in place.

Jul 3, 19:30 UTC
Completed - The scheduled maintenance has been completed.

Jul 3, 18:30 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jul 2, 23:15 UTC
Scheduled - Start Time: July 03 2024 18:30 UTC
End Time: July 03 2024 19:30 UTC

During this time, our Marketplace Engineering team will be performing maintenance on our Managed Kubernetes 1-click application service.

Expected Impact:

During this maintenance, customers will be unable to view existing Kubernetes 1-click installations via the Cloud control panel or install Kubernetes 1-click applications from the Marketplace onto new or existing clusters.

If you have any questions related to this event, please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Thank you,
Team DigitalOcean

Jul 2, 21:09 UTC
Resolved - Our Engineering team identified and resolved an issue impacting multiple services, in multiple regions.

From 19:53 - 20:22 UTC, users may have experienced errors with the following operations, for all regions:
Droplet creates with a root password
Droplet root password resets
Droplet Snapshots
MongoDB cluster creates
Let’sEncrypt certificate creation (for Load Balancers and Spaces)
LoadBalancer creation with certificates
Spaces bucket and key creations
Fetching CSV invoices
Bulk actions in the Spaces UI

Additionally, users could have seen latency while creating and updating Apps.

Our Engineering team quickly identified the root cause of the incident to be an incorrect firewall policy applied to core infrastructure and rolled back the change.

We apologize for the disruption. If you continue to experience any issues, please open a Support ticket from within your account.

Jul 2, 19:35 UTC
Completed - The scheduled maintenance has been completed.

Jul 1, 12:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jul 1, 08:50 UTC
Scheduled - Start: 2024-07-01 1200 UTC
End: 2024-07-08 2100 UTC

During the above window, our Engineering team will be performing ongoing security maintenance on Managed Kubernetes control plane clusters in our SGP1 region.

Expected impact:

While our team will take all possible precautions to prevent impact during this event, some Managed Kubernetes clusters in the region may experience brief periods where control plane connectivity is lost. In such an event, Managed Kubernetes control plane operations such as scaling, resizing, adding nodes and creation of new clusters may be delayed. We will endeavor to keep any such impact to a minimum.

If you have any questions related to this event please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Thank you,
Team DigitalOcean

Jun 28, 21:57 UTC
Resolved - Our Engineering team has confirmed that this incident has been fully resolved.

We appreciate your patience throughout this process and if you continue to experience problems, please open a ticket with our support team for further review.

Jun 28, 21:18 UTC
Monitoring - Our Engineering team has taken actions to mitigate the issue affecting intermittent request failures for Spaces in NYC3 and is monitoring the situation.

The impact has subsided and users should be able to interact with Spaces normally.

We will post an update once we confirm this incident is fully resolved.

Jun 28, 20:37 UTC
Update - Our Engineering team is continuing to investigate the root cause of the issue with intermittent request failures for Spaces in NYC3. Observed error rates are trending down to pre-incident levels, but may spike again.

We will provide an update as soon as we have additional information.

Jun 28, 19:04 UTC
Investigating - Beginning around 17:40 UTC, Our Engineering team is investigating an issue impacting Spaces in our NYC3 region. During this time, users may encounter intermittent request failures for Spaces in NYC3. Additionally, load times for objects in that region may be slower.

We apologize for any inconvenience caused and will provide further updates as soon as possible.

Jun 27, 23:57 UTC
Resolved - Our Engineering team has confirmed that this incident has been fully resolved.

Thank you for your patience. If you continue to experience any problems, please open a ticket with our support team for further review.

Jun 27, 23:22 UTC
Monitoring - Our Engineering team has confirmed that the issue for Droplets with previously failed events has been remediated, and further events should now successfully complete.

We are continuing to monitor this situation and will post an update as soon as the issue is fully resolved.

Jun 27, 21:12 UTC
Update - Our Engineering team has completed deployment of the fix. No additional users should now experience event failures.

Droplets that experienced failed events are now being remediated to fix their event state, so that future events can succeed normally.

We will provide another update once the remediation step is complete.

Jun 27, 20:54 UTC
Identified - Beginning around 16:00 UTC, our Engineering team identified an uptick in event errors for Droplets that utilize a Reserved IP in multiple regions. A subset of users are experiencing errors in processing events, especially "power on", via both API and the Cloud Control Panel.

Our team has identified the root cause of the issue and a fix is currently in progress.

We will provide another update once the fix is confirmed.

Jun 26, 14:10 UTC
Resolved - Our Engineering team has confirmed the full resolution of the issue impacting the Managed Kubernetes cluster creation. As of 12:18 UTC the functionality has been restored completely and users should be able to deploy new clusters via Cloud Control Panel and API.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Jun 26, 13:36 UTC
Monitoring - Our Engineering team has deployed a fix for the issue impacting the Managed Kubernetes cluster creation across all regions. The impact has been mitigated and users should no longer experience any issues deploying new clusters via Cloud Control panel and API.

We are monitoring the situation and will post an update once the incident is completely resolved.

Jun 26, 12:05 UTC
Investigating - Our Engineering team is investigating an issue impacting our Managed Kubernetes service in all regions. Beginning 11:10 UTC, users may have experienced errors while creating new clusters using Control panel and API.

We apologize for the inconvenience and will share an update once we have more information.

Jun 27, 12:00 UTC
Completed - The scheduled maintenance has been completed.

Jun 27, 10:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jun 24, 10:27 UTC
Scheduled - Start: 2024-06-27 10:00 UTC
End: 2024-06-27 12:00 UTC

During the above window, our Networking team will be performing maintenance on core switches in our NYC1 datacenter as part of network upgrades. This is a rescheduled maintenance event previously scheduled for 2024-06-11 10:00 UTC.

Expected Impact:

We anticipate brief interruptions in network traffic from Droplets and Droplet-based services. This impact would be for a duration of 5 to 10 seconds and could occur a few times within the maintenance window. We will endeavor to keep any such impact to a minimum.

We apologize for any inconvenience this short notice causes and thank you for your understanding. If you have any questions related to this issue please send us a ticket from your cloud support page using https://cloudsupport.digitalocean.com/s/createticket


Thank you,
Team DigitalOcean

Jun 25, 16:30 UTC
Completed - The scheduled maintenance has been completed.

Jun 25, 12:30 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jun 23, 15:21 UTC
Scheduled - Start: 2024-06-25 12:30 UTC
End: 2024-06-25 16:30 UTC

During the above window, our Networking team will be making changes to core networking infrastructure, to improve performance and scalability in the BLR1 region.

Expected impact:

These upgrades are designed and tested to be seamless and we do not expect any impact to customer traffic due to this maintenance. If an unexpected issue arises, affected Droplets and Droplet-based services may experience increased latency or a brief disruption in network traffic. We will endeavor to keep any such impact to a minimum.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Jun 19, 23:04 UTC
Resolved - Our engineering team has resolved the issue with creating a monitoring alert in the Monitoring section of the Cloud Control Panel.

Thank you for your patience. If you continue to experience any problems, please open a support ticket from within your account.

Jun 19, 22:22 UTC
Monitoring - Our Engineering team has implemented a fix regarding the issue with creating a monitoring alert in the Monitoring section of the Cloud Control Panel. We are monitoring the situation to ensure there is no recurrence.

We will post another update once we confirm the issue is fully resolved.

Jun 19, 19:25 UTC
Investigating - Our Engineering team is investigating an issue impacting our Monitoring Alerts service.

During this time, users may have been experiencing errors or unexpected delays while trying to create monitoring alerts via Cloud Control Panel.

We apologize for the inconvenience and will post an update as soon as we have additional information.

Jun 18, 13:15 UTC
Resolved - As of 13.10 UTC, our Engineering team has confirmed full resolution of networking in our SFO region.

If you continue to experience problems, please open a ticket with our support team from your Cloud Control Panel. Thank you for your patience and we apologize for any inconvenience.

Jun 18, 12:03 UTC
Monitoring - Our Engineering team has confirmed that the fix implemented earlier was successful in mitigating the issue with connectivity in our SFO regions. Users should now be able to connect to their Droplet and Droplet-based services without any issues.

At this time, all services should now be operating normally. We will monitor this incident for a short period of time to confirm full resolution.

Jun 18, 10:25 UTC
Identified - As of 09:40 UTC, our Engineering team has identified the cause and applied a fix to mitigate the connectivity issue impacting the entire SFO region. We are still looking into this failure, but users should be seeing improvements in reaching their Droplet and Droplet-based services.

We'll continue to monitor the situation to confirm this incident is fully resolved and will post an update soon.

Jun 18, 09:05 UTC
Update - Our Engineering team is continuing to investigate the issue with connectivity in our SFO regions. During this time, users may encounter errors and connection timeouts for Droplet and Droplet-based resources in the SFO region. Our team is diligently working on identifying and mitigating the issue at the earliest.

We regret any inconvenience caused and will post an update as additional information is available.

Jun 18, 08:38 UTC
Investigating - Our Engineering team is investigating an issue with connectivity in our SFO regions. During this time users may experience connection timeouts and errors for Droplet and Droplet-based resources.

We apologize for the inconvenience and will share an update once we have more information.

Jun 17, 19:37 UTC
Resolved - From 17:02 - 18:44 UTC, the Spaces API experienced availability dips. These dips caused Spaces bucket operations in SGP1 for a subset of users to experience latency or 503 errors.

The fix implemented by our Engineering team has been monitored and availability for the Spaces API is stable. Users should be able to interact with their Spaces buckets successfully.

If you continue to experience any issues, please reach out to the Support team from within your account.

Jun 17, 19:07 UTC
Monitoring - Our Engineering team has identified and implemented a fix for an issue impacting Spaces SGP1 region. We are now monitoring the situation closely and will post an update as soon as the issue is fully resolved. Thank you for your patience!

Jun 17, 18:49 UTC
Investigating - Our Engineering team is investigating an issue with spaces in SGP1 region. During this time, users may experience 503 errors while accessing bucket operations. We apologize for the inconvenience and will share an update once we have more information.

Jun 19, 02:00 UTC
Completed - The scheduled maintenance has been completed.

Jun 18, 22:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jun 16, 18:24 UTC
Scheduled - Start: 2024-06-18 22:00 UTC
End: 2024-06-19 02:00 UTC

During the above window, our Networking team will be making changes to our core networking infrastructure to improve performance and scalability in the NYC3 region. This will be the second of the two maintenance activities performed by our team in the region on consecutive days.

Expected Impact:

These upgrades are designed and tested to be seamless and we do not expect any impact to customer traffic due to this maintenance. If an unexpected issue arises, affected Droplets and Droplet-based services may experience a temporary loss of private connectivity between VPCs. We will endeavor to keep any such impact to a minimum.

If you have any questions or concerns regarding this maintenance, please reach out to us by opening up a ticket on your account via https://cloudsupport.digitalocean.com/s/createticket.

Jun 18, 02:00 UTC
Completed - The scheduled maintenance has been completed.

Jun 17, 22:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jun 15, 12:44 UTC
Scheduled - Start: 2024-06-17 22:00 UTC
End: 2024-06-18 02:00 UTC

During the above window, our Networking team will be making changes to our core networking infrastructure to improve performance and scalability in the NYC3 region. This maintenance will occur in two parts on consecutive days and we will send another maintenance notice for the second phase.

Expected Impact:

These upgrades are designed and tested to be seamless and we do not expect any impact to customer traffic due to this maintenance. If an unexpected issue arises, affected Droplets and Droplet-based services may experience a temporary loss of private connectivity between VPCs. We will endeavor to keep any such impact to a minimum.

If you have any questions or concerns regarding this maintenance, please reach out to us by opening up a ticket on your account via https://cloudsupport.digitalocean.com/s/createticket.

Jun 15, 02:56 UTC
Resolved - Our Engineering team has confirmed that this incident has been fully resolved.

We appreciate your patience throughout this process and if you continue to experience problems, please open a ticket with our support team for further review.

Jun 15, 00:11 UTC
Monitoring - Our Engineering team has implemented a fix to fully resolve the issue with our Managed Databases services. At this time, users should no longer face issues creating, resizing, forking, or updating trusted sources for PostgreSQL, MySQL, Redis, and Kafka clusters, in any region, as well as Mongo clusters in NYC regions.

We are monitoring the situation closely and will post an update as soon as we confirm the issue is fully resolved.

Jun 14, 22:48 UTC
Update - Our Engineering team has been able to partially mitigate impact from this incident. Creating, resizing, forking, and updates to trusted sources of PostgreSQL, MySQL, Redis, and Kafka clusters, in all regions, is now working as expected. Operations previously attempted on clusters, such as firewall rule attempts, will need to be resubmitted or have a relevant event (i.e. k8s node changes, tag events, etc.) occur from now on. Users should start to see recovery and be able to initiate new operations normally.

Operations for Mongo clusters remain affected at this time.

Jun 14, 19:22 UTC
Update - Our Engineering team continues work to address the root cause of this issue and remediate customer impact.

We apologize for the continued interruption to the Managed Databases Control Plane and will provide another update as soon as we have more information.

Jun 14, 17:39 UTC
Identified - During the course of investigation, our Engineering team has identified additional operations are impacted by this incident.

Users may experience errors with creating, resizing, or updating trusted sources for PostgreSQL, MySQL, Redis, and Kafka clusters, in all regions, as well as Mongo clusters in NYC regions.

Due to updates to trusted sources being impacted, connections to Database clusters from newly added trusted sources will fail. This includes new Managed Kubernetes nodes, Droplets, and Apps using Databases.

Our team is working to deploy a fix and we will provide another update soon.

Jun 14, 17:27 UTC
Investigating - Our Engineering team is investigating an issue impacting new Managed Database creations. During this time, users may receive failures while creating new Managed Database clusters across all regions. We apologize for the inconvenience and will share an update once we have more information.

Jun 12, 20:23 UTC
Resolved - Our Engineering team identified and resolved an issue impacting multiple pages in the Cloud Control Panel.

From 16:12 - 16:26 UTC, users may have experienced access denied errors while loading pages associated with different services on the Cloud Control Panel. The impacted pages included Managed Database, Block Storage Volumes, Reserved IPs, Domains, Settings, App Platform, Applications & API, Container Registry, Functions, Managed Kubernetes and Monitoring.

Our Engineering team was able to take quick action to mitigate the impact and resolve the issue.

Thank you for your patience, and we apologize for any inconvenience. If you continue to experience any issues, please open a Support ticket for further analysis.