Jul 23, 19:20 UTC
Resolved - From 17:22 UTC to 17:27 UTC, we experienced an issue with requests to the Cloud Control Panel and API

During that timeframe, users may have experienced an increase in 5xx errors for Cloud/API requests. The issue self-resolved quickly and our Engineering team is continuing to investigate root cause to ensure it does not occur again.

Thank you for your patience, and we apologize for any inconvenience. If you continue to experience any issues, please open a Support ticket for further analysis.

Aug 3, 02:00 UTC
Completed - The scheduled maintenance has been completed.

Jul 31, 02:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jul 23, 02:47 UTC
Scheduled - Maintenance Dates: July 31st, August 1st and 2nd
Maintenance Window: 02:00 - 10:00 (UTC)

During the maintenance window on the above dates, our on-site team will be performing ongoing maintenance on redundant power systems in our TOR1 region. This is phase 2 of a 2-part effort and the maintenance will be performed during the same time window (02:00 - 10:00 UTC) across three days: July 31st, August 1st and 2nd.

Expected Impact:

We don’t expect to see any impact to customer resources as the electrical systems are redundant and our team will isolate one unit at a time during the maintenance. In the highly unlikely event that we lose power to both systems simultaneously, affected Droplets and Droplet-based services would experience downtime for a brief period of time. We will endeavor to keep this to a minimum should that situation arise.

If you have any questions or concerns related to this maintenance, please send us a ticket from your cloud support page https://cloudsupport.digitalocean.com/s/createticket

Jul 28, 02:00 UTC
Completed - The scheduled maintenance has been completed.

Jul 25, 02:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jul 23, 02:45 UTC
Scheduled - Maintenance Dates: July 25th, 26th, and 27th
Maintenance Window: 02:00 - 10:00 (UTC)

During the maintenance window on the above dates, our on-site team will be performing ongoing maintenance on redundant power systems in our TOR1 region. This is phase 1 of a 2-part effort and the maintenance will be performed during the same time window (02:00 - 10:00 UTC) across three days: July 25th, 26th, and 27th.

Expected Impact:

We don’t expect to see any impact to customer resources as the electrical systems are redundant and our team will isolate one unit at a time during the maintenance. In the highly unlikely event that we lose power to both systems simultaneously, affected Droplets and Droplet-based services would experience downtime for a brief period of time. We will endeavor to keep this to a minimum should that situation arise.

If you have any questions or concerns related to this maintenance, please send us a ticket from your cloud support page https://cloudsupport.digitalocean.com/s/createticket.

Jul 23, 01:08 UTC
Completed - The scheduled maintenance has been completed.

Jul 22, 22:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jul 20, 22:18 UTC
Scheduled - Start: 2024-07-22 22:00 UTC
End: 2024-07-23 02:00 UTC

Hello,

During the above window, our networking team will be making changes to our core networking infrastructure to improve performance and scalability in the NYC2 region.

Expected Impact:

These upgrades are designed and tested to be seamless, and we do not expect any impact on customer traffic due to this maintenance. Should an unexpected issue arise, a possible outcome would be a temporary loss of connectivity for impacted Droplets. Any impact would also extend to Droplet-dependent services in the NYC2 region such as Managed Databases, Load Balancers, App Platform, and Managed Kubernetes. We will endeavor to keep any such impact to a minimum.

If you have any questions or concerns regarding this maintenance, please reach out to us by opening up a ticket on your account via https://cloudsupport.digitalocean.com/s/createticket/.

Thank you,
DigitalOcean Networking Team

Jul 18, 21:34 UTC
Resolved - From 19:33 on to 21:02 UTC on July 18th, App Platform users may have experienced delays when deploying new Apps or when deploying updates to existing Apps in SFO3.

Our engineering team has deployed a fix for this issue. The impact has been resolved and users should no longer see any issues with the impacted services.

If you continue to experience problems, please open a ticket with our support team from your Cloud Control Panel. Thank you for your patience, and we apologize for any inconvenience.

Jul 19, 01:41 UTC
Completed - The scheduled maintenance has been completed.

Jul 18, 22:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jul 18, 21:10 UTC
Scheduled - During the above window, our Networking team will be making changes to our core networking infrastructure to improve performance and scalability in the NYC1 region. This will be the second of the two maintenance activities performed by our team in the region on consecutive days.

Expected Impact:

These upgrades are designed and tested to be seamless and we do not expect any impact to customer traffic due to this maintenance. If an unexpected issue arises, affected Droplets and Droplet-based services may experience a temporary loss of private connectivity between VPCs. We will endeavor to keep any such impact to a minimum.

If you have any questions or concerns regarding this maintenance, please reach out to us by opening up a ticket on your account via https://cloudsupport.digitalocean.com/s/createticket.

Jul 24, 21:00 UTC
Completed - The scheduled maintenance has been completed.

Jul 24, 20:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jul 17, 13:30 UTC
Scheduled - Start: 2024-07-24 20:00 UTC
End: 2024-07-24 21:00 UTC

During the above window, our Engineering team will be performing maintenance on principal infrastructure in order to improve reliability of the services. Please note that existing infrastructure will continue running without issue. This maintenance impacts create, read, update, and delete (CRUD) operations in all regions.

Expected Impact:

During the maintenance window, users may experience increased latency for the following platform operations:

Cloud Control Panel and API operations
Event processing
Droplet creates, resizes, rebuilds, and power events
Managed Kubernetes reconciliation and scaling
Load Balancer operations
Container Registry operations
App Platform operations
Managed Database creation and scaling

We expect to see two periods of 10 second impact, for a total of 20 seconds within the hour window. If unexpected impact occurs or continues for longer than expected, we will provide updates via our public status page.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Jul 18, 16:00 UTC
Completed - The scheduled maintenance has been completed.

Jul 18, 14:30 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jul 17, 16:26 UTC
Scheduled - Start Time: July 18 2024 14:30 UTC
End Time: July 18 2024 16:00 UTC

During this time, our Marketplace Engineering team will be performing maintenance on our Managed Kubernetes 1-click application service.

Expected Impact:

During this maintenance, customers will be unable to view existing Kubernetes 1-click installations via the Cloud control panel or install new Kubernetes 1-click applications from the Marketplace onto new or existing clusters.

If you have any questions related to this event, please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Thank you,
Team DigitalOcean

Jul 17, 18:49 UTC
Resolved - Our Engineering team has confirmed the full resolution of the issue impacting the ability to create and manage Functions through the Cloud Control Panel and API in our TOR1 region. We appreciate your patience throughout the process.

If you continue to see errors please open a ticket with our Support team and we will be glad to assist you further.

Jul 17, 17:46 UTC
Monitoring - The fix was deployed successfully and the impact appears to have subsided. Users should now be able to create and update functions, triggers and related settings via Cloud Control Panel and API.

We are monitoring the situation and will post an update once we can fully confirm resolution.

Jul 17, 17:24 UTC
Identified - Our Engineering team has identified an issue with Functions in our TOR1 region and a fix is currently being deployed.

During this time, users may experience errors while creating or editing Functions, triggers, and related settings via the Cloud Control Panel and API.

We apologize for the disruption and will post an update once the fix has finished rolling out.

Jul 17, 01:31 UTC
Completed - The scheduled maintenance has been completed.

Jul 16, 22:15 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jul 16, 22:14 UTC
Scheduled - Start: 2024-07-16 22:00 UTC
End: 2024-07-17 02:00 UTC


During the above window, our Networking team will be making changes to our core networking infrastructure to improve performance and scalability in the NYC1 region. This maintenance will occur in two parts on consecutive days and we will send another maintenance notice for the second phase.

Expected Impact:

These upgrades are designed and tested to be seamless and we do not expect any impact to customer traffic due to this maintenance. If an unexpected issue arises, affected Droplets and Droplet-based services may experience a temporary loss of private connectivity between VPCs. We will endeavor to keep any such impact to a minimum.

If you have any questions or concerns regarding this maintenance, please reach out to us by opening up a ticket on your account via https://cloudsupport.digitalocean.com/s/createticket.

Jul 16, 11:48 UTC
Resolved - Our Engineering team has resolved the issue with processing payments via PayPal on our platform. Services should now be operating normally.
If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Jul 16, 11:15 UTC
Update - Our Engineering team has monitored and confirmed that the impact of processing payments via PayPal has subsided. Users should not see any issues with making payments via PayPal on our platform.
We will post an update as soon as the issue is fully resolved.

Jul 16, 11:12 UTC
Monitoring - Our Engineering team continues to monitor the issue with processing payments via PayPal on our platform. The impact is mitigated and users should not see any issues when trying to use PayPal. We will post another update once we confirm the full resolution. We apologize for the inconvenience and will share an update once we have more information.

Jul 16, 10:34 UTC
Identified - Our Engineering team has identified the cause of the issue with processing payments via PayPal on our platform, and has implemented a fix. We are currently monitoring the situation and will post an update as soon as the issue is fully resolved.

Jul 16, 10:08 UTC
Investigating - Our Engineering team is investigating an issue with payments using PayPal. During this time, users may experience timeouts/failures when trying to use PayPal. We apologize for the inconvenience and will share an update once we have more information.

Jul 10, 20:26 UTC
Resolved - Our Engineering team has completed remediation of the previously stalled Mongo clusters and those clusters are now online.

This incident is now fully resolved. If you have any questions or continue to experience issues, please reach out to Support from within your account.

Thank you for your patience throughout this incident.

Jul 10, 19:50 UTC
Update - From 16:24 - 18:44 UTC, Load Balancers and Mongo Managed Databases failed to create successfully. Users experienced success messages, but the create processes did not complete as expected.

As of 18:44 UTC, users are able to create Load Balancers and Mongo clusters normally.

As of 19:00 UTC, all stalled Load Balancer creation operations were completed.

Our Engineering team is working to remediate a small number of stalled Mongo cluster creation attempts from the incident impact period. Users may await these remediation efforts, or simply create a new cluster, which will be successful.

We will post a final update once all stalled clusters have been remediated.

Jul 10, 19:02 UTC
Monitoring - The potential fix has completed deployment and has been confirmed to address the root cause of this incident. Users should now be able to create Load Balancers and Mongo Managed Database. Our Engineering team is monitoring the situation and we will post an update as soon as we confirm the issue is fully resolved.

Jul 10, 18:14 UTC
Identified - After further investigation, our Engineering team has confirmed that only Mongo Managed Database creates are impacted, along with Load Balancers. Managed Kubernetes is not impacted by this incident.
A potential fix is currently in progress. We will provide another update as soon as possible.

Jul 10, 18:02 UTC
Investigating - Our Engineering team is investigating an issue with creation of Load Balancers, Managed Databases, and Managed Kubernetes clusters in all regions, via both the Cloud Control Panel and API. During this time, users may see success messages when initially creating resources, but those requests will not complete. We apologize for the inconvenience and will share an update once we have more information.

Jul 11, 19:44 UTC
Resolved - Our Engineering team has confirmed the issue with SMS delivery report delays when sending messages has been fully resolved.

We appreciate your patience throughout this process and if you continue to experience problems, please open a ticket with our support team for further review.

Jul 10, 16:43 UTC
Update - We are continuing to monitor the issue of 2FA SMS codes failing to deliver. Since our last update, we have observed higher success rates in terms of SMS deliverability for two-factor authentication. Our Engineering team has been in close contact with the upstream provider, who has also noted recovery for their services. We will continue to monitor the situation.

In the meantime, if you are experiencing issues receiving 2FA codes, please reach out to our support team for assistance in accessing your account via the link here: https://www.digitalocean.com/company/contact/support

We apologize for the inconvenience.

Jul 10, 06:32 UTC
Update - We are continuing to investigate the issue of 2FA SMS codes failing to deliver. We are closely working with our upstream provider to resolve the issue, we will provide an update when we have more significant information to share.

In the meantime, if you are experiencing issues receiving 2FA codes, please reach out to our support team for assistance in accessing your account via the link here: https://www.digitalocean.com/company/contact/support

We apologize for the inconvenience and thank you for your patience and continued support.

Jul 9, 22:34 UTC
Identified - Our Engineering team has been investigating customer reports of 2FA SMS codes failing to deliver, beginning on July 8. After an extensive review, our team has identified that there is an issue with the message deliverability of an upstream provider. We are closely monitoring the situation and as soon as we have more information to share we will provide further updates.

In the meantime, customers are suggested to reach out to our support team for assistance in accessing their account if they are encountering issues receiving 2FA codes, via the link here:
https://www.digitalocean.com/company/contact/support

Jul 5, 16:43 UTC
Resolved - Our Engineering team identified and resolved an issue with IPv6 networking in the BLR1 region.

From 12:15 UTC - 15:32 UTC, users may have experienced issues connecting to Droplets and Droplet-based services in the BLR1 region using IPv6 addresses.

Our Engineering team quickly identified the root cause of the incident to be related to a recent maintenance in that region and implemented a fix.

We apologize for the disruption. If you continue to experience any issues, please open a Support ticket from within your account.

Jul 3, 21:36 UTC
Resolved - Our Engineering team identified and resolved an issue with creation of Snapshots and Backups in the NYC3 region.

From 20:11 UTC to 21:04 UTC, users may have experienced errors while taking Snapshots of Droplets in NYC3. Backup creation was also failing, however, Backups will be retried automatically.

Our Engineering team quickly identified the root cause of the incident to be related to capacity on internal storage clusters and were able to rebalance capacity, allowing creations to succeed.

We apologize for the disruption. If you continue to experience any issues, please open a Support ticket from within your account.

Jul 3, 21:16 UTC
Resolved - Beginning July 2nd, 20:55 UTC, team account owners may have seen an issue with removing other users from their team accounts. As of July 3rd 20:47 UTC, a fix was deployed and our Engineering team has confirmed full resolution of the issue. Team owners should be able to remove other users from their teams without issue.

Thank you for your patience. If you continue to experience any problems, please open a support ticket from within your account.

Jul 3, 20:53 UTC
Monitoring - Our Engineering team has completed the rollout of the fix to resolve the issue with team owners being unable to remove users from their team accounts.

We are currently monitoring the situation and will post an update as soon as the issue is fully resolved.

Jul 3, 20:34 UTC
Identified - Our Engineering team has identified an issue with removing team members from active DigitalOcean accounts and is currently working on a fix. During this time, users (team owners) will see an error when attempting to remove team members from their accounts.

We apologize for the inconvenience and will post an update as soon as the fix is in place.

Jul 3, 19:30 UTC
Completed - The scheduled maintenance has been completed.

Jul 3, 18:30 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jul 2, 23:15 UTC
Scheduled - Start Time: July 03 2024 18:30 UTC
End Time: July 03 2024 19:30 UTC

During this time, our Marketplace Engineering team will be performing maintenance on our Managed Kubernetes 1-click application service.

Expected Impact:

During this maintenance, customers will be unable to view existing Kubernetes 1-click installations via the Cloud control panel or install Kubernetes 1-click applications from the Marketplace onto new or existing clusters.

If you have any questions related to this event, please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Thank you,
Team DigitalOcean

Jul 2, 21:09 UTC
Resolved - Our Engineering team identified and resolved an issue impacting multiple services, in multiple regions.

From 19:53 - 20:22 UTC, users may have experienced errors with the following operations, for all regions:
Droplet creates with a root password
Droplet root password resets
Droplet Snapshots
MongoDB cluster creates
Let’sEncrypt certificate creation (for Load Balancers and Spaces)
LoadBalancer creation with certificates
Spaces bucket and key creations
Fetching CSV invoices
Bulk actions in the Spaces UI

Additionally, users could have seen latency while creating and updating Apps.

Our Engineering team quickly identified the root cause of the incident to be an incorrect firewall policy applied to core infrastructure and rolled back the change.

We apologize for the disruption. If you continue to experience any issues, please open a Support ticket from within your account.

Jul 2, 19:35 UTC
Completed - The scheduled maintenance has been completed.

Jul 1, 12:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jul 1, 08:50 UTC
Scheduled - Start: 2024-07-01 1200 UTC
End: 2024-07-08 2100 UTC

During the above window, our Engineering team will be performing ongoing security maintenance on Managed Kubernetes control plane clusters in our SGP1 region.

Expected impact:

While our team will take all possible precautions to prevent impact during this event, some Managed Kubernetes clusters in the region may experience brief periods where control plane connectivity is lost. In such an event, Managed Kubernetes control plane operations such as scaling, resizing, adding nodes and creation of new clusters may be delayed. We will endeavor to keep any such impact to a minimum.

If you have any questions related to this event please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Thank you,
Team DigitalOcean

Jun 28, 21:57 UTC
Resolved - Our Engineering team has confirmed that this incident has been fully resolved.

We appreciate your patience throughout this process and if you continue to experience problems, please open a ticket with our support team for further review.

Jun 28, 21:18 UTC
Monitoring - Our Engineering team has taken actions to mitigate the issue affecting intermittent request failures for Spaces in NYC3 and is monitoring the situation.

The impact has subsided and users should be able to interact with Spaces normally.

We will post an update once we confirm this incident is fully resolved.

Jun 28, 20:37 UTC
Update - Our Engineering team is continuing to investigate the root cause of the issue with intermittent request failures for Spaces in NYC3. Observed error rates are trending down to pre-incident levels, but may spike again.

We will provide an update as soon as we have additional information.

Jun 28, 19:04 UTC
Investigating - Beginning around 17:40 UTC, Our Engineering team is investigating an issue impacting Spaces in our NYC3 region. During this time, users may encounter intermittent request failures for Spaces in NYC3. Additionally, load times for objects in that region may be slower.

We apologize for any inconvenience caused and will provide further updates as soon as possible.