❌

Normal view

Received β€” 16 September 2024 ⏭ DigitalOcean Status - Incident History

Rescheduled: NYC2 Network Maintenance

18 September 2024 at 18:00

Sep 18, 23:00 UTC
Completed - The scheduled maintenance has been completed.

Sep 18, 21:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Sep 16, 03:59 UTC
Scheduled - The Infrastructure Maintenance in NYC2 which was previously scheduled to start on 2024-09-09 at 21:00 UTC has been rescheduled to the following window:

Start: 2024-09-18 21:00 UTC
End: 2024-09-18 23:00 UTC

During the above window, our Networking team will be performing maintenance on the management network firewalls in the NYC2 region.

Expected Impact:

During the maintenance period, users may experience delays of up to 10 minutes with Event Processing on Droplets and Droplet-based services in NYC2. We will endeavor to keep this to a minimum for the duration of the change.

If you have any questions or concerns regarding this maintenance, please reach out to us by opening up a ticket on your account via https://cloudsupport.digitalocean.com/s/createticket/

Connectivity issues to cloud panel, API, and other endpoints

16 September 2024 at 02:00

Sep 16, 07:00 UTC
Resolved - Our Engineering team identified an issue with accessing Cloud Panel and Application deployment which is due to the increased level of HTTP 499 errors rates at Upstream Provider end.

From 04:21 - 06:06 UTC, users may have experienced issues while accessing Cloud Panel and Application deployments.

Our Upstream provider and the Engineering team closely worked together to resolve the issue.

The impact has completely subsided and users should no longer see any issues with the impacted services.

If you continue to experience problems, please open a ticket with our support team from your Cloud Control Panel. Thank you for your patience, and we apologize for any inconvenience.

Received β€” 12 September 2024 ⏭ DigitalOcean Status - Incident History

Droplet creates in LON1

11 September 2024 at 19:56

Sep 12, 00:56 UTC
Resolved - From 22:03 September 11 to 00:13 September 12 UTC, users may have experienced failures with creating Droplets in the LON1 region.

Our Engineering team has confirmed resolution of this issue with users now being able to create Droplets in LON1. Thank you for your patience.

If you continue to experience any problems, please open a support ticket from within your account.

Sep 12, 00:23 UTC
Monitoring - Our Engineering team has implemented mitigations for the intermittent issues with Droplet creates in LON1. The team is monitoring the situation, and we will share an update once this is fully resolved.

Sep 11, 23:45 UTC
Investigating - Our Engineering team is investigating an issue with Droplet creation in LON1. During this time, users may see an elevated failure rate for the creation of Droplets and other Droplet-based products, such as Load Balancers, Managed Databases, or Kubernetes, in LON1.

Network Connectivity in BLR1

11 September 2024 at 12:22

Sep 11, 17:22 UTC
Resolved - From 15:20 - 16:10 UTC, our Engineering team observed an issue with network connectivity in the BLR1 region. During this time, users might have experienced varying amounts of packet loss and high latency with network connections going in or out of the BLR1 region, resulting in issues connecting to DigitalOcean services.

As of 16:10 UTC, the impact has subsided and users should no longer be facing network connectivity issues. We apologize for the inconvenience and if you are still experiencing issues or have any additional questions then please open a support ticket from within your account.

Received β€” 11 September 2024 ⏭ DigitalOcean Status - Incident History

New Customer Sign Ups

10 September 2024 at 16:06

Sep 10, 21:06 UTC
Resolved - From 18:23 to 19:21 UTC, users were unable to complete registration of new cloud.digitalocean.com accounts, due to an upstream outage.

This issue is now fully resolved and users should be able to sign up normally.

We apologize for the inconvenience and invite users to retry any failed attempts. If you continue to experience issues or have any questions, please reach out to Support from within your account.

Sep 10, 20:48 UTC
Monitoring - Our Engineering team has confirmed there was an upstream outage that affected a component of the sign up flow for cloud.digitalocean.com accounts. The outage has been resolved by the provider.

We are monitoring the situation and will post an update shortly.

Sep 10, 19:50 UTC
Identified - Our Engineering team has identified the issue impacting new account sign-ups. Users should no longer encounter errors and should be able to sign up again.

Our Engineering team continues to monitor the situation and we will post an update as soon as additional information is available.

Sep 10, 19:18 UTC
Investigating - Our Engineering team is investigating an issue with users attempting to sign up for new cloud.digitalocean.com accounts. At this time, users attempting to sign up are unable to complete registration.

Existing users are unaffected and should be able to sign in normally.

We apologize for the inconvenience and will share an update once we have more information.

Extension of Core Infrastructure Maintenance in NYC2

10 September 2024 at 16:04

Sep 10, 21:04 UTC
Completed - Our Engineering team has successfully completed this maintenance.

Thank you for your patience throughout the extended window. If you have any questions or concerns, please feel free to reach out to Support from within your account.

Sep 10, 17:55 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Sep 10, 17:54 UTC
Scheduled - Our Engineering team has run into some operations taking longer than expected for the previous maintenance window here: https://status.digitalocean.com/incidents/vmtp7c19bq2l.

No user impact has occurred as a result of those operations and we do not expect any customer impact, but there is a possibility impact could occur.

Users could experience issues interacting with the control plane for services, via both the Cloud Control Panel and API. This would include operations such as creates, deletes, updates, Snapshots, etc.

Existing services will continue to run normally.

We will update this page if impact occurs.

Received β€” 10 September 2024 ⏭ DigitalOcean Status - Incident History

DockerHub Reachability in SGP1

10 September 2024 at 00:11

Sep 10, 05:11 UTC
Resolved - Our Engineering team identified a networking issue with our upstream provider in the SGP1 region. From 00:25 to 04:00 UTC, users may have experienced connectivity issues to DockerHub from Droplets, App Platform and Managed Kubernetes in the SGP1 region.

Our Engineering team has confirmed there is no longer any impact and connectivity should be restored for all customers.

We apologize for the inconvenience. If you continue to experience any issues, please open a support ticket from within your account.

Resizing Droplet Attached Block Storage Volumes

9 September 2024 at 17:59

Sep 9, 22:59 UTC
Resolved - From 17:04 UTC to 21:43 UTC, users may not have seen the option to resize an attached Block Storage Volume via the Cloud Control Panel, or may have encountered errors when attempting to resize attached Volumes via the API.

Our Engineering team has confirmed the resolution of the issue and Volume resizes should now be functioning normally in all regions.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Sep 9, 21:58 UTC
Monitoring - Our Engineering team have implemented a fix for the issue of resizing Droplet-attached Block Storage Volumes in all regions and are monitoring the situation closely. We will post another update once we're confident that the issue is fully resolved.

Sep 9, 21:15 UTC
Investigating - Our Engineering team is investigating an issue with resizing Droplet-attached Block Storage Volumes in all regions. Beginning at 17:04 UTC, users may not see the option to resize the Volume via the Cloud Control Panel, as well as see errors when attempting to resize attached Volumes via the API.

At this time, unattached Volumes are able to be resized correctly.

We will post an update once further information is available.

Received β€” 8 September 2024 ⏭ DigitalOcean Status - Incident History

Rescheduled: Core Infrastructure Maintenance in NYC2

10 September 2024 at 12:30

Sep 10, 17:30 UTC
Completed - Our Engineering team has run into some operations taking longer than expected within this maintenance. No user impact has occurred as a result of those operations.

We will be extending this maintenance window to 22:00 UTC via a new maintenance post.

Sep 9, 13:30 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Sep 7, 13:50 UTC
Scheduled - The Infrastructure Maintenance in NYC2 which was previously scheduled to start on 2024-09-03 at 13:30 UTC has been rescheduled to the following window:

Start: 2024-09-09 13:30 UTC
End: 2024-09-09 17:30 UTC

And

Start: 2024-09-10 13:30 UTC
End: 2024-09-10 17:30 UTC

During the above window, our Engineering team will be performing maintenance on principal infrastructure in the NYC2 region in order to improve the reliability of the services. The maintenance is divided into two phases and will be performed on consecutive days during the same time as mentioned above.

Expected Impact:

During the maintenance window users may experience issues interacting with their services including Droplets, Managed Kubernetes, Load Balancers, Container Registry, and App Platform. Users may also experience delays or failures with event processing on Droplets and Droplet-based services. We will endeavor to keep this to a minimum for the duration of the change.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Thank you,
Team DigitalOcean

Received β€” 7 September 2024 ⏭ DigitalOcean Status - Incident History

NYC2 Network Maintenance

9 September 2024 at 11:05

Sep 9, 16:05 UTC
Completed - Hello,

The NYC2 Network Maintenance which was previously scheduled to start today (2024-09-09) at 21:00 UTC has been rescheduled to the following window:

Start: 2024-09-18 21:00 UTC
End: 2024-09-18 23:00 UTC

We apologize for any inconvenience this short notice may have caused and thank you for your understanding.
If you have questions or concerns about this maintenance, please reach out to us by opening up a ticket on your account.

Thank you,
DigitalOcean Networking Team

Sep 7, 11:42 UTC
Scheduled - Start: 2024-09-09 21:00 UTC
End: 2024-09-09 23:00 UTC

Hello,

During the above window, our Networking team will be performing maintenance on the management network firewalls in the NYC2 region.

Expected Impact:

During the maintenance period, users may experience delays of up to 10 minutes with Event Processing on Droplets and Droplet-based services in NYC2. We will endeavor to keep this to a minimum for the duration of the change.

If you have any questions or concerns regarding this maintenance, please reach out to us by opening up a ticket on your account via https://cloudsupport.digitalocean.com/s/createticket/.

Thank you,
DigitalOcean Networking Team

Received β€” 1 September 2024 ⏭ DigitalOcean Status - Incident History

Core Infrastructure Maintenance

3 September 2024 at 13:00

Sep 3, 18:00 UTC
Completed - The scheduled maintenance has been completed.

Sep 3, 14:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Aug 31, 14:21 UTC
Scheduled - Start: 2024-09-03 14:00 UTC
End: 2024-09-03 18:00 UTC

During the above window, our Engineering team will be performing additional maintenance on core infrastructure in our production environment. This is the second part of the maintenance completed on 29th August (posted here https://status.digitalocean.com/incidents/wb3g6x6q55ct) to resolve the root cause of an issue discovered in a previous incident, details of which can be found here:

https://status.digitalocean.com/incidents/jmj2wx0y78q3.

Expected Impact:

During the maintenance window, users may experience issues with management operations for services including but not limited to, Droplets, Managed Kubernetes, Load Balancers and Managed Databases. Connectivity to existing services should not be impacted. We will endeavor to keep this to a minimum for the duration of the maintenance.

If you have any questions related to this issue, please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Thank you,
Team DigitalOcean

Received β€” 31 August 2024 ⏭ DigitalOcean Status - Incident History

Managed Kubernetes Clusters in SFO3

Aug 30, 14:10 UTC
Resolved - Our Engineering team has resolved the issue with Managed Kubernetes clusters in the SFO3 region. All services should now be operating normally. If you continue to experience any problems, please open a ticket with our support team. We apologize for any inconvenience this may have caused.

Aug 30, 13:45 UTC
Monitoring - Our Engineering team has implemented a fix to resolve the issue affecting Managed Kubernetes clusters in the SFO3 region and is closely monitoring the situation. We will provide an update as soon as the issue is fully resolved.

Aug 30, 13:37 UTC
Investigating - Our Engineering team is currently investigating issues with the accessibility of Managed Kubernetes clusters in the SFO3 region. During this time, some users may experience difficulties connecting to their Managed Kubernetes clusters. We apologize for the inconvenience and will provide an update as soon as we have more information.

Networking in SFO3 Region

Aug 30, 12:19 UTC
Resolved - Our Engineering team identified and resolved an issue affecting the networking in our SFO3 region. There were multiple impacts between 08:48 and 08:52 UTC, 09:08 and 09:12 UTC, and 10:51 and 10:55 UTC. During these periods, users may have experienced timeouts with network connections to and from the SFO3 region.

Our Engineering team was able to take quick action to mitigate the impact and resolve the issue. All services are now functioning as expected. Thank you for your patience, and we apologize for any inconvenience. If you continue to experience any issues, please open a Support ticket for further analysis.

Received β€” 29 August 2024 ⏭ DigitalOcean Status - Incident History

Core Infrastructure Maintenance 2024-08-29 14:00 UTC

Aug 29, 21:35 UTC
Completed - This maintenance window is now over and users should not experience any further interruptions with management operations.

An additional maintenance window will be scheduled for Tuesday, September 3, to fully complete this maintenance work. Further details will be available on our status page soon.

Thank you for your patience. If you have any questions or concerns about this or the upcoming maintenance, please reach out to Support from within your account.

Aug 29, 14:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Aug 29, 04:34 UTC
Scheduled - Start: 2024-08-29 14:00 UTC
End: 2024-08-29 22:00 UTC

During the above window, our Engineering team will be performing an emergency maintenance activity on core infrastructure in our production environment. This maintenance is essential to resolve the root cause of an issue discovered in a previous incident, details of which can be found here: https://status.digitalocean.com/incidents/jmj2wx0y78q3.

Expected Impact:

During the maintenance window, users may experience issues with management operations for services including but not limited to, Droplets, Managed Kubernetes, Load Balancers and Managed Databases. Connectivity to existing services should not be impacted. We will endeavor to keep this to a minimum for the duration of the change.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Thank you,
Team DigitalOcean

Creates/Resizes in SGP & SYD Regions

Aug 28, 23:45 UTC
Resolved - Our Engineering team has identified the root cause of the issue with creating and resizing Droplets and Droplet-based resources in our SGP1 and SYD1 regions.

No further user impact has occurred since our last update.

In order to fully remediate this issue, our Engineering team has scheduled emergency maintenance, which will take place from 14:00 - 22:00 UTC on August 29th.

Please visit the below maintenance link to know more :
link:https://status.digitalocean.com/incidents/np0zw6m04jm1

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Aug 28, 16:05 UTC
Update - Our Engineering team has now put mitigations in place for failing create and resize operations in SYD1 and SGP1 and we are seeing creates and resizes of Droplets and Droplet-based products succeeding.

Our Engineering team continues to investigate the root cause of the issue and work on a full fix, and users may still experience errors until a full fix is in place. We will post a further update when more information is available.

Aug 28, 15:21 UTC
Update - We are continuing to investigate this issue.

Aug 28, 15:17 UTC
Update - We are continuing to investigate this issue.

Aug 28, 15:06 UTC
Investigating - Our Engineering team is currently investigating an issue impacting operations in our SGP1 and SYD1 regions.

During this time, users may experience errors when trying to create new Droplets and Droplet-based resources, or when resizing their existing resources via our Cloud Control Panel and API.

We apologize for any inconvenience, and we'll share more information as soon as it's available.

Received β€” 26 August 2024 ⏭ DigitalOcean Status - Incident History

Droplet connectivity and Event processing in NYC1

Aug 26, 10:40 UTC
Resolved - Our Engineering team has confirmed that the issue impacting our Droplet-based services in the NYC1 region has been completely mitigated. Users should no longer see issues with their Droplets and Droplet-related services.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Aug 26, 09:36 UTC
Monitoring - As of 09:30 UTC, our Engineering team has identified the cause of the issue impacting our Droplet-based services in the NYC1 region and applied a fix. The impact has started to mitigate and users should be able to connect to their Droplets and also start to see events getting processed successfully.

We're now monitoring the fix for stability and will post an update once we are confident it is successful.

Aug 26, 08:13 UTC
Investigating - As of 01:00 UTC, our Engineering team is currently investigating reports of an issue with our Droplet-based services in the NYC1 region.

At this time, users may experience issues reaching their Droplet via network requests and also see errors or stuck events when attempting Droplet actions such as power cycles, resizes, rebuilds, etc. Our team is working to mitigate the issue.

We realize how impacting this is and will provide an update as soon as possible. Thank you for your patience.

Received β€” 23 August 2024 ⏭ DigitalOcean Status - Incident History

Spaces Availability in NYC3

Aug 22, 22:43 UTC
Resolved - From 17:18 UTC until 17:58 UTC, users may have experienced issues when attempting to access Spaces in the NYC3 region.

Our Engineering team has confirmed full resolution of the issue, and users should now be able to access Spaces normally.

Thank you for your patience. If you continue to experience any problems, please open a ticket with our support team for further review.

Aug 22, 22:11 UTC
Monitoring - Our Engineering team has confirmed the mitigations put in place have resolved the impact to Spaces availability in NYC3. We are currently monitoring the situation and will post an update as soon as the issue is fully resolved.

Aug 22, 18:13 UTC
Update - Our Engineering team is continuing to investigate this issue, but has taken action to mitigate the impact. Spaces API availability has returned to pre-incident levels and users' requests should succeed at this time.

We will post an update once we have further information.

Aug 22, 17:55 UTC
Investigating - Our Engineering team is investigating an issue with Spaces availability in the NYC3 region. During this time users may encounter errors when accessing Spaces objects or managing existing buckets in the NYC3 region.

We apologize for the inconvenience and will share an update once we have more information.

Container Registry and App Platform in AMS3/FRA1

Aug 22, 15:04 UTC
Resolved - From 10:30 to 11:35 UTC, our Engineering team observed an issue with Container Registry and App Platform builds in the AMS3 and FRA1 regions.

During this time, users may have experienced delays while building their Apps and could have potentially experienced timeout errors in builds as a result. Additionally, a subset of customers may have experienced latency while interacting with the Container Registries.

Our Engineering team found that a backing component of the Container Registry was experiencing high memory usage. They were able to remediate that component at 11:35 UTC, which resolved the issue.

We apologize for the inconvenience. If you have any questions or continue to experience issues, please reach out via a Support ticket on your account.

Received β€” 15 August 2024 ⏭ DigitalOcean Status - Incident History

DigitalOcean Support Portal Maintenance

Aug 18, 05:00 UTC
Completed - The scheduled maintenance has been completed.

Aug 18, 02:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Aug 15, 02:08 UTC
Scheduled - During the above maintenance window, there will be maintenance performed on our ticketing system.

Expected Impact:

During the course of the maintenance, users will be unable to submit support tickets, update existing tickets, or receive replies to existing tickets. Users will also be unable to log into https://cloudsupport.digitalocean.com/s/ or open the Support Portal from within the Cloud Control Panel.

Any tickets submitted during the maintenance via alternate methods (such as replying to an email chain or via our webform) will be saved and entered into our ticketing system at the completion of the maintenance.

Our Support Team will also be impacted by this maintenance and will be unable to enter the ticketing system to receive any tickets or reply to existing tickets. As soon as the vendor maintenance has completed, our team will address all support queries as quickly as possible.

We appreciate your patience throughout this process.

Networking in SFO2 and SFO3 region

Aug 15, 11:48 UTC
Resolved - As of 10:45 UTC, our engineering team has resolved the issue with networking in SFO2 and SFO3 regions, and networking in the regions should now be operating normally. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Aug 15, 11:21 UTC
Monitoring - Our Engineering team has identified the cause of the issue with networking in SFO2 and SFO3 regions, and has implemented a fix. We are currently monitoring the situation and will post an update as soon as the issue is fully resolved.

Aug 15, 10:02 UTC
Investigating - As of 08:00 UTC, our Engineering team is currently investigating an issue impacting networking in SFO2 and SFO3 regions. Users may experience network connectivity loss to Droplets and Droplet-based services, like Managed Kubernetes and Database Clusters.

We apologize for the inconvenience and will share an update once we have more information.

❌