Sep 10, 05:11 UTC
Resolved - Our Engineering team identified a networking issue with our upstream provider in the SGP1 region. From 00:25 to 04:00 UTC, users may have experienced connectivity issues to DockerHub from Droplets, App Platform and Managed Kubernetes in the SGP1 region.

Our Engineering team has confirmed there is no longer any impact and connectivity should be restored for all customers.

We apologize for the inconvenience. If you continue to experience any issues, please open a support ticket from within your account.

Sep 9, 22:59 UTC
Resolved - From 17:04 UTC to 21:43 UTC, users may not have seen the option to resize an attached Block Storage Volume via the Cloud Control Panel, or may have encountered errors when attempting to resize attached Volumes via the API.

Our Engineering team has confirmed the resolution of the issue and Volume resizes should now be functioning normally in all regions.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Sep 9, 21:58 UTC
Monitoring - Our Engineering team have implemented a fix for the issue of resizing Droplet-attached Block Storage Volumes in all regions and are monitoring the situation closely. We will post another update once we're confident that the issue is fully resolved.

Sep 9, 21:15 UTC
Investigating - Our Engineering team is investigating an issue with resizing Droplet-attached Block Storage Volumes in all regions. Beginning at 17:04 UTC, users may not see the option to resize the Volume via the Cloud Control Panel, as well as see errors when attempting to resize attached Volumes via the API.

At this time, unattached Volumes are able to be resized correctly.

We will post an update once further information is available.

Sep 10, 17:30 UTC
Completed - Our Engineering team has run into some operations taking longer than expected within this maintenance. No user impact has occurred as a result of those operations.

We will be extending this maintenance window to 22:00 UTC via a new maintenance post.

Sep 9, 13:30 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Sep 7, 13:50 UTC
Scheduled - The Infrastructure Maintenance in NYC2 which was previously scheduled to start on 2024-09-03 at 13:30 UTC has been rescheduled to the following window:

Start: 2024-09-09 13:30 UTC
End: 2024-09-09 17:30 UTC

And

Start: 2024-09-10 13:30 UTC
End: 2024-09-10 17:30 UTC

During the above window, our Engineering team will be performing maintenance on principal infrastructure in the NYC2 region in order to improve the reliability of the services. The maintenance is divided into two phases and will be performed on consecutive days during the same time as mentioned above.

Expected Impact:

During the maintenance window users may experience issues interacting with their services including Droplets, Managed Kubernetes, Load Balancers, Container Registry, and App Platform. Users may also experience delays or failures with event processing on Droplets and Droplet-based services. We will endeavor to keep this to a minimum for the duration of the change.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Thank you,
Team DigitalOcean

Sep 9, 16:05 UTC
Completed - Hello,

The NYC2 Network Maintenance which was previously scheduled to start today (2024-09-09) at 21:00 UTC has been rescheduled to the following window:

Start: 2024-09-18 21:00 UTC
End: 2024-09-18 23:00 UTC

We apologize for any inconvenience this short notice may have caused and thank you for your understanding.
If you have questions or concerns about this maintenance, please reach out to us by opening up a ticket on your account.

Thank you,
DigitalOcean Networking Team

Sep 7, 11:42 UTC
Scheduled - Start: 2024-09-09 21:00 UTC
End: 2024-09-09 23:00 UTC

Hello,

During the above window, our Networking team will be performing maintenance on the management network firewalls in the NYC2 region.

Expected Impact:

During the maintenance period, users may experience delays of up to 10 minutes with Event Processing on Droplets and Droplet-based services in NYC2. We will endeavor to keep this to a minimum for the duration of the change.

If you have any questions or concerns regarding this maintenance, please reach out to us by opening up a ticket on your account via https://cloudsupport.digitalocean.com/s/createticket/.

Thank you,
DigitalOcean Networking Team

Sep 3, 18:00 UTC
Completed - The scheduled maintenance has been completed.

Sep 3, 14:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Aug 31, 14:21 UTC
Scheduled - Start: 2024-09-03 14:00 UTC
End: 2024-09-03 18:00 UTC

During the above window, our Engineering team will be performing additional maintenance on core infrastructure in our production environment. This is the second part of the maintenance completed on 29th August (posted here https://status.digitalocean.com/incidents/wb3g6x6q55ct) to resolve the root cause of an issue discovered in a previous incident, details of which can be found here:

https://status.digitalocean.com/incidents/jmj2wx0y78q3.

Expected Impact:

During the maintenance window, users may experience issues with management operations for services including but not limited to, Droplets, Managed Kubernetes, Load Balancers and Managed Databases. Connectivity to existing services should not be impacted. We will endeavor to keep this to a minimum for the duration of the maintenance.

If you have any questions related to this issue, please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Thank you,
Team DigitalOcean

Aug 30, 14:10 UTC
Resolved - Our Engineering team has resolved the issue with Managed Kubernetes clusters in the SFO3 region. All services should now be operating normally. If you continue to experience any problems, please open a ticket with our support team. We apologize for any inconvenience this may have caused.

Aug 30, 13:45 UTC
Monitoring - Our Engineering team has implemented a fix to resolve the issue affecting Managed Kubernetes clusters in the SFO3 region and is closely monitoring the situation. We will provide an update as soon as the issue is fully resolved.

Aug 30, 13:37 UTC
Investigating - Our Engineering team is currently investigating issues with the accessibility of Managed Kubernetes clusters in the SFO3 region. During this time, some users may experience difficulties connecting to their Managed Kubernetes clusters. We apologize for the inconvenience and will provide an update as soon as we have more information.

Aug 30, 12:19 UTC
Resolved - Our Engineering team identified and resolved an issue affecting the networking in our SFO3 region. There were multiple impacts between 08:48 and 08:52 UTC, 09:08 and 09:12 UTC, and 10:51 and 10:55 UTC. During these periods, users may have experienced timeouts with network connections to and from the SFO3 region.

Our Engineering team was able to take quick action to mitigate the impact and resolve the issue. All services are now functioning as expected. Thank you for your patience, and we apologize for any inconvenience. If you continue to experience any issues, please open a Support ticket for further analysis.

Aug 29, 21:35 UTC
Completed - This maintenance window is now over and users should not experience any further interruptions with management operations.

An additional maintenance window will be scheduled for Tuesday, September 3, to fully complete this maintenance work. Further details will be available on our status page soon.

Thank you for your patience. If you have any questions or concerns about this or the upcoming maintenance, please reach out to Support from within your account.

Aug 29, 14:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Aug 29, 04:34 UTC
Scheduled - Start: 2024-08-29 14:00 UTC
End: 2024-08-29 22:00 UTC

During the above window, our Engineering team will be performing an emergency maintenance activity on core infrastructure in our production environment. This maintenance is essential to resolve the root cause of an issue discovered in a previous incident, details of which can be found here: https://status.digitalocean.com/incidents/jmj2wx0y78q3.

Expected Impact:

During the maintenance window, users may experience issues with management operations for services including but not limited to, Droplets, Managed Kubernetes, Load Balancers and Managed Databases. Connectivity to existing services should not be impacted. We will endeavor to keep this to a minimum for the duration of the change.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Thank you,
Team DigitalOcean

Aug 28, 23:45 UTC
Resolved - Our Engineering team has identified the root cause of the issue with creating and resizing Droplets and Droplet-based resources in our SGP1 and SYD1 regions.

No further user impact has occurred since our last update.

In order to fully remediate this issue, our Engineering team has scheduled emergency maintenance, which will take place from 14:00 - 22:00 UTC on August 29th.

Please visit the below maintenance link to know more :
link:https://status.digitalocean.com/incidents/np0zw6m04jm1

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Aug 28, 16:05 UTC
Update - Our Engineering team has now put mitigations in place for failing create and resize operations in SYD1 and SGP1 and we are seeing creates and resizes of Droplets and Droplet-based products succeeding.

Our Engineering team continues to investigate the root cause of the issue and work on a full fix, and users may still experience errors until a full fix is in place. We will post a further update when more information is available.

Aug 28, 15:21 UTC
Update - We are continuing to investigate this issue.

Aug 28, 15:17 UTC
Update - We are continuing to investigate this issue.

Aug 28, 15:06 UTC
Investigating - Our Engineering team is currently investigating an issue impacting operations in our SGP1 and SYD1 regions.

During this time, users may experience errors when trying to create new Droplets and Droplet-based resources, or when resizing their existing resources via our Cloud Control Panel and API.

We apologize for any inconvenience, and we'll share more information as soon as it's available.

Aug 26, 10:40 UTC
Resolved - Our Engineering team has confirmed that the issue impacting our Droplet-based services in the NYC1 region has been completely mitigated. Users should no longer see issues with their Droplets and Droplet-related services.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Aug 26, 09:36 UTC
Monitoring - As of 09:30 UTC, our Engineering team has identified the cause of the issue impacting our Droplet-based services in the NYC1 region and applied a fix. The impact has started to mitigate and users should be able to connect to their Droplets and also start to see events getting processed successfully.

We're now monitoring the fix for stability and will post an update once we are confident it is successful.

Aug 26, 08:13 UTC
Investigating - As of 01:00 UTC, our Engineering team is currently investigating reports of an issue with our Droplet-based services in the NYC1 region.

At this time, users may experience issues reaching their Droplet via network requests and also see errors or stuck events when attempting Droplet actions such as power cycles, resizes, rebuilds, etc. Our team is working to mitigate the issue.

We realize how impacting this is and will provide an update as soon as possible. Thank you for your patience.

Aug 22, 22:43 UTC
Resolved - From 17:18 UTC until 17:58 UTC, users may have experienced issues when attempting to access Spaces in the NYC3 region.

Our Engineering team has confirmed full resolution of the issue, and users should now be able to access Spaces normally.

Thank you for your patience. If you continue to experience any problems, please open a ticket with our support team for further review.

Aug 22, 22:11 UTC
Monitoring - Our Engineering team has confirmed the mitigations put in place have resolved the impact to Spaces availability in NYC3. We are currently monitoring the situation and will post an update as soon as the issue is fully resolved.

Aug 22, 18:13 UTC
Update - Our Engineering team is continuing to investigate this issue, but has taken action to mitigate the impact. Spaces API availability has returned to pre-incident levels and users' requests should succeed at this time.

We will post an update once we have further information.

Aug 22, 17:55 UTC
Investigating - Our Engineering team is investigating an issue with Spaces availability in the NYC3 region. During this time users may encounter errors when accessing Spaces objects or managing existing buckets in the NYC3 region.

We apologize for the inconvenience and will share an update once we have more information.

Aug 22, 15:04 UTC
Resolved - From 10:30 to 11:35 UTC, our Engineering team observed an issue with Container Registry and App Platform builds in the AMS3 and FRA1 regions.

During this time, users may have experienced delays while building their Apps and could have potentially experienced timeout errors in builds as a result. Additionally, a subset of customers may have experienced latency while interacting with the Container Registries.

Our Engineering team found that a backing component of the Container Registry was experiencing high memory usage. They were able to remediate that component at 11:35 UTC, which resolved the issue.

We apologize for the inconvenience. If you have any questions or continue to experience issues, please reach out via a Support ticket on your account.

Aug 18, 05:00 UTC
Completed - The scheduled maintenance has been completed.

Aug 18, 02:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Aug 15, 02:08 UTC
Scheduled - During the above maintenance window, there will be maintenance performed on our ticketing system.

Expected Impact:

During the course of the maintenance, users will be unable to submit support tickets, update existing tickets, or receive replies to existing tickets. Users will also be unable to log into https://cloudsupport.digitalocean.com/s/ or open the Support Portal from within the Cloud Control Panel.

Any tickets submitted during the maintenance via alternate methods (such as replying to an email chain or via our webform) will be saved and entered into our ticketing system at the completion of the maintenance.

Our Support Team will also be impacted by this maintenance and will be unable to enter the ticketing system to receive any tickets or reply to existing tickets. As soon as the vendor maintenance has completed, our team will address all support queries as quickly as possible.

We appreciate your patience throughout this process.

Aug 15, 11:48 UTC
Resolved - As of 10:45 UTC, our engineering team has resolved the issue with networking in SFO2 and SFO3 regions, and networking in the regions should now be operating normally. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Aug 15, 11:21 UTC
Monitoring - Our Engineering team has identified the cause of the issue with networking in SFO2 and SFO3 regions, and has implemented a fix. We are currently monitoring the situation and will post an update as soon as the issue is fully resolved.

Aug 15, 10:02 UTC
Investigating - As of 08:00 UTC, our Engineering team is currently investigating an issue impacting networking in SFO2 and SFO3 regions. Users may experience network connectivity loss to Droplets and Droplet-based services, like Managed Kubernetes and Database Clusters.

We apologize for the inconvenience and will share an update once we have more information.

Aug 15, 05:40 UTC
Resolved - Our Engineering team has resolved the issue with Droplet creates and Snapshots. As of 05:30 UTC, users should be able to create Droplets, Snapshots and process events. Droplet backed services should also be operating normally.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Aug 15, 05:11 UTC
Monitoring - Our Engineering team has identified the cause of the issue with Droplet creates and Snapshots, and has implemented a fix. We are currently monitoring the situation and will post an update as soon as the issue is fully resolved.

Aug 15, 04:47 UTC
Update - Our Engineering team is continuing to investigate the issue with Droplet creates and Snapshots. A subset of users may also encounter errors with creating Droplet based products such as Managed Databases, Load Balancers, Managed Kubernetes in TOR1 region

We will post another update once we have more information.

Aug 15, 04:29 UTC
Investigating - As of 02:11 UTC, our Engineering team is investigating an issue with creation failure of Droplet and Snapshots in TOR1 region. Users may experience errors when rebuilding or restoring Droplets and Snapshots. Creating Droplets from Snapshots is also affected.

We apologize for the inconvenience and will share an update once we have more information.

Aug 16, 22:00 UTC
Completed - The scheduled maintenance has been completed.

Aug 14, 02:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Aug 13, 18:31 UTC
Scheduled - Maintenance Dates: August 14th,15th and 16th
Maintenance Window: 02:00 - 10:00 (UTC)

During the maintenance window on the above dates, our on-site team will be performing ongoing maintenance on redundant power systems in our TOR1 region. The maintenance will be performed during the same time window (02:00 - 10:00 UTC) across 3 days: August 14th,15th and 16th.

Expected Impact:

We don’t expect to see any impact to customer resources as the electrical systems are redundant and our team will isolate one unit at a time during the maintenance. In the highly unlikely event that we lose power to both systems simultaneously, affected Droplets and Droplet-based services would experience downtime for a brief period of time. We will endeavor to keep this to a minimum should that situation arise.

If you have any questions or concerns related to this maintenance, please send us a ticket from your cloud support page https://cloudsupport.digitalocean.com/s/createticket.

Aug 13, 03:10 UTC
Resolved - From 23:08 August 12 to 01:26 August 13 UTC, customers may have experienced failures with Droplet creation, power on events, and restore events in the NYC3 region.

Our Engineering team has confirmed resolution of this issue. Thank you for your patience.

If you continue to experience any problems, please open a support ticket from within your account.

Aug 13, 02:25 UTC
Monitoring - Our Engineering team has implemented mitigations to stop the Droplet creation failures in the NYC3 region. The team is monitoring the situation to ensure that everything is fully resolved. We apologize for the inconvenience and will share an update if there are any new developments.

Aug 13, 01:49 UTC
Update - Multiple Engineering teams are continuing to investigate errors with creating Droplets, other Droplet-based products, and events such as power/restore in the NYC3 region.

Users should no longer encounter errors when performing these operations, as mitigations are now in place, and teams have observed error rates returning to normal.

We apologize for the inconvenience and will share another update once we have more information.

Aug 13, 01:01 UTC
Update - Our Engineering team is continuing to investigate issues with creating Droplets and other Droplet-based products in the NYC3 region. At this time, customers may also encounter errors when powering on some existing Droplets, creating new Droplets, or restoring a backup or snapshot.

We apologize for the inconvenience and will share an update once we have more information.

Aug 13, 00:27 UTC
Investigating - Our Engineering team is investigating an issue with Droplet creation in NYC3. During this time, users will see an elevated failure rate for the creation of Droplets and other Droplet-based products, such as Load Balancers, Managed Databases, or Kubernetes, in NYC3.

Aug 6, 20:40 UTC
Resolved - From 09:52 UTC to 19:32 UTC, customers may have experienced failures with Droplet rebuild and restore events in all regions. Our Engineering team has confirmed full resolution of this issue.

Thank you for your patience. If you continue to experience any problems, please open a support ticket from within your account.

Aug 6, 19:57 UTC
Monitoring - Our Engineering team identified the root cause and performed a service rollback to resolve the issue that was causing failure in Droplet rebuild and restore events in all regions. We are actively monitoring the situation to ensure stability and will provide an update once the incident has been fully resolved.

Thank you for your patience and we apologize for the inconvenience.

Aug 6, 19:19 UTC
Investigating - Our Engineering team is investigating an issue with processing of rebuild and restore events for Droplets, in all regions. During this time, users may experience long delays or errors when restoring or rebuilding a Droplet.

We apologize for the inconvenience and will share an update once we have more information.

Aug 8, 14:00 UTC
Completed - The scheduled maintenance has been completed.

Aug 8, 12:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Aug 5, 15:48 UTC
Scheduled - Hello,

We are reaching out again to inform you that the SYD1 Network Maintenance which was previously scheduled to start today (2024-08-05) at 12:00 UTC has been rescheduled to the following window:

Start: 2024-08-08 12:00 UTC
End: 2024-08-08 14:00 UTC

We apologize for any inconvenience this short notice may have caused and thank you for your understanding. You may find the initial maintenance notice along with a description of any expected impact related to this work included at the bottom of this message.

If you have questions or concerns about this maintenance, please reach out to us by opening up a ticket on your account.

Thank you,


---BEGIN INITIAL MAINTENANCE NOTICE---


SYD1 Network Maintenance 2024-08-05 12:00 UTC

Start: 2024-08-05 12:00 UTC
End: 2024-08-05 14:00 UTC

Hello,

During the above window, we will be performing maintenance in our SYD1 region as part of network upgrades.

Expected Impact:

These upgrades are designed and tested to be seamless and we do not expect any visible impact to customer traffic due to this maintenance. Should an unexpected issue arise, a possible outcome would be a temporary loss of public traffic inbound and outbound connectivity for a short time for the impacted Droplets and Droplet-based services in the region. We will endeavor to keep any such impact to a minimum.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Thank you,
Team DigitalOcean

Aug 5, 19:46 UTC
Resolved - Our engineering team has resolved the issue impacting the management of multiple services via the Cloud Control Panel and the API, including event processing. All services should now be functioning normally. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Aug 5, 19:20 UTC
Monitoring - Our Engineering teams have identified a remediation strategy and are working to roll it out to all regions. We are starting to see services recover as a result of this fix being deployed.

Customers may begin to see some management functionality for services return and we will continue to provide updates as we work to return full functionality.

Aug 5, 17:39 UTC
Identified - Our Engineering teams have now identified the cause of the issue impacting multiple services. They continue to work on resolving the impact to customers.

During this time, users may experience errors when attempting to manage services through the UI or the API, including accessing Droplet information or the processing of Droplet events. Existing customer resources, such as Droplets, remain online and accessible.

We apologize for the inconvenience and will share an update once we have more information.

Aug 5, 17:11 UTC
Update - Multiple Engineering teams are continuing to investigate the issue. Customers may also encounter errors when managing other services and using the API at this time.

We will post another update soon.

Aug 5, 17:05 UTC
Investigating - As of 16:30 UTC, Our Engineering team is investigating an issue with processing of events in all regions. During this time users may experience failed events on Droplets. We apologize for the inconvenience and will share an update once we have more information.