Aug 29, 21:35 UTC
Completed - This maintenance window is now over and users should not experience any further interruptions with management operations.

An additional maintenance window will be scheduled for Tuesday, September 3, to fully complete this maintenance work. Further details will be available on our status page soon.

Thank you for your patience. If you have any questions or concerns about this or the upcoming maintenance, please reach out to Support from within your account.

Aug 29, 14:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Aug 29, 04:34 UTC
Scheduled - Start: 2024-08-29 14:00 UTC
End: 2024-08-29 22:00 UTC

During the above window, our Engineering team will be performing an emergency maintenance activity on core infrastructure in our production environment. This maintenance is essential to resolve the root cause of an issue discovered in a previous incident, details of which can be found here: https://status.digitalocean.com/incidents/jmj2wx0y78q3.

Expected Impact:

During the maintenance window, users may experience issues with management operations for services including but not limited to, Droplets, Managed Kubernetes, Load Balancers and Managed Databases. Connectivity to existing services should not be impacted. We will endeavor to keep this to a minimum for the duration of the change.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Thank you,
Team DigitalOcean

Aug 28, 23:45 UTC
Resolved - Our Engineering team has identified the root cause of the issue with creating and resizing Droplets and Droplet-based resources in our SGP1 and SYD1 regions.

No further user impact has occurred since our last update.

In order to fully remediate this issue, our Engineering team has scheduled emergency maintenance, which will take place from 14:00 - 22:00 UTC on August 29th.

Please visit the below maintenance link to know more :
link:https://status.digitalocean.com/incidents/np0zw6m04jm1

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Aug 28, 16:05 UTC
Update - Our Engineering team has now put mitigations in place for failing create and resize operations in SYD1 and SGP1 and we are seeing creates and resizes of Droplets and Droplet-based products succeeding.

Our Engineering team continues to investigate the root cause of the issue and work on a full fix, and users may still experience errors until a full fix is in place. We will post a further update when more information is available.

Aug 28, 15:21 UTC
Update - We are continuing to investigate this issue.

Aug 28, 15:17 UTC
Update - We are continuing to investigate this issue.

Aug 28, 15:06 UTC
Investigating - Our Engineering team is currently investigating an issue impacting operations in our SGP1 and SYD1 regions.

During this time, users may experience errors when trying to create new Droplets and Droplet-based resources, or when resizing their existing resources via our Cloud Control Panel and API.

We apologize for any inconvenience, and we'll share more information as soon as it's available.

Aug 26, 10:40 UTC
Resolved - Our Engineering team has confirmed that the issue impacting our Droplet-based services in the NYC1 region has been completely mitigated. Users should no longer see issues with their Droplets and Droplet-related services.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Aug 26, 09:36 UTC
Monitoring - As of 09:30 UTC, our Engineering team has identified the cause of the issue impacting our Droplet-based services in the NYC1 region and applied a fix. The impact has started to mitigate and users should be able to connect to their Droplets and also start to see events getting processed successfully.

We're now monitoring the fix for stability and will post an update once we are confident it is successful.

Aug 26, 08:13 UTC
Investigating - As of 01:00 UTC, our Engineering team is currently investigating reports of an issue with our Droplet-based services in the NYC1 region.

At this time, users may experience issues reaching their Droplet via network requests and also see errors or stuck events when attempting Droplet actions such as power cycles, resizes, rebuilds, etc. Our team is working to mitigate the issue.

We realize how impacting this is and will provide an update as soon as possible. Thank you for your patience.

Aug 22, 22:43 UTC
Resolved - From 17:18 UTC until 17:58 UTC, users may have experienced issues when attempting to access Spaces in the NYC3 region.

Our Engineering team has confirmed full resolution of the issue, and users should now be able to access Spaces normally.

Thank you for your patience. If you continue to experience any problems, please open a ticket with our support team for further review.

Aug 22, 22:11 UTC
Monitoring - Our Engineering team has confirmed the mitigations put in place have resolved the impact to Spaces availability in NYC3. We are currently monitoring the situation and will post an update as soon as the issue is fully resolved.

Aug 22, 18:13 UTC
Update - Our Engineering team is continuing to investigate this issue, but has taken action to mitigate the impact. Spaces API availability has returned to pre-incident levels and users' requests should succeed at this time.

We will post an update once we have further information.

Aug 22, 17:55 UTC
Investigating - Our Engineering team is investigating an issue with Spaces availability in the NYC3 region. During this time users may encounter errors when accessing Spaces objects or managing existing buckets in the NYC3 region.

We apologize for the inconvenience and will share an update once we have more information.

Aug 22, 15:04 UTC
Resolved - From 10:30 to 11:35 UTC, our Engineering team observed an issue with Container Registry and App Platform builds in the AMS3 and FRA1 regions.

During this time, users may have experienced delays while building their Apps and could have potentially experienced timeout errors in builds as a result. Additionally, a subset of customers may have experienced latency while interacting with the Container Registries.

Our Engineering team found that a backing component of the Container Registry was experiencing high memory usage. They were able to remediate that component at 11:35 UTC, which resolved the issue.

We apologize for the inconvenience. If you have any questions or continue to experience issues, please reach out via a Support ticket on your account.

Aug 18, 05:00 UTC
Completed - The scheduled maintenance has been completed.

Aug 18, 02:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Aug 15, 02:08 UTC
Scheduled - During the above maintenance window, there will be maintenance performed on our ticketing system.

Expected Impact:

During the course of the maintenance, users will be unable to submit support tickets, update existing tickets, or receive replies to existing tickets. Users will also be unable to log into https://cloudsupport.digitalocean.com/s/ or open the Support Portal from within the Cloud Control Panel.

Any tickets submitted during the maintenance via alternate methods (such as replying to an email chain or via our webform) will be saved and entered into our ticketing system at the completion of the maintenance.

Our Support Team will also be impacted by this maintenance and will be unable to enter the ticketing system to receive any tickets or reply to existing tickets. As soon as the vendor maintenance has completed, our team will address all support queries as quickly as possible.

We appreciate your patience throughout this process.

Aug 15, 11:48 UTC
Resolved - As of 10:45 UTC, our engineering team has resolved the issue with networking in SFO2 and SFO3 regions, and networking in the regions should now be operating normally. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Aug 15, 11:21 UTC
Monitoring - Our Engineering team has identified the cause of the issue with networking in SFO2 and SFO3 regions, and has implemented a fix. We are currently monitoring the situation and will post an update as soon as the issue is fully resolved.

Aug 15, 10:02 UTC
Investigating - As of 08:00 UTC, our Engineering team is currently investigating an issue impacting networking in SFO2 and SFO3 regions. Users may experience network connectivity loss to Droplets and Droplet-based services, like Managed Kubernetes and Database Clusters.

We apologize for the inconvenience and will share an update once we have more information.

Aug 15, 05:40 UTC
Resolved - Our Engineering team has resolved the issue with Droplet creates and Snapshots. As of 05:30 UTC, users should be able to create Droplets, Snapshots and process events. Droplet backed services should also be operating normally.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Aug 15, 05:11 UTC
Monitoring - Our Engineering team has identified the cause of the issue with Droplet creates and Snapshots, and has implemented a fix. We are currently monitoring the situation and will post an update as soon as the issue is fully resolved.

Aug 15, 04:47 UTC
Update - Our Engineering team is continuing to investigate the issue with Droplet creates and Snapshots. A subset of users may also encounter errors with creating Droplet based products such as Managed Databases, Load Balancers, Managed Kubernetes in TOR1 region

We will post another update once we have more information.

Aug 15, 04:29 UTC
Investigating - As of 02:11 UTC, our Engineering team is investigating an issue with creation failure of Droplet and Snapshots in TOR1 region. Users may experience errors when rebuilding or restoring Droplets and Snapshots. Creating Droplets from Snapshots is also affected.

We apologize for the inconvenience and will share an update once we have more information.

Aug 16, 22:00 UTC
Completed - The scheduled maintenance has been completed.

Aug 14, 02:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Aug 13, 18:31 UTC
Scheduled - Maintenance Dates: August 14th,15th and 16th
Maintenance Window: 02:00 - 10:00 (UTC)

During the maintenance window on the above dates, our on-site team will be performing ongoing maintenance on redundant power systems in our TOR1 region. The maintenance will be performed during the same time window (02:00 - 10:00 UTC) across 3 days: August 14th,15th and 16th.

Expected Impact:

We don’t expect to see any impact to customer resources as the electrical systems are redundant and our team will isolate one unit at a time during the maintenance. In the highly unlikely event that we lose power to both systems simultaneously, affected Droplets and Droplet-based services would experience downtime for a brief period of time. We will endeavor to keep this to a minimum should that situation arise.

If you have any questions or concerns related to this maintenance, please send us a ticket from your cloud support page https://cloudsupport.digitalocean.com/s/createticket.

Aug 13, 03:10 UTC
Resolved - From 23:08 August 12 to 01:26 August 13 UTC, customers may have experienced failures with Droplet creation, power on events, and restore events in the NYC3 region.

Our Engineering team has confirmed resolution of this issue. Thank you for your patience.

If you continue to experience any problems, please open a support ticket from within your account.

Aug 13, 02:25 UTC
Monitoring - Our Engineering team has implemented mitigations to stop the Droplet creation failures in the NYC3 region. The team is monitoring the situation to ensure that everything is fully resolved. We apologize for the inconvenience and will share an update if there are any new developments.

Aug 13, 01:49 UTC
Update - Multiple Engineering teams are continuing to investigate errors with creating Droplets, other Droplet-based products, and events such as power/restore in the NYC3 region.

Users should no longer encounter errors when performing these operations, as mitigations are now in place, and teams have observed error rates returning to normal.

We apologize for the inconvenience and will share another update once we have more information.

Aug 13, 01:01 UTC
Update - Our Engineering team is continuing to investigate issues with creating Droplets and other Droplet-based products in the NYC3 region. At this time, customers may also encounter errors when powering on some existing Droplets, creating new Droplets, or restoring a backup or snapshot.

We apologize for the inconvenience and will share an update once we have more information.

Aug 13, 00:27 UTC
Investigating - Our Engineering team is investigating an issue with Droplet creation in NYC3. During this time, users will see an elevated failure rate for the creation of Droplets and other Droplet-based products, such as Load Balancers, Managed Databases, or Kubernetes, in NYC3.

Aug 6, 20:40 UTC
Resolved - From 09:52 UTC to 19:32 UTC, customers may have experienced failures with Droplet rebuild and restore events in all regions. Our Engineering team has confirmed full resolution of this issue.

Thank you for your patience. If you continue to experience any problems, please open a support ticket from within your account.

Aug 6, 19:57 UTC
Monitoring - Our Engineering team identified the root cause and performed a service rollback to resolve the issue that was causing failure in Droplet rebuild and restore events in all regions. We are actively monitoring the situation to ensure stability and will provide an update once the incident has been fully resolved.

Thank you for your patience and we apologize for the inconvenience.

Aug 6, 19:19 UTC
Investigating - Our Engineering team is investigating an issue with processing of rebuild and restore events for Droplets, in all regions. During this time, users may experience long delays or errors when restoring or rebuilding a Droplet.

We apologize for the inconvenience and will share an update once we have more information.

Aug 8, 14:00 UTC
Completed - The scheduled maintenance has been completed.

Aug 8, 12:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Aug 5, 15:48 UTC
Scheduled - Hello,

We are reaching out again to inform you that the SYD1 Network Maintenance which was previously scheduled to start today (2024-08-05) at 12:00 UTC has been rescheduled to the following window:

Start: 2024-08-08 12:00 UTC
End: 2024-08-08 14:00 UTC

We apologize for any inconvenience this short notice may have caused and thank you for your understanding. You may find the initial maintenance notice along with a description of any expected impact related to this work included at the bottom of this message.

If you have questions or concerns about this maintenance, please reach out to us by opening up a ticket on your account.

Thank you,


---BEGIN INITIAL MAINTENANCE NOTICE---


SYD1 Network Maintenance 2024-08-05 12:00 UTC

Start: 2024-08-05 12:00 UTC
End: 2024-08-05 14:00 UTC

Hello,

During the above window, we will be performing maintenance in our SYD1 region as part of network upgrades.

Expected Impact:

These upgrades are designed and tested to be seamless and we do not expect any visible impact to customer traffic due to this maintenance. Should an unexpected issue arise, a possible outcome would be a temporary loss of public traffic inbound and outbound connectivity for a short time for the impacted Droplets and Droplet-based services in the region. We will endeavor to keep any such impact to a minimum.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Thank you,
Team DigitalOcean

Aug 5, 19:46 UTC
Resolved - Our engineering team has resolved the issue impacting the management of multiple services via the Cloud Control Panel and the API, including event processing. All services should now be functioning normally. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Aug 5, 19:20 UTC
Monitoring - Our Engineering teams have identified a remediation strategy and are working to roll it out to all regions. We are starting to see services recover as a result of this fix being deployed.

Customers may begin to see some management functionality for services return and we will continue to provide updates as we work to return full functionality.

Aug 5, 17:39 UTC
Identified - Our Engineering teams have now identified the cause of the issue impacting multiple services. They continue to work on resolving the impact to customers.

During this time, users may experience errors when attempting to manage services through the UI or the API, including accessing Droplet information or the processing of Droplet events. Existing customer resources, such as Droplets, remain online and accessible.

We apologize for the inconvenience and will share an update once we have more information.

Aug 5, 17:11 UTC
Update - Multiple Engineering teams are continuing to investigate the issue. Customers may also encounter errors when managing other services and using the API at this time.

We will post another update soon.

Aug 5, 17:05 UTC
Investigating - As of 16:30 UTC, Our Engineering team is investigating an issue with processing of events in all regions. During this time users may experience failed events on Droplets. We apologize for the inconvenience and will share an update once we have more information.

Aug 5, 05:48 UTC
Resolved - Our Engineering team has confirmed the full resolution of the issue with the DigitalOcean App Platform and Container Registry in our NYC regions.

Users should no longer experience any issues while pushing to Container Registries and working with App Platform builds.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Aug 5, 05:21 UTC
Monitoring - As of 03:08 UTC, our Engineering team has deployed a fix to resolve the issue with the DigitalOcean App Platform and Container Registry in our NYC regions.

At this time, users should no longer experience issues while pushing to Container Registries and working with App Platform builds.

We are currently monitoring the situation closely and will post an update as soon as the issue is fully resolved. Thank you for your patience and we apologize for any inconvenience.

Aug 5, 03:21 UTC
Investigating - Our Engineering team is investigating an issue with the DigitalOcean App Platform and Container Registry in our NYC regions. At this time, users may experience delays or timeouts while pushing to Container Registries, as well as delays when working with App Platform builds.

We apologize for the inconvenience and will share an update once we have more information.

Aug 5, 10:57 UTC
Completed - SYD1 Network Maintenance previously scheduled today (2024-08-05) at 12:00 UTC has been rescheduled for 2024-08-08 at 12:00 UTC. We have already notified you via email and will add a new maintenance post on status.digitalocean.com with the updated date and timings for this maintenance.

We apologize for any inconvenience and thank you for your understanding. If you have questions or concerns about this maintenance, please reach out to us by opening up a ticket on your account.

Aug 2, 12:08 UTC
Scheduled - Start: 2024-08-05 12:00 UTC
End: 2024-08-05 14:00 UTC

During the above window, we will be performing maintenance in our SYD1 region as part of network upgrades.

Expected Impact:

These upgrades are designed and tested to be seamless and we do not expect any visible impact to customer traffic due to this maintenance. Should an unexpected issue arise, a possible outcome would be a temporary loss of public traffic inbound and outbound connectivity for a short time for the impacted Droplets and Droplet-based services in the region. We will endeavor to keep any such impact to a minimum.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Jul 31, 01:29 UTC
Resolved - From 23:47 UTC until 01:11 UTC, users may have experienced errors when attempting to create Spaces Access Keys in the Cloud Control Panel.

Our Engineering team has identified and resolved the issue. The impact has been resolved and users should now be able to create Spaces Access Keys.

We apologize for any inconvenience this may have caused. If you have any questions or continue to experience issues, please reach out via a Support ticket on your account.

Jul 31, 18:32 UTC
Completed - The scheduled maintenance has been completed.

Jul 31, 16:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jul 29, 16:55 UTC
Scheduled - Start: 2024-07-31 16:00 UTC
End: 2024-07-31 20:00 UTC

Hello,

During the above window, our Networking team will be making changes to our core networking infrastructure to improve performance and scalability in the AMS3 region.

Expected Impact:

These upgrades are designed and tested to be seamless and we do not expect any impact on customer traffic due to this maintenance. Should an unexpected issue arise, a possible outcome would be a temporary loss of connectivity for impacted Droplets. Any impact would also extend to Droplet-dependent services in the AMS3 region such as Managed Databases, Load Balancers, App Platform, and Managed Kubernetes. We will endeavor to keep any such impact to a minimum.

If you have any questions or concerns regarding this maintenance, please reach out to us by opening up a ticket on your account via https://cloudsupport.digitalocean.com/s/createticket/.

Thank you,
DigitalOcean Networking Team

Jul 29, 05:52 UTC
Resolved - As of 05:05 UTC, our Engineering team has confirmed the full resolution of the issue impacting Snapshot and Backup Images in the TOR1 region. We have verified that the Snapshot and Backup events in the region are processing without any failures.

Users should also be able to create Droplets from Snapshot and Backup images in this region without any issues.

Thank you for your patience and understanding. If you should encounter any further issues at all, then please open a ticket with our Support team.

Jul 29, 05:33 UTC
Monitoring - As of 04:30 UTC, our Engineering team was able to take action to mitigate the impact of this issue. Snapshot and Backup events will be processed normally in the TOR1 region. Users may no longer experience errors when creating or restoring Snapshots and Backups images.

Creating Droplets from snapshots and backup images in this region should also be processed without any issues.

We apologize for any inconvenience caused and will post an update once the issue is fully resolved.

Jul 29, 04:27 UTC
Investigating - As of 03:14 UTC, our Engineering team is investigating an issue with Snapshot and Backup failures in our TOR1 region. Users may experience errors when creating or restoring Snapshots and Backups images.

Creating droplets from Snapshots and Backup images in this region is also affected.

We apologize for the inconvenience and will share an update once we have more information.

Jul 29, 12:00 UTC
Completed - The scheduled maintenance has been completed.

Jul 29, 10:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jul 27, 10:09 UTC
Scheduled - Start: 2024-07-29 10:00 UTC
End: 2024-07-29 12:00 UTC

During the above window, our networking team will be performing maintenance on the management network firewall in the NYC3 region.

Expected Impact:

During the maintenance period, no customer impact is expected. However, there is a very low probability that there could be a delay of up to 15 minutes with Event Processing on Droplets and related products.

If you have any questions or concerns regarding this maintenance, please reach out to us by opening up a ticket on your account via https://cloudsupport.digitalocean.com/s/createticket/.

Jul 25, 05:48 UTC
Resolved - As of 04:49 UTC our Engineering team has confirmed full resolution of the issue impacting network connectivity in multiple regions. Users should now be able to process events normally for Droplets and Droplet-based services like Load Balancers, Kubernetes or Database clusters, etc.

If you continue to experience problems, please open a ticket with our support team from your Cloud Control Panel.

Thank you for your patience and we apologize for any inconvenience.

Jul 25, 04:55 UTC
Monitoring - Our Engineering teams have applied a fix to all impacted hypervisors. Customers should no longer see networking issues with their Droplets and other Droplet-based resources.

We continue to monitor the situation closely and will post another update once we're confident that the issue is fully resolved.

Jul 25, 02:57 UTC
Identified - Our Engineering teams have identified the cause of the issue with network connectivity in multiple regions and are actively working on a fix.

We apologize for any inconvenience, and we will share more information as soon as it's available.

Jul 25, 01:47 UTC
Update - Multiple Engineering teams are continuing to investigate the cause of the issue impacting networking in multiple regions. At this time, we have identified that the issue impacts FRA1, AMS3, LON1, SGP1, and NYC1. We are actively remediating hypervisors in order to restore connectivity to customer resources, while working on determining root cause and a long-term remediation.

Users who have resources on affected hypervisors (Droplets and Droplet-based services) may continue to experience loss of network connectivity, loss of monitoring graphs, errors when processing events like power off/on, and/or inability to access the recovery console for Droplets.

We will post another update soon.

Jul 25, 00:33 UTC
Investigating - Our Engineering team is currently investigating an issue impacting networking in multiple regions, especially FRA1 and AMS3. Users may experience network connectivity loss to Droplets and Droplet-based services, like Managed Kubernetes and Database Clusters, as well as encounter errors when trying to process events like power off/on, etc.

We apologize for the inconvenience and will share an update once we have more information.