❌

Normal view

Received β€” 11 November 2024 ⏭ DigitalOcean Status - Incident History

SFO3 Network Maintenance

13 November 2024 at 06:10

Nov 13, 11:10 UTC
Completed - The SFO3 Network Maintenance which was previously scheduled at 2024-11-13 10:00 UTC has been cancelled. We will notify you via email and a new maintenance post on status.digitalocean.com with the updated date and timings for this maintenance.

We apologize for any inconvenience and thank you for your understanding.

If you have questions or concerns about this maintenance, please reach out to us by opening up a ticket on your account.

Nov 13, 10:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Nov 11, 10:20 UTC
Scheduled - Start: 2024-11-13 10:00 UTC
End: 2024-11-13 14:00 UTC

During the above window, our Networking team will be performing maintenance on core switches in our SFO3 datacenter as part of network upgrades.

Expected Impact:

These upgrades are designed and tested to be seamless, and we do not expect any impact to customer traffic due to this maintenance. Should an unexpected issue arise, a possible outcome would be temporary loss of connectivity or increased latency for a brief period of time. We will endeavor to keep any such impact to a minimum.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Received β€” 6 November 2024 ⏭ DigitalOcean Status - Incident History

AMS3 Power Maintenance

6 November 2024 at 14:49

Nov 6, 19:49 UTC
Completed - Hello,

The AMS3 Power Maintenance which was previously scheduled at 2024-11-07 20:00 UTC has been cancelled. We will notify you via email and a new maintenance post on status.digitalocean.com with the updated date and timings for this maintenance.

We apologize for any inconvenience and thank you for your understanding. You may find the initial maintenance notice along with a description of any expected impact related to this work included at the bottom of this message.

If you have questions or concerns about this maintenance, please reach out to us by opening up a ticket on your account.

Thank you,

---BEGIN INITIAL MAINTENANCE NOTICE---

Start: 2024-11-07 20:00 UTC
End: 2024-11-08 06:00 UTC


Hello,

During the above window, our Engineering team will be performing maintenance on redundant upstream electrical systems in order to improve reliability in our AMS3 region.

Expected Impact:

We don’t expect to see any downtime as the power to hypervisors is redundant. In a highly unlikely situation, where we lose power to both the supplies, the Droplet(s) listed below may experience downtime for a brief period. We will endeavor to keep this to a minimum for the duration of the change.


If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket


Thank you,
Team DigitalOcean

Nov 5, 12:53 UTC
Scheduled - Start: 2024-11-07 20:00 UTC
End: 2024-11-08 06:00 UTC

Hello,

During the above window, our Engineering team will be performing maintenance on redundant upstream electrical systems in order to improve reliability in our AMS3 region.

Expected Impact:

We don’t expect to see any downtime as the power to hypervisors is redundant. In a highly unlikely situation, where we lose power to both the supplies, the Droplet(s) listed below may experience downtime for a brief period. We will endeavor to keep this to a minimum for the duration of the change.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Thank you,
Team DigitalOcean

Received β€” 5 November 2024 ⏭ DigitalOcean Status - Incident History

Core Infrastructure Maintenance

6 November 2024 at 13:10

Nov 6, 18:10 UTC
Completed - The scheduled maintenance has been completed.

Nov 6, 09:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Nov 4, 22:20 UTC
Scheduled - Start: 2024-11-06 09:00 UTC
End: 2024-11-07 22:00 UTC


During the above window, our Engineering team will be performing maintenance on principal infrastructure in order to improve reliability of DigitalOcean services.

Expected Impact:

During the maintenance window, users may experience increased latency for platform management operations, including:

Cloud Control Panel and API operations
Event processing
Droplet creates, resizes, rebuilds, and power events
Managed Kubernetes reconciliation and scaling
Load Balancer operations
Container Registry operations
App Platform operations
Managed Database creation and scaling

We do not expect any impact to customer traffic due to this maintenance. In the event that an unexpected issue should arise, users may experience errors with creating new resources such as Droplets, Load Balancers, Managed Kubernetes clusters, and Managed Databases. We will endeavor to keep any such impact to a minimum.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Received β€” 1 November 2024 ⏭ DigitalOcean Status - Incident History

Event Processing in NYC2 Region

1 November 2024 at 12:37

Nov 1, 17:37 UTC
Resolved - Our Engineering team has confirmed resolution of the issue impacting Droplet creation events and event processing and all services should now be operating normally. If you continue to experience any problems, please open a ticket with our Support team. Thank you for your patience and we apologize for any inconvenience.

Nov 1, 13:39 UTC
Update - We are continuing to monitor for any further issues.

Nov 1, 13:36 UTC
Monitoring - Our Engineers are currently monitoring an issue related to event processing and Droplet creates in our NYC2 region. Between 12:34 UTC and 13:09, users may have experienced failures with events such as creating, destroying, or rebuilding Droplets and Droplet-backed products. Our team has mitigated the impact and is monitoring the situation closely. We will post another update as soon as we can confirm the issue to be fully resolved.

Delays Reconciling Kubernetes Clusters

31 October 2024 at 13:56

Oct 31, 18:56 UTC
Resolved - From 10:30 UTC to 18:04 UTC, users may have experienced delays when adding new nodes to the Cluster or reconciling the Kubernetes Clusters.

Our Engineering team has confirmed the full resolution of the issue. Users should now be able to add new nodes or reconcile Kubernetes Clusters without any delays.

If you continue to experience problems, please open a ticket with our support team from within your Cloud Control Panel.

Oct 31, 18:04 UTC
Monitoring - Our Engineering team has confirmed that some backend changes caused this issue and as of now, there are no delays observed when reconciling or adding new nodes to the Clusters. At this time, users should be able to add new nodes or reconcile their Clusters without any delay.

We are monitoring the situation closely and will post an update as soon as we confirm the issue is fully resolved.

Oct 31, 17:17 UTC
Identified - Our Engineering team has identified the cause of the issue and is working on implementing a fix to make sure no delays are observed when reconciling the Kubernetes Clusters.

We apologize for the inconvenience and will post an update as soon as additional information is available.

Oct 31, 16:59 UTC
Investigating - Our Engineering team is currently investigating issues with Managed Kubernetes clusters. During this time, some users may experience delays when reconciling or adding nodes to their Kubernetes clusters.

We apologize for the inconvenience and will provide an update as soon as we have more information.

Received β€” 30 October 2024 ⏭ DigitalOcean Status - Incident History

Core Network Maintenance

8 November 2024 at 09:00

Nov 8, 14:00 UTC
Completed - The scheduled maintenance has been completed.

Nov 8, 13:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Nov 8, 18:07 UTC
Scheduled - Start Time: Friday, Nov 8, 2024 at 13:00 UTC
End Time: Friday, Nov 8, 2024 at 14:00 UTC

During the above window, our Engineering team will perform maintenance on the authoritative DNS infrastructure in order to improve reliability of the service. During this operation, the IPv4 and IPv6 addresses of β€œns3.digitalocean.com” will change. Customers do NOT need to take any action in preparation for this update.

Expected Impact:

This update is designed to be seamless and we do not expect any impact to DNS queries made to customer domains hosted by DigitalOcean as a result of this maintenance.

If an unexpected issue arises, resolution of some records in zones hosted with DigitalOcean may experience increased latency or a brief disruption. We will endeavor to keep any such impact to a minimum.

If you have any questions or concerns regarding this maintenance, please reach out to us by opening up a ticket on your account via https://cloudsupport.digitalocean.com/s/.

Core Network Maintenance

1 November 2024 at 09:00

Nov 1, 14:00 UTC
Completed - The scheduled maintenance has been completed.

Nov 1, 13:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Oct 29, 18:03 UTC
Scheduled - Start Time: Tuesday, Nov 1, 2024 at 13:00 UTC
End Time: Tuesday, Nov 1, 2024 at 14:00 UTC

During the above window, our Engineering team will perform maintenance on the authoritative DNS infrastructure in order to improve reliability of the service. During this operation, the IPv4 and IPv6 addresses of β€œns2.digitalocean.com” will change. Customers do NOT need to take any action in preparation for this update.

Expected Impact:

This update is designed to be seamless and we do not expect any impact to DNS queries made to customer domains hosted by DigitalOcean as a result of this maintenance.

If an unexpected issue arises, resolution of some records in zones hosted with DigitalOcean may experience increased latency or a brief disruption. We will endeavor to keep any such impact to a minimum.

If you have any questions or concerns regarding this maintenance, please reach out to us by opening up a ticket on your account via https://cloudsupport.digitalocean.com/s/.

Managed Database Creation

30 October 2024 at 00:27

Oct 30, 05:27 UTC
Resolved - From 04:36 to 04:51 UTC, users may have experienced errors when attempting to create Managed Database clusters.

Our Engineering team has confirmed the full resolution of the issue. Users should now be able to create Managed Database clusters normally.

If you continue to experience problems, please open a ticket with our support team from within your Cloud Control Panel.

Oct 30, 05:07 UTC
Monitoring - Our Engineering team has implemented a fix to resolve the issues impacting the creation of Managed Database Clusters. Users should now be able to create Managed Database clusters without any issues.

We are actively monitoring the situation and will post an update as soon as the issue is fully resolved.

Oct 30, 04:53 UTC
Investigating - As of 04:53 UTC ur Engineering team is investigating an issue with Managed Databases. At this time, users may experience errors while attempting to create new Managed Database instances in all regions.

We apologize for the inconvenience and will provide another update as soon as possible.

Received β€” 29 October 2024 ⏭ DigitalOcean Status - Incident History

AMS3 Power Maintenance

1 November 2024 at 02:00

Nov 1, 07:00 UTC
Completed - The scheduled maintenance has been completed.

Oct 31, 20:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Oct 29, 12:04 UTC
Scheduled - Start: 2024-10-31 20:00 UTC
End: 2024-11-01 06:00 UTC

During the above window, our Engineering team will be performing maintenance on redundant upstream electrical systems in order to improve reliability in our AMS3 region.

Expected Impact:

We don’t expect to see any downtime as the power to hypervisors is redundant. In a highly unlikely situation, where we lose power to both the supplies, the impacted Droplet(s) may experience downtime for a brief period. We will endeavor to keep this to a minimum for the duration of the change.


If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Container Registry and App Platform in BLR1

29 October 2024 at 02:27

Oct 29, 07:27 UTC
Resolved - From 01:56 to 06:36 UTC, our Engineering team observed an issue with Container Registry and App Platform builds in the BLR1 region.

During this time, users may have experienced delays while building their Apps and could have potentially experienced timeout errors in builds as a result. Additionally, a subset of customers may have experienced latency while interacting with the Container Registries.

Our Engineering team was able to take action to mitigate the impact and resolve the issue. All services are now functioning as expected. Thank you for your patience, and we apologize for any inconvenience.

If you have any questions or continue to experience issues, please reach out via a Support ticket on your account.

Received β€” 26 October 2024 ⏭ DigitalOcean Status - Incident History

Droplet Creation

25 October 2024 at 17:03

Oct 25, 22:03 UTC
Resolved - Our Engineering team identified an issue with new Droplet creates and rebuilds using Ubuntu 24.10 based Snapshot and Backup Images. From Oct 10 13:33 to Oct 25 19:58 UTC, users may have experienced issues while creating new Droplets using Ubuntu 24.10 based Snapshots and Backup Images in all of our regions.

We have disabled new Droplet creates and rebuilds using Ubuntu 24.10 based Image until new image has been published. Users who are still running Ubuntu 24.10 based Droplets should create offline Snapshots.

We apologize for the inconvenience. If you continue to experience any issues, please open a support ticket from within your account.

Received β€” 24 October 2024 ⏭ DigitalOcean Status - Incident History

Core Network Maintenance

29 October 2024 at 09:00

Oct 29, 14:00 UTC
Completed - The scheduled maintenance has been completed.

Oct 29, 13:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Oct 23, 19:21 UTC
Scheduled - Start Time: Tuesday, Oct 29, 2024 at 13:00 UTC
End Time: Tuesday, Oct 29, 2024 at 14:00 UTC

During the above window, our Engineering team will perform maintenance on the authoritative DNS infrastructure in order to improve reliability of the service. During this operation, the IPv4 and IPv6 addresses of β€œns1.digitalocean.com” will change. Customers do NOT need to take any action in preparation for this update.

Expected Impact:

This update is designed to be seamless and we do not expect any impact to DNS queries made to customer domains hosted by DigitalOcean as a result of this maintenance.

If an unexpected issue arises, resolution of some records in zones hosted with DigitalOcean may experience increased latency or a brief disruption. We will endeavor to keep any such impact to a minimum.

If you have any questions or concerns regarding this maintenance, please reach out to us by opening up a ticket on your account via https://cloudsupport.digitalocean.com/s/.

Spaces Bucket/DOCR Creates and App Platform Deployments

23 October 2024 at 17:00

Oct 23, 22:00 UTC
Resolved - From 20:10 - 21:24 UTC, we experienced an issue affecting the creation of Spaces Buckets across all regions, both through the Cloud Control Panel and the API. During this period, users may also have experienced errors in DOCR registry creations and App Platform application creations and deployments.

Our Engineering team was able to take quick action to mitigate the impact and resolve the issue. All services are now functioning as expected. Thank you for your patience, and we apologize for any inconvenience.

If you continue to experience any issues, please open a Support ticket for further analysis.

Received β€” 22 October 2024 ⏭ DigitalOcean Status - Incident History

NYC1 Snapshot Transfers

21 October 2024 at 20:01

Oct 22, 01:01 UTC
Resolved - Our Engineering team identified an issue with NYC1 Snapshot Image Transfers. From Oct 17, 15:09 to Oct 21, 23:08 UTC, users may have experienced issues with transferring images to NYC1 region.

A fix was deployed at Oct 21, 23:08 UTC by our Engineering team and all the pending image transfer events were settled. Users should now be able to transfer Snapshot images to NYC1.

We apologize for the inconvenience. If you continue to experience any issues, please open a support ticket from within your account.

Received β€” 17 October 2024 ⏭ DigitalOcean Status - Incident History

Intermittent Snapshot and Custom Image Uploads in LON1

17 October 2024 at 01:01

Oct 17, 06:01 UTC
Resolved - From 04:28 - 05:13 UTC, our Engineering team observed an intermittent issue with uploading Custom Images in LON1 region. During this time, users might have experienced an issue with uploading Custom Images and few Snapshot create failures.

Swift action was taken by our Engineering team to restore full functionality, and now everything is operating normally.

We apologize for any inconvenience this may have caused. If you have any questions or continue to experience issues, please reach out via a Support ticket on your account.

Received β€” 16 October 2024 ⏭ DigitalOcean Status - Incident History

Creating MongoDB Managed Databases

15 October 2024 at 18:25

Oct 15, 23:25 UTC
Resolved - After investigation, our Engineering team was able to determine that an internal-only issue was impacting MongoDB creation for DigitalOcean employees.

No customer impact occurred during the course of this incident.

We apologize for any confusion. If you are seeing any issues, please out to Support from within your account.

Oct 15, 22:54 UTC
Investigating - Our Engineering team is investigating an issue with MongoDB Managed Databases. At this time, users may experience errors while attempting to create MongoDB clusters in all regions.

We apologize for the inconvenience and will provide another update as soon as possible.

Spaces Page in Control Panel

15 October 2024 at 13:44

Oct 15, 18:44 UTC
Resolved - Our Engineering team has confirmed resolution of the issue impacting display of Spaces page in Cloud control panel.

From 17:35 UTC to 18:00 UTC, users were experiencing errors when accessing Spaces page in the Cloud Control Panel.

We apologize for the inconvenience. If you have any questions or continue to experience issues, please reach out via a Support ticket on your account.

Oct 15, 18:25 UTC
Monitoring - Our Engineering team has successfully deployed a fix for the display issue with Spaces Object Storage in the Cloud control panel. Some users may have experienced 400 errors or difficulties accessing the page when using Spaces through the Cloud Control Panel.

We are currently monitoring the situation and will post an update as soon as the issue is fully resolved.

Oct 15, 17:56 UTC
Identified - Our Engineering team has identified a display issues with Spaces Object Storage in the Cloud control panel and a fix is currently being deployed. During this time, some users may see 400 errors or be unable to access the page when accessing Spaces through the Cloud Control Panel.

We apologize for the inconvenience and will share an update once the fix has finished deploying.

Received β€” 15 October 2024 ⏭ DigitalOcean Status - Incident History

Internal Service Maintenance

15 October 2024 at 12:09

Oct 15, 17:09 UTC
Completed - The scheduled maintenance has been completed.

Oct 15, 15:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Oct 14, 23:10 UTC
Scheduled - Start Time: October 15 15:00 UTC
End Time: October 15 18:00 UTC

During this time, our Engineering Team will be performing maintenance on an internal service, which necessitates a period of read-only access to that service.

Engineering teams have taken steps to ensure their services do not experience downtime during the read-only period. Due to dependencies on this internal service, there is a possibility that some customer-facing services may experience minor disruptions, particularly while using Marketplace Add Ons or processing App Platform builds (small delays).

Existing infrastructure will continue running without issue and no public or private networking disruption will occur.

If other customer-facing services experience any disruption, we will update this post.

If you have any questions or concerns, please reach out to the Support team from within your account.

Received β€” 14 October 2024 ⏭ DigitalOcean Status - Incident History

Core Infrastructure Maintenance in NYC3

15 October 2024 at 16:00

Oct 15, 21:00 UTC
Completed - The scheduled maintenance has been completed.

Oct 15, 20:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Oct 13, 20:17 UTC
Scheduled - During the above window, our Engineering team will be performing maintenance on principal infrastructure in order to improve reliability of the services. Please note that existing infrastructure will continue running without issue. This maintenance may impact create, read, update, and delete (CRUD) operations in all regions.

Expected Impact:

During the maintenance window, users may experience increased latency for the following platform operations:

Cloud Control Panel and API operations
Event processing
Droplet creates, resizes, rebuilds, and power events
Managed Kubernetes reconciliation and scaling
Load Balancer operations
Container Registry operations
App Platform operations
Managed Database creation and scaling

We do not expect any impact to customer traffic due to this maintenance. If an unexpected issue arises, we will endeavor to keep any impact to minimum and may revert if required.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Received β€” 12 October 2024 ⏭ DigitalOcean Status - Incident History

Spaces and Container Registry in SYD1 region

11 October 2024 at 10:50

Oct 11, 15:50 UTC
Resolved - From 23:00 UTC on October 10th until 13:28 UTC on October 11th, users may have encountered intermittent errors when accessing Spaces endpoints or using the Container Registry in the SYD1 region. Our Engineering team has successfully resolved the issue affecting the functionality of Spaces and the Container Registry.

All services have been fully restored and are now functioning normally. If you continue to experience any problems, please open a ticket with our support team. We apologize for any inconvenience caused.

Oct 11, 14:10 UTC
Monitoring - Our Engineering team has implemented a fix to resolve the issue with Spaces and the Container Registry in the SYD1 region and is monitoring the situation closely. All services have been restored and are functioning as expected. We will post an update as soon as the issue is fully resolved.

Oct 11, 13:18 UTC
Investigating - Our Engineering team is currently investigating an issue with Spaces and the Container Registry in the SYD1 region. During this time, users may experience intermittent errors when accessing Spaces endpoints or interacting with the Container Registry. We apologize for the inconvenience and will provide an update as soon as we have more information.

❌