Feb 25, 18:28 UTC
Resolved - From 17:14 UTC to 17:34 UTC, users experienced issues accessing the Cloud Control Panel and API.

Our Engineering team has resolved the issue, and the Cloud Control Panel, API, and events should now be functioning normally.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Feb 25, 17:58 UTC
Monitoring - Our Engineering team has implemented a fix to resolve the issue impacting the Cloud Control Panel and API. Users should no longer experience issues with accessing the Cloud Control Panel and API.

We will continue to monitor the situation and post an update once this issue is fully resolved. Thank you for your patience.

Feb 25, 17:44 UTC
Identified - Our Engineering team has identified the cause of the issue impacting our Cloud Control Panel and API and is actively working on implementing a fix. We are already seeing signs of recovery, and users should now have access to the Cloud Control Panel and API.

We will post an update as soon as additional information is available.

Feb 25, 17:32 UTC
Update - We are continuing to investigate this issue.

Feb 25, 17:27 UTC
Investigating - Our Engineering team is investigating an issue impacting our Cloud Control Panel and API.

During this time, users will experience errors when trying to use the public API at api.digitalocean.com or while accessing our Cloud Control Panel at https://cloud.digitalocean.com. Users may also see issues with processing Droplet and Kubernetes cluster creations along with Droplet-based events.

We are now confirming the full impact. We regret any inconvenience and will let you know as soon as more information is available.

Feb 25, 20:16 UTC
Resolved - From 6:00 UTC to 17:45 UTC, users have experienced issues with Droplet creation and Droplet-based services in the NYC3 region.

Our Engineering team has fully resolved the issue impacting Droplet creation in the NYC3 region. Users can now create Droplets and Droplet-based services, including Managed Kubernetes, Managed Databases, Load Balancers, and snapshots, without any issues.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Feb 25, 18:43 UTC
Monitoring - Our Engineering team has implemented a fix to resolve the issue impacting the creation of Droplets in the NYC3 region. Users should now be able to create Droplets and Droplet-based services including Managed Kubernetes, Managed Databases, Load Balancers, and snapshots without any issues.

We are actively monitoring the situation and will post an update as soon as the issue is fully resolved.

Feb 25, 16:02 UTC
Identified - Our Engineering team has identified the reason behind the failed and delayed Droplet creates in the NYC3 region and implemented a partial mitigation to address this issue. As of now, customers may be able to create Droplets and Droplet-based services including Managed Kubernetes, Managed Databases, Load Balancers, and snapshots, however, there might be still some delay in the creates.

We apologize for the inconvenience and appreciate your patience as we work to resolve this issue. Further updates will be shared as they become available.

Feb 25, 12:45 UTC
Investigating - Our Engineering team is investigating an issue with Droplet creation in the NYC3 region. During this time, users may see an elevated failure rate for the creation of Droplets and other Droplet-based products, such as Load Balancers, Managed Databases, or Kubernetes in the NYC3 region.

We apologize for the inconvenience and will share an update once we have more information.

Feb 20, 00:20 UTC
Resolved - From February 6 21:28:35 UTC to February 19 23:14:00 UTC, users may have experienced issues receiving monitoring alerts for tagged resources.

Our Engineering team has resolved the issue, alerts for tagged resources are now working properly and users should now start receiving new alerts going forward.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Feb 19, 23:46 UTC
Monitoring - Our engineering team has implemented a fix for the issues that affect receiving monitoring alerts for tagged resources, and are currently monitoring the situation. Any alerts that should've been received beginning February 6th to today will not be received, but new alerts for tagged resources should work going forward.

We appreciate your patience, we will post an update once we've confirmed the issue is fully resolved.

Feb 19, 20:36 UTC
Investigating - Our Engineering team is investigating reports that alerts are not received when alerts are configured with tags. Users may not receive email notification of alerts when using tags, although resources specified by name are still monitored and receiving alerts properly.

We apologize for the inconvenience and will share an update once we have more information.

Feb 11, 11:44 UTC
Resolved - Our Engineering team has confirmed the full resolution of the issue affecting the Spaces Buckets creation and App Platform deployment in BLR1 region . All our services in the region should now be functioning normally.

Users should no longer experience any issues with the creation of Spaces Buckets and App Platform application creations and deployments.

If you continue to experience problems, please open a ticket with our support team from within your Cloud Control Panel. Thank you for your patience and we apologize for any inconvenience.

Feb 11, 10:38 UTC
Monitoring - Our engineering team has identified the cause and deployed a fix for the issue with Spaces Buckets creation and App Platform deployment in BLR1 region.

Spaces Buckets and App Platform application creations and deployments should now function as expected.

We are actively monitoring the situation and will provide an update once we confirm the incident is completely resolved.

Feb 11, 10:18 UTC
Identified - Our Engineering team has identified the cause of the issue affecting the creation of Spaces Buckets and App Platform application creations and deployments in BLR1. The impact is mitigated by implementing a temporary fix and is actively working on a permanent fix.

Users should now be able to create Spaces Bucket and deploy App Platform application in the BLR1 region.

We apologize for the inconvenience and will provide further updates as more information becomes available.

Feb 11, 09:07 UTC
Investigating - As of 08:32 UTC, our Engineering team is investigating an issue affecting the creation of Spaces Buckets in BLR1. During this period, users may also have experienced errors in DigitalOcean Container Registry creations and App Platform application creations and deployments.

We apologize for the inconvenience and will share an update once we have more information.

Feb 6, 01:29 UTC
Resolved - From 00:04 UTC to 00:41 UTC, users experienced issues accessing the Cloud Control Panel and API as well as experienced event failures.

Our Engineering team has resolved the issue, and Cloud Control Panel, API and events should now be functioning normally.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Feb 6, 01:01 UTC
Monitoring - Our Engineering team has implemented a fix regarding the issue impacting our Cloud Control Panel and API. We are monitoring the situation to ensure there is no recurrence.

We will post another update once we confirm the issue is fully resolved.

Feb 6, 00:37 UTC
Update - We are continuing to investigate this issue.

Feb 6, 00:27 UTC
Update - We are continuing to investigate this issue.

Feb 6, 00:26 UTC
Investigating - As of 00:16 UTC, our Engineering team is investigating an issue impacting our Cloud Control Panel and API.

During this time, users will experience errors when trying to use the public API at api.digitalocean.com or while accessing our Cloud Control Panel at https://cloud.digitalocean.com. Users may also see issues with processing Droplet and Kubernetes cluster creations along with Droplet based events.

We are now confirming the full impact. We regret any inconvenience and will let you know as soon as more information is available.

Feb 5, 04:21 UTC
Resolved - Our Engineering team identified an issue affecting Reserved IP assignments across all regions. From 00:26 to 03:26 UTC, users may have experienced issues with assigning Reserved IPs to Droplets.

A fix has been implemented and as of 03:26 UTC, all Reserved IP assignments have been successful. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Feb 5, 03:26 UTC
Resolved - Our Engineering Team has confirmed that the issue affecting GenAI agents in the Cloud has been fully resolved. Users should no longer see the "Failed to Update" banner and all functionality remains unaffected.

If you continue to experience any problems, please reach out to our support team by opening a ticket from within your Cloud Control Panel.

Feb 5, 02:09 UTC
Monitoring - Our Engineering team has implemented a fix to resolve the issues impacting GenAI agents in the Cloud. The banner should no longer appear, and functionality remains unaffected.

We are actively monitoring the situation and will post an update as soon as the issue is fully resolved.

Feb 4, 20:41 UTC
Identified - Our Engineering Team has identified an issue with GenAI agents where some customers may see a "Failed to Update" banner in the Cloud. Despite this message, customers can still update their images and use them as normal.

We apologize for the inconvenience and will provide another update as soon as possible.

Feb 3, 10:00 UTC
Resolved - Our Engineering team has confirmed the full resolution of the issue affecting Droplet, Managed Kubernetes, and Managed Database creation in the BLR region. All our services in the region should now be functioning normally.

Users should no longer experience any issues with Droplet creation, event processing, and the creation and provisioning of Managed Kubernetes and Managed Database clusters.

If you continue to experience problems, please open a ticket with our support team from within your Cloud Control Panel. Thank you for your patience and we apologize for any inconvenience.

Feb 3, 06:06 UTC
Monitoring - Our Engineering team has deployed a fix for the issue affecting Droplet, Managed Kubernetes, and Managed Database creation in the BLR region. As of 6:00 AM UTC, the issue has been fully mitigated, and users should no longer experience any disruptions.

Droplet creation, event processing, and the creation and provisioning of Managed Kubernetes and Managed Database clusters should now function as expected.

We are actively monitoring the situation and will provide an update once we confirm the incident is completely resolved.

Feb 3, 05:49 UTC
Identified - As of 5:40 AM UTC, our Engineering team has identified the cause of the issue affecting Droplet, Managed Kubernetes, and Managed Database creation in the BLR region and is actively implementing a fix.

Users should begin to see improvements in Droplet creation and event processing. Additionally, the successful creation and provisioning of Managed Kubernetes and Managed Database clusters should also resume shortly.

We apologize for the inconvenience and will provide further updates as more information becomes available.

Feb 3, 04:28 UTC
Investigating - As of 3:16 AM UTC, our Engineering team is investigating an issue affecting Droplet, Managed Kubernetes, and Managed Database creation in the BLR region.

Users may also experience delays or issues with event processing related to Droplets in the BLR region. Additionally, users may see errors with Managed Kubernetes and Managed Databases cluster creation and provisioning.

We apologize for the inconvenience and will provide an update as soon as more information becomes available.

Jan 30, 18:11 UTC
Resolved - Our Engineering team has confirmed the full resolution of the issue impacting the GenAI Platform. Users should now be able to fetch resource data, usage, and other details from within the Cloud Panel without any issues.

If you continue to experience problems, please open a ticket with our support team from within your Cloud Control Panel.

Jan 30, 17:56 UTC
Monitoring - Our Engineering team has implemented a fix to resolve the issues impacting the GenAI Platform in the Cloud Panel. Users should now be able to interact with the GenAI Platform or fetch the resource data, usage, etc without any issues.

We are actively monitoring the situation and will post an update as soon as the issue is fully resolved.

Jan 30, 17:43 UTC
Investigating - Our Engineering team is investigating an issue with the GenAI Platform. At this time, users may experience errors in the Cloud Panel when performing tasks such as requesting resource data, usage, etc.

We apologize for the inconvenience and will provide another update as soon as possible.

Jan 29, 16:16 UTC
Resolved - The issue affecting Snapshots and Backups access from the DigitalOcean Cloud Panel has been fully resolved. Users should now be able to access these services without any errors.

We appreciate your patience while we worked to fix this issue. If you continue to experience any problems, please open a ticket from your account for further investigation.

Jan 29, 15:49 UTC
Monitoring - Our Engineering team have implemented a fix for the issue affecting Snapshots and Backups access from the DigitalOcean Cloud Panel. Services are now recovering, and we are closely monitoring the situation to ensure stability.

If you continue to experience any issues, please open a ticket from your account for further review.

Jan 29, 15:40 UTC
Investigating - Our Engineering team is investigating an issue with Backups and Snapshots on Cloud Control Panel. During this time, users may experience error while accessing Backups and Snapshot webpage. We apologize for the inconvenience and will share an update once we have more information.

Jan 24, 14:21 UTC
Resolved - Our Engineering team has successfully resolved the issue affecting both the creation and regeneration of Spaces Access Keys. Between 13:05 UTC and 13:50 UTC, users may have faced intermittent errors when creating or regenerating Access Keys. All services are now fully restored and operating normally.

If you're still encountering any issues, please open a ticket with our support team. We apologize for any inconvenience caused.

Jan 23, 23:47 UTC
Resolved - From 19:44 UTC to 23:19 UTC, users may have experienced issues with accessing the Droplet Recovery Console.

We have confirmed that the issue is fully resolved, and users should be able to access the Recovery Console from the Cloud Control Panel as normal.

If you continue to experience problems, please open a ticket with our support team. Thank you for your patience, and we apologize for any inconvenience.

Jan 23, 23:25 UTC
Monitoring - Our Engineering team has implemented a fix to resolve the issue with the Droplet Recovery Console and are currently monitoring the situation.

Users should no longer experience issues with accessing the Recovery Console in the Cloud Control Panel.

We will continue to monitor the situation and post an update once this issue is fully resolved. Thank you for your patience.

Jan 23, 23:17 UTC
Identified - Our Engineering team has identified the cause of the issue with accessing the Droplet Recovery Console in the Cloud Control Panel and is actively working on a fix.

We will post an update as soon as additional information is available.

Jan 23, 22:23 UTC
Investigating - Our Engineering team is investigating an issue affecting the ability to access the Droplet Recovery Console in the Cloud Control Panel.

At this time, users may experience issues with accessing the recovery console for their Droplets in all regions. The normal Droplet web console seems to be working fine.

We apologize for the inconvenience and will share an update once we have more information.

Jan 21, 22:21 UTC
Resolved - Prior to Tuesday, January 21, 2025 at 19:20 UTC, customers that used an AWS CLI or AWS SDK version released on or after January 15, 2025 may have experienced issues with uploading files to a Spaces bucket. This impacted PutObject and UploadPart requests initiated by recent versions of the AWS CLI and AWS SDKs.

Our Engineering team has confirmed resolution of this issue for almost all Spaces buckets (including all new Spaces buckets), which are now compatible with the latest versions of the AWS CLI and AWS SDKs (and tools and applications that depend on these versions).

Note that Spaces does not currently verify data integrity checksums sent by the AWS CLI and AWS SDKs as part of upload requests (see Data Integrity Protections for Amazon S3 - https://docs.aws.amazon.com/sdkref/latest/guide/feature-dataintegrity.html).

Customers with a bucket that is still incompatible will receive an email from DigitalOcean with follow up steps. If you continue to experience problems, please open a ticket with our Support team. Thank you for your patience through this issue.

Jan 16, 19:14 UTC
Identified - DigitalOcean Spaces Customers using the latest versions (released on Jan 15, 2025) of the AWS CLI or any AWS SDK, or relying on a tool or application that depends on any of these versions of the AWS CLI or an AWS SDK, are experiencing failed uploads to Spaces buckets in all regions.

This is caused by a change made by AWS on January 15, 2024 to the AWS CLI and almost all AWS SDKs (including boto3) to require use of “Data Integrity Protections for Amazon S3” (https://docs.aws.amazon.com/sdkref/latest/guide/feature-dataintegrity.html).

The currently recommended solution is to downgrade (roll back) to a version of the AWS CLI or AWS SDK released prior to January 15, 2025, or to an earlier version of tools or applications that do not depend on one of these versions of the AWS CLI or AWS SDK. The list of known impacted AWS client tools are:
AWS CLI v2.23.0 (workaround - move to last known good version: v2.22.35)
AWS SDK for Python (boto3) v1.36.0 (workaround - move to last known good version: 1.35.99)
AWS SDK for Go 2025-01-15 release (workaround - move to last known good release: 2025-01-14)

Jan 15, 12:54 UTC
Resolved - From 11:29 to 12:31 UTC, our Engineering team observed a Networking issue in our BLR and SFO regions. During this time, users may have experienced Droplet connectivity issue, Users should no longer be experiencing these issues.
We apologize for the inconvenience. If you have any questions or continue to experience issues, please reach out via a Support ticket on your account.

Jan 12, 00:00 UTC
Completed - The scheduled maintenance has been completed.

Jan 11, 21:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jan 11, 20:25 UTC
Scheduled - Start: 2025-01-11 21:00 UTC
End: 2025-01-12 24:00 UTC

Hello,

During the above window, our Networking team will be making changes to core networking infrastructure, to improve performance and scalability in the NYC3 region.

Expected impact:

During the maintenance window users may experience delays or failures with event processing for a brief duration on Droplets and Droplet-based services including Droplets, Managed Kubernetes, Load Balancers, Container Registry, and App Platform. We will endeavor to keep this to a minimum for the duration of the change.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Jan 11, 21:12 UTC
Resolved - As of 18:55 UTC, our Engineering team has confirmed the full resolution of the issue that impacted Managed Kubernetes Clusters in all of our regions. All of the services should now be working normally.

If you continue to experience problems, please open a ticket with our Support team from within your Cloud Control Panel.

Thank you for your patience and we apologise for any inconvenience.

Jan 11, 18:29 UTC
Investigating - Our Engineering team is currently investigating an issue with Managed Kubernetes clusters. During this time, some users may experience delays when provisioning Kubernetes clusters, specifically version 1.31.1-5.

We apologize for the inconvenience and will provide an update as soon as we have more information.

Jan 9, 18:01 UTC
Resolved - From 15:28 to 15:32 UTC, an internal service disruption may have resulted in users experiencing errors while using the Cloud Panel or API to manage Spaces Buckets, Apps, Managed Database Clusters, or Load Balancers as well as other actions due to impacted downstream services.

If you continue to experience problems, please open a ticket with our support team.

Thank you and we apologize for any inconvenience.

Jan 7, 16:43 UTC
Resolved - From 13:09 - 14:48 UTC, we experienced an issue with IPv6 network connectivity in our SFO3 region. Our Engineering team was able to identify the root cause, revert changes, and resolve the issue.

During that time, connectivity via IPv6 to services in SFO3 was impacted. This included, in particular, connectivity to and provisioning of Load Balancers, control plane connectivity to Kubernetes clusters, and connectivity to App Platform Apps.

If you continue to experience problems, please open a ticket with our support team. Thank you for your patience and we apologize for any inconvenience.

Jan 7, 15:51 UTC
Monitoring - Our engineering team has implemented a fix to resolve the networking issue in our SFO3 region. Users should no longer encounter connectivity issues, provisioning Load Balancers, latency, and timeout errors while interacting with resources in the SFO3 region.

We will post an update as soon as the issue is fully resolved.

Jan 7, 14:47 UTC
Update - Our Engineering team is continuing to investigate the cause of the issue with networking in our SFO3 region and is taking action to mitigate this incident by reverting recent changes. In addition to connectivity issues, users may see failures with provisioning Load Balancers in SFO3.

We are seeing some services recover, but expect to see a few more short periods of network connectivity issues as the reverts progress.

We will continue to share updates as soon as we have new information.

Jan 7, 13:35 UTC
Investigating - Our engineering team is investigating an issue related to the SFO3 network maintenance here: https://status.digitalocean.com/incidents/lf0njg8ysj0w

During this time users may experience connectivity issues, latency, and timeout errors while interacting with resources in the SFO3 data center.

We apologize for the inconvenience and will share an update once we have more information.

Jan 7, 14:14 UTC
Resolved - Our engineering team has resolved the issue preventing some users from adding new credit cards from the control panel. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Jan 7, 13:38 UTC
Monitoring - Our engineering team has implemented a fix to resolve issues that some users experienced when attempting to add new credit cards to the Cloud Control Panel.

We are monitoring the situation and will post an update as soon as the issue is fully resolved.

Jan 7, 13:24 UTC
Investigating - Our engineering team is investigating reports of issues occurring when users attempt to add a new credit card in the billing settings page of the Cloud Control Panel.

During this time, some users may be unable to input card information. We apologize for the inconvenience and will share an update once we have more information.

Jan 7, 14:00 UTC
Completed - The scheduled maintenance has been completed.

Jan 7, 10:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jan 5, 10:15 UTC
Scheduled - Start: 2025-01-07 10:00 UTC
End: 2025-01-07 14:00 UTC

During the above window, our Networking team will be making changes to core networking infrastructure, to improve performance and scalability in the SFO3 region.

Expected impact:

These upgrades are designed and tested to be seamless and we do not expect any impact to customer traffic due to this maintenance. If an unexpected issue arises, affected Droplets and Droplet-based services may experience increased latency or a brief disruption in network traffic. We will endeavor to keep any such impact to a minimum.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket