Normal view

New Kubernetes Clusters GPU Nodes

Jul 10, 23:13 UTC
Resolved - Our Engineering team has fully resolved the issue with new Kubernetes cluster GPU creations. Users with new clusters should be able to create GPU nodes without issue.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Jul 10, 21:19 UTC
Identified - Our Engineering team has identified an issue impacting creations of GPU nodes on new Kubernetes clusters. New clusters on the latest version will not be able to launch a new GPU node. Existing workloads are not impacted.

Our team is working on deploying a fix for this issue, We will post an update once we have more information.

Managed Database Provisioning and App Platform

Jul 8, 17:01 UTC
Resolved - Our Engineering team has resolved the issue affecting all control plane operations (Create, Read, Update, Delete) for non-MongoDB Managed Database clusters. All services and App Platform deployments with Managed Databases should now be operating normally.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Jul 8, 16:01 UTC
Monitoring - Our Engineering team has deployed a fix to address the issue affecting all control plane operations (Create, Read, Update, Delete) for non-MongoDB Managed Database clusters. Users should now be able to perform standard database operations successfully. App Platform deployments with Managed Databases should also be succeeding now.

We are now monitoring the situation in order to ensure stability and will post an update once the issue is resolved completely.

Jul 8, 14:57 UTC
Investigating - Our Engineering team is investigating an issue that is impacting all control plane operations (Create, Read, Update, Delete) for all non-Mongo Managed Database clusters. During this time, users may encounter long running queries when attempting those operations, including provisioning new clusters, listing clusters, etc.

In addition, App Platform deploys with Managed Databases are also affected, which may result in delayed or failed deployments for some applications.

We will provide updates as more information becomes available.

Dangerous mines: A death at the bottom of the EV supply chain

Darlington Vivito told his parents goodbye on a chilly, drizzly evening in March 2023. He put on warm clothes, preparing for the night ahead. The 22-year-old left the family’s small...

© Zinyange Auntony for Rest of World

A security guard carries a gun while patrolling the open pit mines at Bikita Minerals, a lithium mine operated by the Chinese conglomerate Sinomine Resource Group in the Bikita district of Masvingo, a southern province of Zimbabwe.

Despite being situated above one of the largest lithium deposits, the Bikita district remains underdeveloped. Residents face poor road conditions, a lack of piped water, and insufficient educational facilities. Zimbabwe possesses significant lithium reserves and is emerging as a key player in the global metal market, which is essential for producing batteries for electric vehicles and other applications.

Rescheduled: NYC1 Network Maintenance 2025-07-09 22:00 UTC

Jul 10, 00:00 UTC
Completed - The scheduled maintenance has been completed.

Jul 9, 22:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jul 6, 23:35 UTC
Update - We will be undergoing scheduled maintenance during this time.

Jul 6, 23:32 UTC
Scheduled - We are reaching out again to inform you that the Network maintenance in NYC1 region which was previously scheduled to be complete on 2025-06-18 22:00 UTC has been rescheduled to the following window:

Start: 2025-07-09 22:00 UTC
End: 2025-07-10 00:00 UTC

We apologize for any inconvenience this short notice causes and thank you for your understanding. You may find the initial maintenance notice along with a description of any expected impact related to this work included at the bottom of this message.

If you have questions or concerns about this maintenance, please reach out to us by opening up a ticket on your account.

---BEGIN INITIAL MAINTENANCE NOTICE—

Start: 2025-06-18 22:00 UTC
End: 2025-06-19 00:00 UTC

Hello,

During the above window, our Networking team will be making changes to the core networking infrastructure to improve performance and scalability in the NYC1 region.

Expected impact:

During the maintenance window, users may experience delays or failures with event processing for a brief duration on Droplets and Droplet-based services, including Droplets, Managed Kubernetes, Load Balancers, Container Registry, and App Platform. Additionally, regular requests via the Public API, as well as viewing and submitting requests through the Cloud Control Panel, may time out or fail for up to 5 minutes. We will endeavor to keep this to a minimum for the duration of the change.

If you have any questions related to this issue, please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Managed Database Provisioning and App Platform

Jul 2, 18:40 UTC
Resolved - Our Engineering team has resolved the issue affecting all control plane operations (Create, Read, Update, Delete) for non-MongoDB Managed Database clusters. All services and App Platform deployments with Managed Databases should now be operating normally. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Jul 2, 17:43 UTC
Monitoring - Our Engineering team has deployed a fix to address the issue affecting all control plane operations (Create, Read, Update, Delete) for non-MongoDB Managed Database clusters. Users should now be able to perform standard database operations successfully. App Platform deployments with Managed Databases should also be succeeding now.

We are now monitoring the situation in order to ensure stability and will post an update once the issue is resolved completely.

Jul 2, 16:59 UTC
Investigating - Our Engineering team is investigating an issue that is impacting all control plane operations (Create, Read, Update, Delete) for all non-Mongo Managed Database clusters. During this time, users may encounter 401/403 errors when attempting those operations, including provisioning new clusters, listing clusters, etc.

In addition, App Platform deploys with Managed Databases are also affected, which may result in delayed or failed deployments for some applications.

We will provide updates as more information becomes available.

Creating Agents with Attached Knowledge Base

Jul 2, 20:15 UTC
Resolved - This incident has been resolved.

Jul 2, 19:50 UTC
Monitoring - All previously reported issues have now been resolved.

Resolved Issues:

- Indexing jobs are now operational and starting as expected.
- Indexing job statuses are updating normally.
- Evaluation runs on new teams are fully operational.
- Knowledge base creation is functioning correctly.
- Linking functions to agents is now operational.

We appreciate your patience while our team worked to resolve these issues. We will continue to monitor the fixes to ensure everything remains stable.

Jul 2, 19:19 UTC
Update - Our Engineering team continues to work on a fix for this issue. We will share further updates as soon as they become available. Thank you for your continued patience.

Jul 2, 17:39 UTC
Identified - Our Engineering team has identified several issues impacting indexing jobs and knowledge base creation.

At this time, users may experience the following:

- Indexing jobs are currently failing to start. As a result, most teams will be unable to initiate new jobs.
- For teams where jobs do start, they may remain stuck in a "queued" status.
- Knowledge base creation is still failing and may result in an error.

We apologize for the disruption and will provide further updates as we learn more.

Jul 2, 16:29 UTC
Monitoring - A fix has been implemented and we are monitoring the results.

Jul 2, 15:08 UTC
Identified - The issue has been identified and a fix is being implemented.

Jul 2, 14:24 UTC
Investigating - Our Engineering team is investigating an issue with creating agents using an attached knowledge base.

At this time, users may experience errors when attempting to create a new agent and specify a knowledge base during the creation process.

We apologize for the inconvenience and will share an update once we have more information.

SSL/TLS Handshake Failures in FRA1 & SFO3

Jul 3, 19:55 UTC
Resolved - Our Engineering team has confirmed that the issue affecting access to Object Storage Buckets in the FRA1 and SFO3 regions has been fully resolved. Users should no longer experience SSL/TLS handshake failures when accessing their Buckets in these regions.

We appreciate your patience and understanding while we worked to resolve this matter. If you continue to experience any issues, please don't hesitate to open a support ticket from your account for further assistance.

Jul 3, 19:15 UTC
Monitoring - Our Engineering team has identified the root cause of the issue as a block imposed by a third-party provider on users utilizing residential cable services and their advanced security offerings.

During this time, a subset of users accessing their Spaces buckets in the FRA1 and SFO3 regions via residential cable services may have experienced SSL/TLS handshake failures.

This block was removed as of 17:48 UTC, and users should no longer encounter errors when accessing the contents of their buckets.

We are actively working with the third-party provider to ensure that such blocks are not imposed again in the future.

We will continue to monitor the situation and post an update once this issue is fully resolved. Thank you for your patience.

Jul 2, 13:43 UTC
Update - Our Engineering team continues to investigate reports of SSL/TLS handshake failures affecting Object Storage access in the FRA1 and SFO3 regions. The team is reviewing logs and traffic flows to isolate the root cause and determine whether a misconfiguration or external dependency is contributing to the failures.

Our team is actively working to resolve the issue and we will provide further updates as soon as we have more information. Thank you for your continued patience.

Jul 1, 19:01 UTC
Investigating - Our Engineering team is currently investigating reports of SSL/TLS handshake failures for Object Storage access in FRA1 and SFO3 regions. Affected users may see connection errors or timeouts when attempting to access buckets over HTTPS.

We are actively working to identify the root cause and implement a solution and will share further updates as more information becomes available.

❌