May 15, 17:01 UTC
Resolved - From 13:45 UTC to 15:50 UTC, users may have experienced an issue affecting the GenAI Platform agents using Llama 3.3 70B.

Our Engineering team has confirmed full resolution of the issue.

If you continue to experience any issues, please reach out to our support team by opening a ticket from within your Cloud Control Panel.

May 15, 16:18 UTC
Monitoring - Our Engineering team has implemented a fix to address the issue impacting the GenAI Platform agents using Llama 3.3 70B. As of now, GenAI Platform agents using Llama 3.3 70B should respond without any issues.

We are actively monitoring the situation and will post an update as soon as the issue is fully resolved.

We apologize for the inconvenience and appreciate your patience.

May 15, 15:26 UTC
Identified - Our Engineering team has identified the cause of the issue affecting the GenAI Platform. During this time, GenAI Platform agents using Llama 3.3 70B will fail to respond.

A fix is in progress, and we will provide an update as soon as we have more information.

May 15, 14:58 UTC
Investigating - Our Engineering Team is currently investigating an issue with the GenAI Platform where the GenAI Platform agents using Llama 3.3 70B are failing to respond.

We are actively working to identify the root cause. We apologize for the inconvenience and will share an update once we have more information.

May 15, 06:53 UTC
Resolved - From 03:20 UTC to 06:01 UTC, users may have experienced an issue affecting the GenAI Platform using Llama 3.370B where chatbots are failing to respond

Our Engineering team has confirmed full resolution of the issue. Users should now be able to run chatbots normally.

If you continue to experience any issues, please reach out to our support team by opening a ticket from within your Cloud Control Panel.

May 15, 06:17 UTC
Monitoring - Our Engineering team has implemented a fix to address the issue impacting with the GenAI Platform agents using Llama 3.370B where chatbots are failing to respond. We are currently monitoring the issue.

We will provide an update as soon as more information becomes available.

We apologize for the inconvenience and appreciate your patience.

May 15, 05:41 UTC
Investigating - We are investigating an issue where GenAI Platform agents are failing to respond. We apologize for the inconvenience this may cause.

May 14, 15:43 UTC
Resolved - Our Engineering team has completely resolved the issue that was affecting the Droplet search functionality within the DigitalOcean Cloud Control Panel. We have observed normal functionality following the fix and continued monitoring has shown stable behavior. Users should be able to successfully search for and locate their Droplets. The issue is now considered resolved.

We appreciate your patience and apologize for any inconvenience this may have caused. However, if you continue to face issues then please open a ticket with our Support team for further review.

May 14, 14:32 UTC
Monitoring - Our Engineering team has implemented a fix for the issue affecting the droplet search functionality within the DigitalOcean Cloud Control Panel. Users should now be able to search for the droplets using the search box on the droplets listing page.

We are continuing to monitor the situation closely to ensure stability. Thank you for your patience during the process and we will provide a final update once we confirm the issue is fully resolved.

May 14, 12:34 UTC
Investigating - Our Engineering team is currently investigating an issue affecting the Droplet search functionality within the DigitalOcean Cloud Panel. Some customers may be experiencing difficulties when attempting to search for and locate their Droplets.

We are actively working to identify the root cause and will provide updates as we learn more. We understand how critical this functionality is for managing your infrastructure, and we appreciate your patience during the process.

We will share an update as more information becomes available.

May 13, 23:22 UTC
Resolved - Our Engineering team has confirmed that the issue affecting the GenAI Platform DeepSeek model has been fully resolved. Users should no longer encounter the message "It looks like the agent ran out of tokens while reasoning. Please try again or increase the max tokens."

All functionality remains unaffected, and users should no longer experience any interruptions.

If you continue to experience any issues, please reach out to our support team by opening a ticket from within your Cloud Control Panel.

May 13, 22:51 UTC
Monitoring - Our Engineering team has implemented a fix to address the issue impacting the GenAI Platform's DeepSeek model, where some users were encountering the message "It looks like the agent ran out of tokens while reasoning. Please try again or increase the max tokens".

At this time, users should be seeing agents run normally.

We are actively monitoring the situation and will post an update as soon as the issue is fully resolved.

May 13, 21:50 UTC
Investigating - Our Engineering Team is currently investigating an issue with the GenAI Platform DeepSeek model where some users may encounter the message "It looks like the agent ran out of tokens while reasoning. Please try again or increase the max tokens."

We will provide an update as soon as more information becomes available.

We apologize for the inconvenience and appreciate your patience.

May 8, 13:01 UTC
Resolved - Our Engineering team has confirmed the resolution of the issue affecting certain Droplet metrics in the BLR1 region. Users should no longer see intermittent gaps in monitoring graphs for Droplets in the BLR1 region.

We appreciate your patience and apologize for any inconvenience this may have caused.

May 8, 01:26 UTC
Monitoring - Our Engineering team has implemented a fix to address the intermittent unavailability of Droplet metrics and delayed Resource Alerts in the BLR1 region.

Our team is now and monitoring the results and will continue to do so for an extended period to ensure this sporadic issue is addressed by the implemented fix.

At this time, we expect users not to experience further metrics gaps or delays in alerts.

We will post an update as soon as we confirm this incident is fully resolved or once we have further information. Thank you for your patience.

May 7, 22:04 UTC
Update - Our Engineering team is continuing to investigate the root cause of the issue with intermittent unavailability of Droplet metrics in the BLR1 region. Multiple reproduction efforts and testing against potential root causes are underway. Due to the sporadic nature of the issue, we anticipate incident updates for this issue to be less frequent, but we will share new information as soon as it is available.

At this time, users impacted by this incident will experience intermittent gaps in monitoring graphs for Droplets in BLR1, as well as delayed notifications for Resource Alerts (for any resources in BLR1).

We apologize for the inconvenience, and we'll share an update once we have more information.

May 7, 09:41 UTC
Update - Our Engineering team continues to investigate the intermittent unavailability of certain Droplet metrics in the BLR1 region. Due to the sporadic nature of this issue, our analysis requires additional time to correlate patterns and isolate the root cause.

We apologize for the inconvenience, and we'll share an update once we have more information.

May 6, 13:00 UTC
Investigating - Our Engineering team is currently investigating issues with certain Droplet metrics that are missing for the BLR1 region. Users may experience issues when accessing certain metrics on Droplets.

We apologize for the inconvenience, and we'll share an update once we have more information.

May 5, 15:46 UTC
Resolved - Our Engineering team has confirmed the complete resolution of the issue that was affecting Droplet creation and console access in the NYC1 region. Users should now be able to deploy new droplets and access them normally in the NYC1 region.

All systems are now operating normally. We appreciate your patience and apologize for any inconvenience this may have caused.

May 5, 15:18 UTC
Monitoring - Our Engineering team has implemented a fix for the issue affecting Droplet creation and console access in the NYC1 region. Impacted users may have seen 504 Gateway Timeout errors during Droplet creation or when accessing the Droplet console.

Our team is currently monitoring the situation to ensure stability. We will provide a final update once we confirm the issue is fully resolved.

May 5, 14:18 UTC
Investigating - Our Engineering team is currently investigating an issue with creating Droplets in NYC1 region. During this time, users may see 504 Gateway Timeout errors when creating droplets in the NYC1 region.

We apologize for the inconvenience and will provide an update as soon as we have more information.

May 7, 23:00 UTC
Completed - The scheduled maintenance has been completed.

May 7, 20:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

May 4, 20:10 UTC
Scheduled - Start: 2025-05-07 20:00 UTC
End: 2025-05-07 23:00 UTC

During the above window, our Networking team will be making changes to the core networking infrastructure to improve performance and scalability in the LON1 region.

Expected impact:

During the maintenance window, users may experience delays or failures with event processing for a brief duration on Droplets and Droplet-based services, including Droplets, Managed Kubernetes, Load Balancers, Container Registry, and App Platform. We will endeavor to keep this to a minimum for the duration of the change.

If you have any questions related to this issue, please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Apr 30, 14:57 UTC
Resolved - Our Engineering team has confirmed the complete resolution of the issue that was impacting the availability of Spaces in our FRA1 region. Users should no longer experience elevated error rates when accessing Spaces in the region.

We appreciate your patience and understanding while we worked to restore normal service.

If you continue to experience any issues, please open a support ticket from within the Cloud Control Panel for further review.

Apr 30, 14:14 UTC
Monitoring - Our Engineering team has implemented a fix for the issue affecting the availability of Spaces in our FRA1 region. Users should now see improved performance, and error rates have decreased.

We are continuing to monitor the situation closely to ensure stability. Thank you for your patience during the process and we will provide a final update once we confirm the issue is fully resolved.

Apr 30, 13:06 UTC
Investigating - Our Engineering team is currently investigating an issue affecting the availability of Spaces in our FRA1 region. During this time, users may experience high error rate when attempting to access Spaces in this region.

We are actively working on identifying the root cause and resolving the issue as quickly as possible. We understand the inconvenience this may cause and appreciate your patience.

We will provide further updates in the due course.

Apr 30, 00:11 UTC
Resolved - Our Engineering team has confirmed full resolution of the network issue with new DOKS worker nodes in our SFO2 region.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Apr 29, 23:49 UTC
Monitoring - Our Engineering team identified the cause of the issue and has rolled out a fix to resolve the network issue with new DOKS worker nodes. At this time, all services should be operating normally.

We are actively monitoring the fix and will post an update as soon as the issue is fully resolved.

Apr 29, 22:25 UTC
Investigating - Our Engineering team is investigating an issue causing new DOKS worker nodes to have network issues in our SFO2 region. Operations such as creating new DOKS nodes, manually or autoscaled, will have network connectivity issues within the clusters they are added to during this time.

Our Engineering team is actively working to identify the root cause and restore service as quickly as possible. We will continue to provide updates here as we make progress.

Apr 30, 21:00 UTC
Completed - The scheduled maintenance has been completed.

Apr 30, 13:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Apr 28, 13:25 UTC
Scheduled - Start: 2025-04-30 13:00 UTC
End: 2025-04-30 21:00 UTC

Hello,

During the above window, our Engineering team will be performing maintenance on core control plane infrastructure. Please note that the existing infrastructure will continue running without issue. This maintenance may impact create, read, update, and delete (CRUD) operations in all regions.

Expected Impact:

During the maintenance window, users may experience increased latency during two 5 minute windows with the following platform operations:

Cloud Control Panel and API operations
Event processing
Droplet creates, resizes, rebuilds, and power events
Managed Kubernetes reconciliation and scaling
Load Balancer operations
Container Registry operations
App Platform operations
Managed Database creation and scaling

We do not expect any impact to customer traffic due to this maintenance. If an unexpected issue for the control plane arises, we will endeavor to keep any impact to a minimum and may revert if required.

If you have any questions related to this maintenance please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Thank you,
Team DigitalOcean

Apr 28, 11:17 UTC
Resolved - This incident has been resolved.

Apr 28, 11:14 UTC
Investigating - Between 09:58 UTC and 10:11 UTC, our Engineering team observed issues impacting DNS resolution in our LON1 region. During this time, some users may have experienced errors while trying to resolve domain names from within Digitalocean resources.

Our team was able to take quick action to mitigate the impact and resolve the issue, and all services in the LON1 region are now functioning normally. Thank you for your patience, and we apologize for any inconvenience.

If you are still experiencing issues or have additional questions, please open a Support ticket right away.

Apr 24, 16:31 UTC
Resolved - Our Engineering team has confirmed full resolution of the issue impacting event processing in the FRA1, AMS2, BLR1, SGP1, NYC1, and SYD1 regions.

If you continue to experience any problems, please open a ticket with our Support team. Thank you for your patience, and we apologize for any inconvenience.

Apr 24, 15:53 UTC
Monitoring - Our Engineering team has identified the cause of the issue causing delays in Droplet event processing in our FRA1 region, and has implemented a fix.

While investigating this issue, our Engineers also observed some events appearing to be stuck or delayed in the AMS2, BLR1, SGP1, NYC1, and SYD1 regions. As of now, the event processing rates have returned to normal in all these regions.

We are currently monitoring the situation and will post an update as soon as the issue is fully resolved.

Apr 24, 13:41 UTC
Investigating - Our Engineering team is investigating an issue causing delays in Droplet event processing in our FRA1 region.

This includes operations such as creating, resizing, powering on/off, or deleting Droplets. These actions may take longer than usual to complete or may appear to be temporarily stalled.

Our engineering team is actively working to identify the root cause and restore normal processing speeds as quickly as possible. We will continue to provide updates here as we make progress.

Apr 24, 04:41 UTC
Resolved - Our Engineering team has resolved the issue with the email delivery delays. Services should now be operating normally.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Apr 24, 00:28 UTC
Monitoring - Our Engineering team has confirmed the email delivery delays caused by an issue with an upstream provider has been mitigated. We are now seeing successful delivery of previously delayed emails, as well as normal flow of new messages.

We are currently monitoring the situation closely and will share an update as soon as the issue is fully resolved.

Apr 23, 22:34 UTC
Identified - Our Engineering team is aware of an issue with an upstream provider that is impacting the delivery of emails from DigitalOcean to DigitalOcean customers.

Beginning 20:55 UTC, users may experience delays in receiving emails from DigitalOcean, including sign-in verification codes, sign-up verifications, maintenance notifications, and other transactional emails.

Our team is closely monitoring the situation and has begun to see signs of recovery, with both delayed and recent emails starting to be delivered.

We will provide further updates as they are available.

Apr 22, 19:05 UTC
Resolved - From 16:54 to 18:17 UTC, users may have experienced issues creating Spaces from control panel due to the datacenter region options failing to list.

Our Engineering team has implemented a fix and confirmed full resolution of the issue. Users should be able to create Spaces from Cloud Control Panel without issues now.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Apr 22, 18:30 UTC
Monitoring - Our Engineering team has implemented a fix to resolve the issue impacting the creation of Spaces. Users should now be able to create Spaces without any issues.

We are actively monitoring the situation and will post an update as soon as the issue is fully resolved.

Apr 22, 18:08 UTC
Identified - Our Engineering team has identified an issue with creating Spaces through the Cloud Control Panel. During this time, users are unable to create Spaces buckets due to the datacenter region options failing to list. Creating Spaces buckets through the API remains functional. All other Spaces functionality is unaffected.

A fix is in progress and we will provide an update as soon as we have more information.

Apr 22, 18:03 UTC
Investigating - Our Engineering team is currently investigating an issue with creating Spaces through the Cloud Control Panel. During this time, users may experience difficulties when selecting the Data Center option, as it is not listing correctly. We apologize for the inconvenience and will provide an update as soon as we have more information.

Apr 6, 01:01 UTC
Resolved - Our Engineering team has resolved the issue affecting Mongo 8 Database operations. Users should be able to create new Mongo 8 Databases, and other operations such as upgrading, scaling, and forking should complete successfully now.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Apr 5, 22:01 UTC
Update - Our Engineering team is actively investigating the issue affecting Mongo 8 Database operations.

We apologize for the continued inconvenience. We will provide additional updates as soon as more information becomes available.

Apr 5, 19:54 UTC
Update - Our Engineering team is continuing to investigate the cause of the issue with Mongo 8 Database creation in our AMS3 region and is taking action to resolve this issue. In addition to the database creation, users may see failures when scaling the instance or performing any control plane operations.

We will continue to share updates as soon as we have new information.

Apr 5, 16:52 UTC
Investigating - Our Engineering team is investigating an issue with Mongo 8 database creation in the AMS3 region. At this time, users may experience errors while attempting to create a new Mongo 8 database instance in the AMS3 region.

We apologize for the inconvenience and will provide another update as soon as possible.

Apr 5, 13:59 UTC
Resolved - Our Engineering team has confirmed the full resolution of the networking issue affecting the LON1 region. Users should be able to access all resources without any issues.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Apr 5, 13:35 UTC
Monitoring - Our Engineering team made routing changes to remediate the networking issue due to an impact by an upstream provider and confirmed that it has successfully mitigated the connectivity issues in the LON1 region. Users should not be seeing any issues with networking in the LON1 region.

We will monitor this incident for a short period to confirm full resolution.

Apr 5, 12:45 UTC
Identified - Our Engineering team has identified the cause and applied a fix to mitigate the connectivity issue impacting the entire LON1 region. Users should be seeing improvements in reaching their resources in the LON1 region.

We'll continue to monitor the situation to confirm this incident is fully resolved and will post an update soon.

Apr 5, 12:17 UTC
Investigating - Our Engineering team is currently investigating an issue impacting networking in the LON1 region. Users may experience network connectivity loss to the resources residing in this region.

We apologize for the inconvenience and will share an update once we have more information.

Apr 3, 23:10 UTC
Resolved - Our Engineering team has implemented a fix to resolve the issue with creating certain GenAI Platform agents. Users should now be able to create OpenAI and Llama 3.1 8B agents without issue.

If you continue to experience any problems, please reach out to our support team by opening a ticket from within your Cloud Control Panel.

Apr 3, 21:06 UTC
Investigating - Our Engineering team is investigating an issue affecting GenAI Platform. The GenAI platform is seeing errors creating agents with certain LLMs, specifically OpenAI and Llama 3.1 8B. The user is shown an agreement checkbox even though they have already accepted the agreement. If they try to accept the agreement, they will be thrown an error and prevented from creating the agent.

We apologize for the inconvenience, we will share an update once we have more information.

Apr 2, 04:46 UTC
Resolved - Our Engineering Team has implemented a fix to resolve the issues impacting GenAI platform agents. They have also confirmed that the issue has been fully resolved. Now, customers should be able to view and update their GenAI platform agents in the console.

If you continue to experience any problems, please reach out to our support team by opening a ticket from within your Cloud Control Panel.

Apr 2, 03:26 UTC
Investigating - Our Engineering Team is currently investigating an issue where some customers are unable to view their GenAI platform agent in the console. As a result, they are also unable to update their agent via the console. We are working to determine which types of agents are affected, including whether newly created agents are impacted.

We apologize for the inconvenience and will provide another update as soon as possible.

Mar 31, 09:30 UTC
Resolved - From 00:13 UTC to 9:09 UTC, users may have experienced errors when attempting to create Managed Database Caching Clusters.

Our Engineering team has confirmed the full resolution of the issue. Users should now be able to create Caching clusters normally.

If you continue to experience issues, please open a support ticket from your Cloud Control Panel.

Mar 31, 07:58 UTC
Monitoring - Our Engineering team has implemented a fix to resolve the issues impacting the creation of Managed Database Clusters.

Users should no longer see any errors while creating a Caching Database Cluster.

We are actively monitoring the situation and will post an update as soon as the issue is fully resolved.

Mar 31, 05:41 UTC
Identified - Our Engineering team has identified the root cause of the issue affecting the creation of Managed Database Clusters and is actively working on a fix.

Users who already have a Caching Database Cluster in their account should be able to create a new Cluster. During this time, users without an existing caching cluster in their account may still see an error while creating a new cluster.

We apologize for the inconvenience and will provide another update as soon as possible.

Mar 31, 04:09 UTC
Investigating - As of 00:13 UTC, our Engineering team is investigating an issue with creating Managed Database clusters.

During this time, users may face issues creating caching clusters.

We apologize for the inconvenience and will share an update once we have more information.

Apr 8, 18:51 UTC
Completed - The scheduled maintenance for our core networking infrastructure in the LON1 region, originally planned for April 8, 2025, from 20:00 UTC to 23:00 UTC, has been cancelled. No changes will be made at this time, and all services will continue to operate as usual.

We will provide an update when this maintenance has a new confirmed date. If you have any questions, please reach out to our support team via Cloud Support.

Thank you for your understanding.

Mar 30, 20:29 UTC
Scheduled - Start: 2025-04-08 20:00 UTC
End: 2025-04-08 23:00 UTC

During the above window, our Networking team will be making changes to the core networking infrastructure to improve performance and scalability in the LON1 region.

Expected impact:

During the maintenance window, users may experience delays or failures with event processing for a brief duration on Droplets and Droplet-based services, including Droplets, Managed Kubernetes, Load Balancers, Container Registry, and App Platform. We will endeavor to keep this to a minimum for the duration of the change.

If you have any questions related to this issue, please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket