❌

Normal view

Received β€” 15 October 2024 ⏭ DigitalOcean Status - Incident History

Internal Service Maintenance

15 October 2024 at 12:09

Oct 15, 17:09 UTC
Completed - The scheduled maintenance has been completed.

Oct 15, 15:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Oct 14, 23:10 UTC
Scheduled - Start Time: October 15 15:00 UTC
End Time: October 15 18:00 UTC

During this time, our Engineering Team will be performing maintenance on an internal service, which necessitates a period of read-only access to that service.

Engineering teams have taken steps to ensure their services do not experience downtime during the read-only period. Due to dependencies on this internal service, there is a possibility that some customer-facing services may experience minor disruptions, particularly while using Marketplace Add Ons or processing App Platform builds (small delays).

Existing infrastructure will continue running without issue and no public or private networking disruption will occur.

If other customer-facing services experience any disruption, we will update this post.

If you have any questions or concerns, please reach out to the Support team from within your account.

Received β€” 14 October 2024 ⏭ DigitalOcean Status - Incident History

Core Infrastructure Maintenance in NYC3

15 October 2024 at 16:00

Oct 15, 21:00 UTC
Completed - The scheduled maintenance has been completed.

Oct 15, 20:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Oct 13, 20:17 UTC
Scheduled - During the above window, our Engineering team will be performing maintenance on principal infrastructure in order to improve reliability of the services. Please note that existing infrastructure will continue running without issue. This maintenance may impact create, read, update, and delete (CRUD) operations in all regions.

Expected Impact:

During the maintenance window, users may experience increased latency for the following platform operations:

Cloud Control Panel and API operations
Event processing
Droplet creates, resizes, rebuilds, and power events
Managed Kubernetes reconciliation and scaling
Load Balancer operations
Container Registry operations
App Platform operations
Managed Database creation and scaling

We do not expect any impact to customer traffic due to this maintenance. If an unexpected issue arises, we will endeavor to keep any impact to minimum and may revert if required.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Received β€” 12 October 2024 ⏭ DigitalOcean Status - Incident History

Spaces and Container Registry in SYD1 region

11 October 2024 at 10:50

Oct 11, 15:50 UTC
Resolved - From 23:00 UTC on October 10th until 13:28 UTC on October 11th, users may have encountered intermittent errors when accessing Spaces endpoints or using the Container Registry in the SYD1 region. Our Engineering team has successfully resolved the issue affecting the functionality of Spaces and the Container Registry.

All services have been fully restored and are now functioning normally. If you continue to experience any problems, please open a ticket with our support team. We apologize for any inconvenience caused.

Oct 11, 14:10 UTC
Monitoring - Our Engineering team has implemented a fix to resolve the issue with Spaces and the Container Registry in the SYD1 region and is monitoring the situation closely. All services have been restored and are functioning as expected. We will post an update as soon as the issue is fully resolved.

Oct 11, 13:18 UTC
Investigating - Our Engineering team is currently investigating an issue with Spaces and the Container Registry in the SYD1 region. During this time, users may experience intermittent errors when accessing Spaces endpoints or interacting with the Container Registry. We apologize for the inconvenience and will provide an update as soon as we have more information.

Received β€” 9 October 2024 ⏭ DigitalOcean Status - Incident History

AMS3 Power Maintenance 2024-10-10 19:00 UTC

11 October 2024 at 00:00

Oct 11, 05:00 UTC
Completed - The scheduled maintenance has been completed.

Oct 10, 19:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Oct 8, 15:27 UTC
Scheduled - Start: 2024-10-10 19:00 UTC
End: 2024-10-11 05:00 UTC

Hello,

During the above window, our Engineering team will be performing maintenance on redundant upstream electrical systems in order to improve reliability in our AMS3 region.

Expected Impact:

Power systems are redundant and we don’t expect to see any downtime, however, in the very unlikely situation that we lose power to both supplies, users may experience disruption with the following services or for a brief period:
Networking
Backups
Snapshots
Droplet create & restore operations

We will endeavor to keep any downtime to a minimum for the duration of the change.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Thank you,
Team DigitalOcean

Received β€” 7 October 2024 ⏭ DigitalOcean Status - Incident History

Core Infrastructure Maintenance in NYC3

Oct 8, 21:00 UTC
Completed - The scheduled maintenance has been completed.

Oct 8, 20:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Oct 6, 17:19 UTC
Scheduled - Start: 2024-10-08 20:00 UTC
End: 2024-10-08 21:00 UTC

During the above window, our Engineering team will be performing maintenance on principal infrastructure in order to improve reliability of the services. Please note that existing infrastructure will continue running without issue. This maintenance may impact create, read, update, and delete (CRUD) operations in all regions.

Expected Impact:

During the maintenance window, users may experience increased latency for the following platform operations:

Cloud Control Panel and API operations
Event processing
Droplet creates, resizes, rebuilds, and power events
Managed Kubernetes reconciliation and scaling
Load Balancer operations
Container Registry operations
App Platform operations
Managed Database creation and scaling

We do not expect any impact to customer traffic due to this maintenance. If an unexpected issue arises, we will endeavor to keep any impact to minimum and may revert if required.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

SGP1 Network Maintenance

Oct 8, 21:01 UTC
Completed - The scheduled maintenance has been completed.

Oct 8, 16:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Oct 6, 17:25 UTC
Scheduled - Start: 2024-10-08 16:00 UTC
End: 2024-10-08 21:00 UTC

During the above window, our Engineering team will be performing maintenance in our SGP1 region as part of network upgrades.

Expected Impact:

These upgrades are designed and tested to be seamless, and we expect minimal impact on customer traffic during the maintenance. There could be a brief disruption in inbound and outbound connectivity, lasting approximately 60 seconds, for the affected Droplets and Droplet-based services in the region. We will make every effort to minimize any such impact

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Received β€” 3 October 2024 ⏭ DigitalOcean Status - Incident History

Creating or Regenerating Spaces Access Key

Oct 3, 19:05 UTC
Resolved - Our Engineering team has confirmed the resolution of the issue impacting Spaces Access Keys.

From 15:21 UTC to 17:52 UTC, users were experiencing errors when creating or regenerating Spaces Access Keys in the Cloud Control Panel.

We apologize for the inconvenience. If you have any questions or continue to experience issues, please reach out via a Support ticket on your account.

Oct 3, 18:08 UTC
Monitoring - Our Engineering team has identified the cause of the issue with Spaces Access Keys and has implemented a fix. Users should now be able to create or regenerate Spaces Access Keys without any issues.

We are currently monitoring the situation and will post an update as soon as the issue is fully resolved.

Oct 3, 17:52 UTC
Investigating - Our Engineering team is investigating an issue with creating or regenerating Spaces Access Keys in the Cloud Control Panel. During this time, users may experience errors when attempting to create or regenerate Spaces Access Keys in the Cloud Control Panel. We apologize for the inconvenience and will share an update once we have more information.

Received β€” 1 October 2024 ⏭ DigitalOcean Status - Incident History

Rescheduled: BLR1 Network Maintenance

Oct 3, 14:54 UTC
Completed - The scheduled maintenance has been completed.

Oct 3, 12:30 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Oct 1, 03:57 UTC
Scheduled - Start: 2024-10-03 12:30 UTC
End: 2024-10-03 15:30 UTC

During the above window, our Networking team will be making changes to core networking infrastructure to improve performance and scalability in the BLR1 region. This is a rescheduled maintenance event previously scheduled for 2024-09-24.

Expected Impact:

These upgrades are designed and tested to be seamless and we do not expect any impact on customer traffic due to this maintenance. Should an unexpected issue arise, a possible outcome would be a temporary loss of connectivity for impacted Droplets. Any impact would also extend to Droplet-dependent services in the BLR1 region such as Managed Databases, Load Balancers, App Platform, and Managed Kubernetes. We will endeavor to keep any such impact to a minimum.

If you have any questions or concerns regarding this maintenance, please reach out to us by opening up a ticket on your account via https://cloudsupport.digitalocean.com/s/createticket/.

Received β€” 28 September 2024 ⏭ DigitalOcean Status - Incident History

SFO3 Network maintenance

Oct 1, 12:00 UTC
Completed - The scheduled maintenance has been completed.

Oct 1, 08:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Sep 28, 08:19 UTC
Scheduled - The Network maintenance in SFO3 region which was previously scheduled to start on 2024-09-25 at 08:00 UTC has been rescheduled to the following window:

Start: 2024-10-01 08:00 UTC
End: 2024-10-01 12:00 UTC

During the above window, our Networking team will be performing maintenance on core switches in our SFO3 datacenter as part of network upgrades.

Expected Impact:

These upgrades are designed and tested to be seamless, and we do not expect any impact to customer traffic due to this maintenance. Should an unexpected issue arise, a possible outcome would be temporary loss of connectivity for impacted Droplets. We will endeavor to keep any such impact to a minimum.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Thank you,
Team DigitalOcean

Networking in NYC region

28 September 2024 at 06:09

Sep 28, 11:09 UTC
Resolved - From 07:45 to 08:55 UTC, users may have experienced networking connectivity issues in our NYC region.

Our Engineering team has confirmed the full resolution of the issue. Users should be able to access all resources as normal.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Sep 28, 10:25 UTC
Monitoring - Our Engineering team made changes to remediate the networking issue due to our upstream provider. Users should not be seeing any issues with networking in NYC region.

We are currently monitoring the situation and will post an update as soon as the issue is fully resolved.

Sep 28, 08:43 UTC
Identified - Our Engineering team has identified the cause of the issue impacting networking connectivity in NYC to be with our upstream provider. Our team is actively working on remediation steps. We will post an update as soon as we have more information.

Sep 28, 08:29 UTC
Investigating - As of 09:45 UTC, our Engineering team is currently investigating an issue impacting networking in NYC region. Users may experience network connectivity loss to Droplets and Droplet-based services, like Managed Kubernetes and Database Clusters.

We apologize for the inconvenience and will share an update once we have more information.

New Customer Sign Ups

27 September 2024 at 13:16

Sep 27, 18:16 UTC
Resolved - As of 17:05 UTC, our Engineering team has resolved the issue affecting new account sign-ups. Users should no longer experience errors and are now able to complete the sign-up process successfully.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Sep 27, 17:23 UTC
Monitoring - Our Engineering team has implemented a fix to resolve the issue affecting new account sign-ups. Users should no longer encounter errors and can now successfully sign up again.

Our Engineering team is continuing to monitor the situation, and we will provide an update as soon as the issue is fully resolved.

Sep 27, 16:41 UTC
Investigating - Our Engineering team is currently investigating an issue affecting registering for a new account at cloud.digitalocean.com. Since 00:00 UTC, some users may encounter errors and be unable to complete the registration process.

We apologize for any inconvenience this may cause and will provide an update as soon as we have more information.

Received β€” 27 September 2024 ⏭ DigitalOcean Status - Incident History

Control Plane in SYD1

26 September 2024 at 15:43

Sep 26, 20:43 UTC
Resolved - From 18:22 UTC to 19:13 UTC, users may have experienced issues or errors when attempting to create or modify DigitalOcean services deployed in the SYD1 region and also when attempting to create or manage Volumes globally.

Our Engineering team has confirmed the full resolution of this issue. If you continue to experience problems, please open a ticket with our support team. Thank you for being so patient, and we apologize for any inconvenience.

Sep 26, 19:33 UTC
Monitoring - Our Engineering team identified the issue and took quick action to mitigate the impact to the SYD1 control plane. Customers may have also encountered errors when attempting to create or manage Volumes globally during this time.

Currently, users should no longer experience issues or errors when attempting to create or modify DigitalOcean services deployed in the SYD1 region.

We are monitoring the situation closely and will post an update as soon as the issue is fully resolved.

Sep 26, 19:08 UTC
Investigating - As of 18:24 UTC, our Engineering team is investigating an issue impacting the control plane in the SYD1 region. At this time customers may encounter errors when attempting to create or modify DigitalOcean services deployed in the SYD1 region. Connectivity to existing services should not be impacted.

This is believed to be related to earlier network maintenance being carried out in the region: https://status.digitalocean.com/incidents/1sf8jl4s1sp4

We apologize for the inconvenience and will share an update once we have further information.

Received β€” 26 September 2024 ⏭ DigitalOcean Status - Incident History

Network Connectivity in SFO3 Region

25 September 2024 at 21:45

Sep 26, 02:45 UTC
Resolved - From 22:25 to 22:50 UTC, users may have experienced network connectivity issues with all DigitalOcean services deployed in the SFO3 region. Additionally, from 22:25 until 01:40 UTC, users may have encountered network connectivity issues affecting their Managed Kubernetes clusters in the SFO3 region.

Our Engineering team has fully resolved the issues impacting connectivity in the region, and all services are now operating normally.

If you continue to experience any problems, please open a ticket with our Support team. We apologize for any inconvenience caused.

Sep 26, 02:15 UTC
Monitoring - Our Engineering team has implemented a fix to resolve the issue affecting networking connectivity with Managed Kubernetes clusters in the SFO3 region and is closely monitoring the situation. We will provide an update as soon as the issue is fully resolved.

Sep 26, 01:27 UTC
Identified - Our Engineering team has identified the cause of the networking connectivity issues with Managed Kubernetes clusters in the SFO3 region and is actively working on rolling out a fix. Once we have additional information, we will share another update.

Sep 26, 00:38 UTC
Update - We are continuing to investigate the network connectivity issue with Managed Kubernetes clusters.

During this investigation, we have also identified an earlier period of network connectivity disruption affecting all services in the region, from 22:25 to 22:50 UTC.

At this time, some Managed Kubernetes clusters may continue experiencing network connectivity issues, but all other services in the region should be accessible as normal.

Sep 25, 23:34 UTC
Investigating - Our Engineering team is currently investigating issues with network connectivity for Managed Kubernetes clusters in the SFO3 region. During this time, some users may experience difficulties connecting to their Managed Kubernetes clusters, or connecting to other services from their clusters.

We apologize for the inconvenience and will provide an update as soon as we have more information.

Droplet Event Processing and API Availability

25 September 2024 at 17:59

Sep 25, 22:59 UTC
Resolved - Our Engineering team has identified and resolved an issue that impacted the ability to resize Droplets via both the API and UI from 18:15 until 21:55 UTC. During this time, users might have experienced errors when attempting to resize their Droplets through the API or the UI.

Additionally, in an effort to resolve the issue with resizes, a secondary issue affected all event processing and some API calls for Droplets and related services from 21:50 until 22:00 UTC.

Swift action was taken by our Engineering team to restore full functionality, and now everything is operating normally.

We apologize for any inconvenience this may have caused. If you have any questions or continue to experience issues, please reach out via a Support ticket on your account.

Resizing MongoDB Managed Databases

25 September 2024 at 11:10

Sep 25, 16:10 UTC
Resolved - From 00:01 to 15:15 UTC, users may have experienced errors when attempting to resize their MongoDB Managed Database clusters.

Our Engineering team has confirmed the full resolution of issue. Users should now be able to resize their MongoDB clusters normally.

If you continue to experience problems, please open a ticket with our support team from within your Cloud Control Panel.

Sep 25, 15:36 UTC
Monitoring - Our Engineering team has implemented a fix to resolve the issues impacting the resizing of Managed Database Clusters. Users should now be able to resize their existing MongoDB clusters without any issues.

We are actively monitoring the situation and will post an update as soon as the issue is fully resolved.

Sep 25, 15:14 UTC
Investigating - Our Engineering team is investigating an issue with MongoDB Managed Databases. At this time, users may experience errors while attempting to resize their existing MongoDB clusters.

We apologize for the inconvenience and will provide another update as soon as possible.

Received β€” 25 September 2024 ⏭ DigitalOcean Status - Incident History

SYD1 Network Maintenance

26 September 2024 at 14:00

Sep 26, 19:00 UTC
Completed - The scheduled maintenance has been completed.

Sep 26, 16:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Sep 24, 16:08 UTC
Scheduled - Start: 2024-09-26 16:00 UTC
End: 2024-09-26 19:00 UTC

During the above window, we will be performing maintenance in our SYD1 region as part of network upgrades.

Expected Impact:

These upgrades are designed and tested to be seamless, and we expect minimal impact on customer traffic during the maintenance. There could be a brief disruption in inbound and outbound connectivity, lasting approximately 30 seconds, for the affected Droplets and Droplet-based services in the region. We will make every effort to minimize any such impact

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Thank you,
Team DigitalOcean

Received β€” 22 September 2024 ⏭ DigitalOcean Status - Incident History

SFO3 Network Maintenance

25 September 2024 at 06:07

Sep 25, 11:07 UTC
Completed - The scheduled maintenance has been completed.

Sep 25, 08:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Sep 22, 08:52 UTC
Scheduled - Start: 2024-09-25 08:00 UTC
End: 2024-09-25 12:00 UTC

During the above window, our Networking team will be performing maintenance on core switches in our SFO3 datacenter as part of network upgrades.

Expected Impact:

These upgrades are designed and tested to be seamless, and we do not expect any impact to customer traffic due to this maintenance. Should an unexpected issue arise, a possible outcome would be temporary loss of connectivity for impacted Droplets. We will endeavor to keep any such impact to a minimum.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Thank you,
Team DigitalOcean

Received β€” 21 September 2024 ⏭ DigitalOcean Status - Incident History

BLR1 Network Maintenance

24 September 2024 at 10:30

Sep 24, 15:30 UTC
Completed - Our Engineering team has run into some operations that didn't complete within this maintenance. No user impact has occurred as a result of those operations.

We will be rescheduling this maintenance window to Oct 3rd 12.30 UTC via a new maintenance post.

Sep 24, 12:30 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Sep 21, 13:24 UTC
Scheduled - Start: 2024-09-24 12:30 UTC
End: 2024-09-24 15:30 UTC

During the above window, our Networking team will be making changes to our core networking infrastructure to improve performance and scalability in the BLR1 region.

Expected Impact:

These upgrades are designed and tested to be seamless and we do not expect any impact on customer traffic due to this maintenance. Should an unexpected issue arise, a possible outcome would be a temporary loss of connectivity for impacted Droplets. Any impact would also extend to Droplet-dependent services in the BLR1 region such as Managed Databases, Load Balancers, App Platform, and Managed Kubernetes. We will endeavor to keep any such impact to a minimum.

If you have any questions or concerns regarding this maintenance, please reach out to us by opening up a ticket on your account via https://cloudsupport.digitalocean.com/s/createticket/.

Received β€” 20 September 2024 ⏭ DigitalOcean Status - Incident History

DigitalOcean Support Portal Maintenance

22 September 2024 at 06:00

Sep 22, 11:00 UTC
Completed - The scheduled maintenance has been completed.

Sep 22, 08:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Sep 20, 06:47 UTC
Scheduled - Start: 2024-09-22 08:00 UTC
End: 2024-09-22 11:00 UTC

During the above maintenance window, there will be maintenance performed on our ticketing system.

Expected Impact:

During the course of the maintenance, users will be unable to use our Live Agent/Chat system. Users can still submit support tickets, update existing tickets, or receive replies to existing tickets.

As soon as the vendor maintenance has completed, the chat functionality should work as expected.

We appreciate your patience throughout this process. If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

App Platform Connectivity

19 September 2024 at 17:48

Sep 19, 22:48 UTC
Resolved - Our Engineering team has confirmed the full resolution of this incident.

From 20:14 to 22:03 UTC, users may have experienced errors when connecting to their Apps or with their Apps connecting to other services, as well as delays in deploying Apps. Users should now be able to connect to their Apps normally, and any delayed deployments should complete successfully.

If you continue to experience problems, please open a ticket with our support team from within your Cloud Control Panel.

Sep 19, 22:22 UTC
Monitoring - Our Engineering team has implemented a fix to resolve the App Platform connectivity and deployment issues in all regions. Users should now be able to connect to and deploy Apps as normal. Our team is monitoring the situation, and we will post an update as soon as the issue is fully resolved.

Sep 19, 22:03 UTC
Identified - Our Engineering teams have identified the cause of connectivity and deploy issues in App Platform.

Our teams are actively deploying a fix across all regions, however, users may still experience connection and deployment errors with Apps until that process is complete. We will continue to post updates as we have more information. Thank you for your patience.

Sep 19, 21:53 UTC
Update - Our Engineering teams continue to investigate the issue impacting App Platform. At this time, our teams believe that Apps in all regions may be impacted by connectivity issues, as well as users seeing delays in Apps deployments.

We have engaged multiple teams to investigate the issue, and we apologize for the continued inconvenience. We will share more updates as soon as we have more information.

Sep 19, 21:24 UTC
Update - Our Engineering team is seeing App connectivity improvement in NYC, although we have confirmed that the LON region is affected as well.

Our team is actively working to identify and fix the issue impacting connectivity for TOR and LON Apps, and we will post an update as soon as we have more information.

Sep 19, 21:07 UTC
Investigating - Our Engineering team is currently investigating an issue potentially impacting DigitalOcean services.

At this time, we're seeing reports of App Platform Apps unable to connect to other DigitalOcean resources, such as Spaces buckets and Managed Databases. Users may see connection errors or be unable to access Apps correctly.

We are working to understand the issue that is occurring, as well as what services and regions are impacted. We will provide another update as soon as we have further information.

❌