Dec 19, 05:03 UTC
Resolved - From 1:23 UTC to 3:04 UTC, users may have experienced issues with events being stuck or delayed, such as powering on/off, and resizing Droplets in the NYC3, AMS2, BLR1, SGP1, and SYD1 regions. Additionally, Managed Database creates were delayed in all regions.

Our Engineering team has confirmed full resolution of the issue, delayed events and new events should complete as normal now.

Thank you for your patience through this issue. If you continue to experience any issues, please open a support ticket from within your account.

Dec 19, 04:28 UTC
Update - Our Engineering team has confirmed that new events are succeeding, and we're finalizing cleaning up any stalled events. We will post an update as soon this issue is fully resolved.

Dec 19, 03:08 UTC
Monitoring - Events in NYC3 has been enabled now and users may continue to submit new events. We are starting to see success rates with event processing go up.

We will post another update soon. Thank you for your patience.

Dec 19, 02:42 UTC
Update - Our Engineering team continues to work to mitigate the issue with event processing in multiple regions.

To help address the issue, we are temporarily disabling all Droplet-related actions in the NYC3 region for a period of 20 minutes. During this time, users in NYC3 will not be able to submit actions, such as creating, resizing, powering on/off, etc.

We apologize for the inconvenience and appreciate your patience as we work to resolve this issue. Further updates will be shared as they become available.

Dec 19, 01:45 UTC
Investigating - Our Engineering team is investigating an issue impacting Droplet event processing in multiple regions.

At this time, users may experience issues with events appearing to be stuck or delayed, such as powering on/off, and resizing Droplets in NYC3, AMS2, BLR1, SGP1, and SYD1 regions. Additionally, Managed Database creates are delayed in all regions.

We apologize for the inconvenience and will share an update once we have more information.

Dec 13, 13:25 UTC
Resolved - From 13:25 to 14:45 UTC, our Engineering team observed a Networking issue in our NYC3 region. During this time, users may have experienced Droplet and VPC connectivity issue, Users should no longer be experiencing these issues.
We apologize for the inconvenience. If you have any questions or continue to experience issues, please reach out via a Support ticket on your account.

Dec 13, 11:01 UTC
Resolved - Our Engineering team has confirmed that the issues with Spaces CDN functionality has been fully resolved. Users should now be able to use CDN functionality normally.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Dec 13, 10:55 UTC
Monitoring - Our Engineering team has implemented a fix to resolve the CDN functionality issue and is currently monitoring the situation.

We will continue to monitor this at our end and will share an update once the issue is resolved completely.

Dec 13, 09:55 UTC
Investigating - Our Engineering team is investigating an issue affecting Spaces CDN functionality globally. During this time, users may experience issues when enabling Spaces CDN and with existing CDN services.

We apologize for the inconvenience and will share an update once we have more information.

Dec 13, 04:05 UTC
Resolved - Our Engineering team has confirmed the full resolution of the issue impacting Managed Database Operations, and all systems are now operating normally. Users may safely resume operations, including upgrades, resizes, forking, and ad-hoc maintenance patches.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Dec 13, 03:13 UTC
Monitoring - Our Engineering team has implemented a fix for the issue impacting Managed Database Operations. The team is monitoring the situation, and we will share an update once this is fully resolved.

Dec 12, 22:03 UTC
Update - Our Engineering team is continuing to work on investigating the root cause of the issue affecting Managed Database Operations. At this time, we are seeing improvement and some new cluster creations are completing successfully.

To avoid potential downtime, we continue to ask users to refrain from performing operations that trigger node rotations, such as upgrades, resizes, forking, and ad-hoc maintenance patches.

We apologize for any inconvenience caused and appreciate your patience. We will post further updates as soon we have more information.

Dec 12, 19:35 UTC
Update - During our investigation, we identified that operations triggering node rotation, such as upgrades, resizes, forking and ad-hoc maintenance patches, may cause issues connecting to the cluster.

To prevent potential downtime, we recommend avoiding these operations until the issue is fully resolved.

We apologize for the inconvenience and appreciate your patience as we work to address the situation.

Dec 12, 18:34 UTC
Investigating - Our Engineering team is investigating an issue with the Managed Database clusters which is causing forks, restores and new creates to fail. The Mongo clusters are unaffected and should be operating normally.

Our team is currently assessing the root cause and working to resolve the issue as quickly as possible.

We apologize for any inconvenience this may cause and will share an update as soon as we have more information.

Dec 13, 01:24 UTC
Resolved - Our Engineering team has confirmed that the issues with Authoritative DNS resolution in NYC1 has been fully resolved. DNS queries should now be resolving normally.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Dec 13, 01:10 UTC
Monitoring - Our Engineering team has implemented a fix to resolve the issues with Authoritative DNS Resolution in NYC1 and is currently monitoring the situation.

We will continue to monitor this at our end and will share an update once the issue is resolved completely.

Dec 12, 23:20 UTC
Investigating - Our Engineering team is investigating an issue with intermittent failures and increased latency in DNS resolution in NYC1.

During this time, a subset of users may experience issues with DNS resolution, or see errors returned when querying DNS records which are hosted on the DigitalOcean authoritative DNS infrastructure.

We apologize for the inconvenience and will share an update once we have more information.

Dec 11, 23:33 UTC
Resolved - Our Engineering team has resolved the issue impacting prohibiting a subset of users from logging in, and the login flow is now operating normally.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Dec 11, 22:53 UTC
Monitoring - Our Engineering team has implemented a fix for the issue prohibiting a subset of users from logging in. The team is monitoring the situation, and we will share an update once this is fully resolved.

Dec 11, 19:33 UTC
Investigating - Our Engineering team is currently investigating an issue prohibiting users from logging in if they are flagged for an additional security challenge.

We are actively reviewing the root cause and working closely to resolve the issue as quickly as possible.

We apologize for the inconvenience and will provide updates as more information becomes available. Thank you for your patience.

Dec 11, 01:30 UTC
Completed - The scheduled maintenance has been completed.

Dec 10, 19:30 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Dec 10, 19:26 UTC
Scheduled - We are reaching out again to inform you that the Network maintenance in AMS3 region which was previously scheduled to start on 2024-12-10 at 12:00 UTC has been rescheduled to the following window:

Start: 2024-12-10 19:30 UTC
End: 2024-12-11 01:00 UTC


We apologize for any inconvenience this short notice causes and thank you for your understanding. You may find the initial maintenance notice along with a description of any expected impact related to this work included at the bottom of this message.

Expected impact:

During the maintenance window users may experience delays or failures with event processing for a brief duration on Droplets and Droplet-based services including Droplets, Managed Kubernetes, Load Balancers, Container Registry, and App Platform. We will endeavor to keep this to a minimum for the duration of the change.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Dec 11, 00:56 UTC
Resolved - From 16:10 UTC to 22:46 UTC, users may have experienced issues while executing Managed Database CRUD Operations.

Our Engineering team has confirmed the full resolution of the issue impacting Managed Database CRUD Operations, and all systems are now operating normally.
Users may safely resume operations, including upgrades, resizes, forking, and ad-hoc maintenance patches.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Dec 10, 22:54 UTC
Monitoring - Our Engineering team has implemented a fix for the issue impacting Managed Database CRUD Operations. The team is monitoring the situation, and we will share an update once this is fully resolved.

Dec 10, 22:44 UTC
Identified - Our Engineering team has identified the cause of the issue that is impacting Managed Database CRUD Operations and is actively working on a fix.

To avoid potential downtime, we continue to ask users to refrain from performing operations that trigger node rotations, such as upgrades, resizes, forking, and ad-hoc maintenance patches.

Existing database clusters remain unaffected as long as no node rotation occurs due to DNS issues, and all other services are functioning as expected.

We apologize for any inconvenience caused and appreciate your patience as we work diligently to address the situation. Further updates will be shared as soon as they become available.

Dec 10, 20:11 UTC
Update - During our investigation, we identified that operations triggering node rotation, such as upgrades, resizes, forking and ad-hoc maintenance patches, may also be impacted. To prevent potential downtime, we recommend avoiding these operations until the issue is fully resolved.

We apologize for the inconvenience and appreciate your patience as we work to address the situation.

Dec 10, 19:22 UTC
Investigating - Our Engineering team is investigating an issue causing Managed Database clusters to take longer than usual to be created. Our team is currently assessing the root cause and working to resolve the issue as quickly as possible.

Existing database clusters are not affected, and all other services are operating normally.

We apologize for any inconvenience this may cause and will share an update as soon as we have more information.

Dec 6, 21:00 UTC
Completed - The scheduled maintenance has been completed.

Dec 6, 19:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Dec 6, 16:50 UTC
Scheduled - Start: 2024-12-06 19:00 UTC
End: 2024-12-06 21:00 UTC

Our Networking team has identified a faulty redundant networking component in our NYC3 region, and are proceeding with emergency replacement during the above window today to ensure redundancy and networking availability in the region.

Expected impact:

As the component under maintenance is part of a redundant group, we do not expect any impact on customer traffic or resources due to this maintenance.

We apologize for the short notice with this maintenance event, and thank you for your understanding. If you have any questions related to this event, please create a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Dec 6, 18:38 UTC
Resolved - Our Engineering team identified an issue affecting App Platform deployments across all regions. The deploy-on-push functionality may not have triggered new builds between 15:45 and 18:15 UTC.

A fix has been implemented, and as of 18:15 UTC, deployments have been functioning successfully. If you continue to experience problems, please open a ticket with our support team. . We apologize for any inconvenience.

Dec 6, 00:50 UTC
Resolved - Our Engineering has confirmed full resolution of the issue. Users should now be able to manage their Reserved IPs via the Cloud Panel without issue.

If you continue to experience any issues, please open a support ticket from within your account. Thank you.

Dec 6, 00:16 UTC
Monitoring - Our Engineering team has deployed a fix for the issue with managing Reserved IPs via the Cloud Panel. Users should now be able to view the Reserved IPs page in their account.

We are currently monitoring the situation and will post an update as soon as the issue is fully resolved.

Dec 5, 23:59 UTC
Investigating - Our Engineering team is investigating an issue with managing Reserved IPs in the Cloud Panel. During this time, users may experience an error message when trying to view their Reserved IPs page at https://cloud.digitalocean.com/networking/reserved_ips. Managing Reserved IPs via the API should still be functioning.

We apologize for the inconvenience and will share an update once we have more information.

Nov 29, 05:35 UTC
Resolved - Our Engineering team has confirmed the full resolution of this issue. If you continue to experience problems, please open a ticket with our support team. Thank you for being so patient, and we apologize for any inconvenience.

Nov 29, 04:24 UTC
Monitoring - Our domain registrar has taken action and the digitaloceanspaces.com domain is now recovering. We have verified the domain is resolving and are seeing error rates dropping for all impacted services. Users should begin seeing services returning to normal operation.

Our Engineering teams are monitoring the situation to ensure complete recovery and we will post a final update soon.

Nov 29, 03:47 UTC
Update - Our Engineering team has identified additional downstream services seeing elevated failure rates and broken functionality. At this time, the following services are impacted:

- Spaces object accessibility in all regions with objects hosted under digitaloceanspaces.com.

- Requests to create, push, or pull to/from Container Registries

- Backups for the Mongo Managed Databases product are not successfully uploading, so users attempting to use those backups will see errors.

- Creates and resizes for Load Balancers in all regions

- New builds for App Platform in all regions

- Create or updating Functions in all regions

We are in active contact with our vendor to resolve this as quickly as possible. Any DNS record under digitaloceanspaces.com will continue to fail until this incident is resolved.

We will continue to provide updates as we have new information.

Nov 29, 03:36 UTC
Update - Our Engineering team has identified additional downstream services seeing elevated failure rates and broken functionality. At this time, the following services are impacted:

- Spaces object accessibility in all regions with objects hosted under digitaloceanspaces.com.

- Requests to create, push, or pull to/from Container Registries

- Backups for the Mongo Managed Databases product are not successfully uploading, so users attempting to use those backups will see errors.

- Creates and resizes for Load Balancers in all regions

- New builds for the App Platform in all regions

We are in active contact with our vendor to resolve this as quickly as possible. Any DNS record under digitaloceanspaces.com will continue to fail until this incident is resolved.

We will continue to provide updates as we have new information.

Nov 29, 03:04 UTC
Update - Spaces accessibility in all regions continues to be impacted and users will experience errors when attempting to interact with objects hosted under digitaloceanspaces.com. Container Registries are also impacted and users will experience errors when attempting to create, push, or pull containers. Backups for the Mongo Managed Databases product are also not successfully uploading, so users attempting to use those backups will see errors.

We are in active contact with our vendor to resolve this as quickly as possible. Any DNS record under digitaloceanspaces.com will continue to fail until this incident is resolved.

We will continue to provide updates as we have new information.

Nov 29, 02:38 UTC
Update - Spaces accessibility in all regions continues to be impacted and users will experience errors when attempting to interact with objects hosted under digitaloceanspaces.com. Backups for the Mongo Managed Databases product are also not successfully uploading, so users attempting to use those backups will see errors.

We have engaged with our upstream vendor to resolve this incident as quickly as possible and are awaiting a further update. Any DNS record under digitaloceanspaces.com will continue to fail until this incident is resolved.

We understand these are critical issues for our customers and are working as quickly as possible to resolve this.

Nov 29, 01:50 UTC
Update - Our Engineering team has successfully deployed a solution to bring the Cloud Control Panel back online. Please refresh the page a few times to clear your cache.

However, we are still experiencing issues with Spaces. We are engaging with our upstream vendor to resolve this incident as quickly as possible. Any DNS record under digitaloceanspaces.com will continue to fail until this incident is resolved. We understand this is a critical issue for our customers and are working as quickly as possible to resolve this.

Nov 29, 01:31 UTC
Identified - Our Engineering team has identified this as a domain registrar issue for the entire digitaloceanspaces.com domain. We are engaging with our upstream vendor to resolve this incident as quickly as possible. Any DNS record under digitaloceanspaces.com will continue to fail until this incident is resolved. We understand this is a critical issue for our customers.

Our Cloud Control Panel is dependent on our Spaces product and is also experiencing an outage due to this issue.

Due to the Cloud Control Panel being down, if users need to submit a support ticket, they may do so here: https://www.digitalocean.com/company/contact/support

Nov 29, 01:10 UTC
Update - Our Engineering team is continuing to investigate this issue and all possible paths to remediation, as well as confirm if this issue is related to an upstream provider outage. We understand this is a critical issue for our customers.

Due to the Cloud Control Panel being down, if users need to submit a support ticket, they may do so here: https://www.digitalocean.com/company/contact/support

Nov 29, 00:34 UTC
Update - We are continuing to investigate this issue.

Nov 29, 00:21 UTC
Investigating - Our Engineering team is currently investigating an issue affecting portions of the Cloud Panel across all regions. Additionally, some Space Buckets may be inaccessible via the CDN. Users may experience difficulties accessing the cloud control panel, encounter DNS-related problems, or face interruptions when managing or accessing Spaces and Spaces CDN through.

We apologize for the inconvenience and will share an update once we have more information.

Nov 25, 22:05 UTC
Resolved - Our Engineering team is investigating an issue related to our ongoing SFO3 maintenance here: https://status.digitalocean.com/incidents/4kj7krrpyg3k

From 20:23 - 20:25 UTC, some services were impacted by a drop in networking. During that time, some Managed Kubernetes clusters experienced errors from the Kubernetes API and/or an increase in 5xx errors. Communication between other services and Block Storage Volumes may have been impacted as well.

The impact has been mitigated and services should be working normally at this time.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Nov 27, 12:00 UTC
Completed - The scheduled maintenance has been completed.

Nov 27, 10:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Nov 25, 10:11 UTC
Scheduled - Start: 2024-11-27 10:00 UTC
End: 2024-11-27 14:00 UTC

During the above window, our Networking team will be performing maintenance on core switches in our SFO3 datacenter as part of network upgrades.

Expected Impact:

These upgrades are designed and tested to be seamless, and we do not expect any impact to customer traffic due to this maintenance. Should an unexpected issue arise, a possible outcome would be temporary loss of connectivity or increased latency for a brief period of time. We will endeavor to keep any such impact to a minimum.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Nov 27, 03:00 UTC
Completed - The scheduled maintenance has been completed.

Nov 27, 01:52 UTC
Update - Our Engineering team has decided to extend the maintenance window by an hour to monitor the situation.

Maintenance is still underway and we will update once the maintenance is fully completed.

We sincerely apologize for any inconvenience this may have caused and appreciate your understanding.

Nov 27, 00:01 UTC
Update - From 22:25 - 23:15 UTC, all event-related actions including creating or deleting resources, were temporarily disabled in NYC3.

Users attempting any event-based operations across products hosted in NYC3 would have encountered error messages. While we anticipated event failures and delays, this impact period was longer than expected. We are taking steps to ensure this does not happen in future maintenances.

As of now, maintenance is still underway and we will update once the maintenance is fully completed.

We sincerely apologize for any inconvenience this may have caused and appreciate your understanding.

Nov 26, 22:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Nov 24, 22:03 UTC
Scheduled - Start: 2024-11-26 22:00 UTC
End: 2024-11-27 02:00 UTC

During the above window, our Networking team will be making changes to core networking infrastructure, to improve performance and scalability in the NYC3 region.

Expected impact:

During the maintenance window users may experience delays or failures with event processing for a brief duration on Droplets and Droplet-based services including Droplets, Managed Kubernetes, Load Balancers, Container Registry, and App Platform. We will endeavor to keep this to a minimum for the duration of the change.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Nov 25, 22:00 UTC
Completed - The scheduled maintenance has been completed.

Nov 25, 16:30 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Nov 23, 16:35 UTC
Scheduled - Start: 2024-11-25 16:30 UTC
End: 2024-11-25 22:00 UTC

During the above window, our Networking team will be making changes to core networking infrastructure, to improve performance and scalability in the SFO3 region.

Expected impact:

During the maintenance window users may experience delays or failures with event processing for a brief duration on Droplets and Droplet-based services including Droplets, Managed Kubernetes, Load Balancers, Container Registry, and App Platform. We will endeavor to keep this to a minimum for the duration of the change.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Nov 22, 17:44 UTC
Resolved - From 09:05 to 09:50 UTC (November 21), our Engineering team observed an issue with Droplet creation.

During this time, users may have experienced intermittent errors while creating the Droplets via the Cloud Panel and API. Users should no longer be experiencing these issues.

We apologize for the inconvenience. If you have any questions or continue to experience issues, please reach out via a Support ticket on your account.

Nov 25, 19:20 UTC
Resolved - Given the absence of outages for the Support Portal, we will now resolve this incident. Our Engineering team will continue to work with our vendor to ensure continued stability of this service. If we observe further outages, we will communicate those to our users via new updates on our status page.

We sincerely apologize and thank you for your patience as we worked through this issue. In case of any questions or concerns, please open a ticket with our Support team.

Nov 25, 15:56 UTC
Update - Our Engineering team continues to work with our vendor to find a solution, but we have not observed any further outages of our Support Portal over the last 48 hours. We will provide additional updates as information becomes available. If an outage does occur, users may still contact us via the form here: https://www.digitalocean.com/company/contact/support

Thank you for your patience. We will post an update as soon as more information becomes available.

Nov 22, 03:49 UTC
Update - Our Engineering team continues to work with our vendor in order to resolve the issue with users accessing the Support Portal. The issue has been escalated and we are awaiting feedback.

As a reminder, if users need to reach out, they can utilize the form here as a workaround:

https://www.digitalocean.com/company/contact/support

Thank you for your patience. We will post an update as soon as further information becomes available.

Nov 21, 15:40 UTC
Investigating - Our Engineering team has observed issues with our Support Portal. During brief, recurring outages, users are unable to access the Support Portal at https://cloudsupport.digitalocean.com/. We are aware of these events and our engineers are working with our vendor to resolve this.

As this issue is intermittent, we expect users will be able to log support tickets normally. However, during any periods of unavailability, as a temporary workaround, users may still contact us via the form here: https://www.digitalocean.com/company/contact/support

Thank you for your patience. We will post an update as soon as further information becomes available.

Nov 20, 01:54 UTC
Resolved - Our Engineering team has confirmed that the issue with displaying users for PostgreSQL Managed Databases across all regions has been fully resolved.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Nov 20, 01:34 UTC
Monitoring - Our Engineering team has rolled out a fix to resolve the issue with displaying users for PostgreSQL Managed Databases across all regions.

Users should no longer see errors while creating new users for PostgreSQL Managed Databases using Cloud Control Panel.

We will continue to monitor this at our end and will share an update once the problem is resolved completely.

Nov 20, 00:50 UTC
Investigating - Our Engineering team is investigating an issue with displaying users for PostgreSQL Managed Databases across all regions.

During this time, users are unable to see existing users in the Cloud Control Panel. Additionally, when creating users in the Cloud Control Panel, an error will be displayed, despite the user being created successfully.

Creating and listing users for PostgreSQL clusters via the API remains functional.

We apologize for the inconvenience and will share an update once we have more information.

Nov 18, 19:33 UTC
Resolved - Our Engineering team has confirmed complete resolution of the issue that was impacting Spaces Object Storage, DigitalOcean Container Registry, and App Platform, across all regions.

From 16:52 UTC - 18:41 UTC, users may have experienced increased error rates when accessing Spaces objects, interacting with the DigitalOcean Container Registry and while creating/deploying Apps with App Platform. Functionality is completely restored and all operations are succeeding normally.

If you continue to experience any issues with these services please submit a ticket to our customer support team for assistance. Thank you for your patience.

Nov 18, 19:07 UTC
Monitoring - Our Engineering team has completed the mitigation efforts and a fix is now deployed in all regions.

As of 18:41 UTC, error rates have returned to normal and users should no longer experience issues accessing, creating, or performing updates on their existing services with the impacted products.

We are monitoring the situation and will provide an update shortly.

Nov 18, 18:32 UTC
Investigating - Our Engineering team is investigating an issue impacting Spaces Object Storage, DigitalOcean Container Registry, and App Platform, across all regions.

Beginning at 16:52 UTC, users may have started experiencing 500 errors when trying to access and perform operations with their Spaces bucket, as well as increased failure rates when pushing to Container Registries and creating/deploying Apps.

Our team has started to apply mitigations to correct the issue, which are rolling out across all regions.

We'll provide another update once that process has completed.