❌

Normal view

Received β€” 1 July 2024 ⏭ DigitalOcean Status - Incident History

SGP1 Ongoing Managed Kubernetes Maintenance 2024-07-01

Jul 2, 19:35 UTC
Completed - The scheduled maintenance has been completed.

Jul 1, 12:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jul 1, 08:50 UTC
Scheduled - Start: 2024-07-01 1200 UTC
End: 2024-07-08 2100 UTC

During the above window, our Engineering team will be performing ongoing security maintenance on Managed Kubernetes control plane clusters in our SGP1 region.

Expected impact:

While our team will take all possible precautions to prevent impact during this event, some Managed Kubernetes clusters in the region may experience brief periods where control plane connectivity is lost. In such an event, Managed Kubernetes control plane operations such as scaling, resizing, adding nodes and creation of new clusters may be delayed. We will endeavor to keep any such impact to a minimum.

If you have any questions related to this event please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Thank you,
Team DigitalOcean

Received β€” 29 June 2024 ⏭ DigitalOcean Status - Incident History

Spaces in NYC3

Jun 28, 21:57 UTC
Resolved - Our Engineering team has confirmed that this incident has been fully resolved.

We appreciate your patience throughout this process and if you continue to experience problems, please open a ticket with our support team for further review.

Jun 28, 21:18 UTC
Monitoring - Our Engineering team has taken actions to mitigate the issue affecting intermittent request failures for Spaces in NYC3 and is monitoring the situation.

The impact has subsided and users should be able to interact with Spaces normally.

We will post an update once we confirm this incident is fully resolved.

Jun 28, 20:37 UTC
Update - Our Engineering team is continuing to investigate the root cause of the issue with intermittent request failures for Spaces in NYC3. Observed error rates are trending down to pre-incident levels, but may spike again.

We will provide an update as soon as we have additional information.

Jun 28, 19:04 UTC
Investigating - Beginning around 17:40 UTC, Our Engineering team is investigating an issue impacting Spaces in our NYC3 region. During this time, users may encounter intermittent request failures for Spaces in NYC3. Additionally, load times for objects in that region may be slower.

We apologize for any inconvenience caused and will provide further updates as soon as possible.

Received β€” 27 June 2024 ⏭ DigitalOcean Status - Incident History

Events for Droplets with Reserved IP's in Multiple Regions

Jun 27, 23:57 UTC
Resolved - Our Engineering team has confirmed that this incident has been fully resolved.

Thank you for your patience. If you continue to experience any problems, please open a ticket with our support team for further review.

Jun 27, 23:22 UTC
Monitoring - Our Engineering team has confirmed that the issue for Droplets with previously failed events has been remediated, and further events should now successfully complete.

We are continuing to monitor this situation and will post an update as soon as the issue is fully resolved.

Jun 27, 21:12 UTC
Update - Our Engineering team has completed deployment of the fix. No additional users should now experience event failures.

Droplets that experienced failed events are now being remediated to fix their event state, so that future events can succeed normally.

We will provide another update once the remediation step is complete.

Jun 27, 20:54 UTC
Identified - Beginning around 16:00 UTC, our Engineering team identified an uptick in event errors for Droplets that utilize a Reserved IP in multiple regions. A subset of users are experiencing errors in processing events, especially "power on", via both API and the Cloud Control Panel.

Our team has identified the root cause of the issue and a fix is currently in progress.

We will provide another update once the fix is confirmed.

Received β€” 26 June 2024 ⏭ DigitalOcean Status - Incident History

Managed Kubernetes Cluster Creation

Jun 26, 14:10 UTC
Resolved - Our Engineering team has confirmed the full resolution of the issue impacting the Managed Kubernetes cluster creation. As of 12:18 UTC the functionality has been restored completely and users should be able to deploy new clusters via Cloud Control Panel and API.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Jun 26, 13:36 UTC
Monitoring - Our Engineering team has deployed a fix for the issue impacting the Managed Kubernetes cluster creation across all regions. The impact has been mitigated and users should no longer experience any issues deploying new clusters via Cloud Control panel and API.

We are monitoring the situation and will post an update once the incident is completely resolved.

Jun 26, 12:05 UTC
Investigating - Our Engineering team is investigating an issue impacting our Managed Kubernetes service in all regions. Beginning 11:10 UTC, users may have experienced errors while creating new clusters using Control panel and API.

We apologize for the inconvenience and will share an update once we have more information.

Received β€” 24 June 2024 ⏭ DigitalOcean Status - Incident History

Rescheduled: NYC1 Network Maintenance 2024-06-27 10:00 UTC

Jun 27, 12:00 UTC
Completed - The scheduled maintenance has been completed.

Jun 27, 10:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jun 24, 10:27 UTC
Scheduled - Start: 2024-06-27 10:00 UTC
End: 2024-06-27 12:00 UTC

During the above window, our Networking team will be performing maintenance on core switches in our NYC1 datacenter as part of network upgrades. This is a rescheduled maintenance event previously scheduled for 2024-06-11 10:00 UTC.

Expected Impact:

We anticipate brief interruptions in network traffic from Droplets and Droplet-based services. This impact would be for a duration of 5 to 10 seconds and could occur a few times within the maintenance window. We will endeavor to keep any such impact to a minimum.

We apologize for any inconvenience this short notice causes and thank you for your understanding. If you have any questions related to this issue please send us a ticket from your cloud support page using https://cloudsupport.digitalocean.com/s/createticket


Thank you,
Team DigitalOcean

BLR1 Network Maintenance

Jun 25, 16:30 UTC
Completed - The scheduled maintenance has been completed.

Jun 25, 12:30 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jun 23, 15:21 UTC
Scheduled - Start: 2024-06-25 12:30 UTC
End: 2024-06-25 16:30 UTC

During the above window, our Networking team will be making changes to core networking infrastructure, to improve performance and scalability in the BLR1 region.

Expected impact:

These upgrades are designed and tested to be seamless and we do not expect any impact to customer traffic due to this maintenance. If an unexpected issue arises, affected Droplets and Droplet-based services may experience increased latency or a brief disruption in network traffic. We will endeavor to keep any such impact to a minimum.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Received β€” 19 June 2024 ⏭ DigitalOcean Status - Incident History

Monitoring Alerts Creation

Jun 19, 23:04 UTC
Resolved - Our engineering team has resolved the issue with creating a monitoring alert in the Monitoring section of the Cloud Control Panel.

Thank you for your patience. If you continue to experience any problems, please open a support ticket from within your account.

Jun 19, 22:22 UTC
Monitoring - Our Engineering team has implemented a fix regarding the issue with creating a monitoring alert in the Monitoring section of the Cloud Control Panel. We are monitoring the situation to ensure there is no recurrence.

We will post another update once we confirm the issue is fully resolved.

Jun 19, 19:25 UTC
Investigating - Our Engineering team is investigating an issue impacting our Monitoring Alerts service.

During this time, users may have been experiencing errors or unexpected delays while trying to create monitoring alerts via Cloud Control Panel.

We apologize for the inconvenience and will post an update as soon as we have additional information.

Received β€” 18 June 2024 ⏭ DigitalOcean Status - Incident History

Network Connectivity in SFO region

Jun 18, 13:15 UTC
Resolved - As of 13.10 UTC, our Engineering team has confirmed full resolution of networking in our SFO region.

If you continue to experience problems, please open a ticket with our support team from your Cloud Control Panel. Thank you for your patience and we apologize for any inconvenience.

Jun 18, 12:03 UTC
Monitoring - Our Engineering team has confirmed that the fix implemented earlier was successful in mitigating the issue with connectivity in our SFO regions. Users should now be able to connect to their Droplet and Droplet-based services without any issues.

At this time, all services should now be operating normally. We will monitor this incident for a short period of time to confirm full resolution.

Jun 18, 10:25 UTC
Identified - As of 09:40 UTC, our Engineering team has identified the cause and applied a fix to mitigate the connectivity issue impacting the entire SFO region. We are still looking into this failure, but users should be seeing improvements in reaching their Droplet and Droplet-based services.

We'll continue to monitor the situation to confirm this incident is fully resolved and will post an update soon.

Jun 18, 09:05 UTC
Update - Our Engineering team is continuing to investigate the issue with connectivity in our SFO regions. During this time, users may encounter errors and connection timeouts for Droplet and Droplet-based resources in the SFO region. Our team is diligently working on identifying and mitigating the issue at the earliest.

We regret any inconvenience caused and will post an update as additional information is available.

Jun 18, 08:38 UTC
Investigating - Our Engineering team is investigating an issue with connectivity in our SFO regions. During this time users may experience connection timeouts and errors for Droplet and Droplet-based resources.

We apologize for the inconvenience and will share an update once we have more information.

Received β€” 17 June 2024 ⏭ DigitalOcean Status - Incident History

503 errors in SGP1 region

Jun 17, 19:37 UTC
Resolved - From 17:02 - 18:44 UTC, the Spaces API experienced availability dips. These dips caused Spaces bucket operations in SGP1 for a subset of users to experience latency or 503 errors.

The fix implemented by our Engineering team has been monitored and availability for the Spaces API is stable. Users should be able to interact with their Spaces buckets successfully.

If you continue to experience any issues, please reach out to the Support team from within your account.

Jun 17, 19:07 UTC
Monitoring - Our Engineering team has identified and implemented a fix for an issue impacting Spaces SGP1 region. We are now monitoring the situation closely and will post an update as soon as the issue is fully resolved. Thank you for your patience!

Jun 17, 18:49 UTC
Investigating - Our Engineering team is investigating an issue with spaces in SGP1 region. During this time, users may experience 503 errors while accessing bucket operations. We apologize for the inconvenience and will share an update once we have more information.

NYC3 Network Maintenance 2024-06-18 22:00 UTC Phase 2

Jun 19, 02:00 UTC
Completed - The scheduled maintenance has been completed.

Jun 18, 22:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jun 16, 18:24 UTC
Scheduled - Start: 2024-06-18 22:00 UTC
End: 2024-06-19 02:00 UTC

During the above window, our Networking team will be making changes to our core networking infrastructure to improve performance and scalability in the NYC3 region. This will be the second of the two maintenance activities performed by our team in the region on consecutive days.

Expected Impact:

These upgrades are designed and tested to be seamless and we do not expect any impact to customer traffic due to this maintenance. If an unexpected issue arises, affected Droplets and Droplet-based services may experience a temporary loss of private connectivity between VPCs. We will endeavor to keep any such impact to a minimum.

If you have any questions or concerns regarding this maintenance, please reach out to us by opening up a ticket on your account via https://cloudsupport.digitalocean.com/s/createticket.

Received β€” 16 June 2024 ⏭ DigitalOcean Status - Incident History

NYC3 Network Maintenance 2024-06-17 22:00 UTC Phase 1

Jun 18, 02:00 UTC
Completed - The scheduled maintenance has been completed.

Jun 17, 22:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jun 15, 12:44 UTC
Scheduled - Start: 2024-06-17 22:00 UTC
End: 2024-06-18 02:00 UTC

During the above window, our Networking team will be making changes to our core networking infrastructure to improve performance and scalability in the NYC3 region. This maintenance will occur in two parts on consecutive days and we will send another maintenance notice for the second phase.

Expected Impact:

These upgrades are designed and tested to be seamless and we do not expect any impact to customer traffic due to this maintenance. If an unexpected issue arises, affected Droplets and Droplet-based services may experience a temporary loss of private connectivity between VPCs. We will endeavor to keep any such impact to a minimum.

If you have any questions or concerns regarding this maintenance, please reach out to us by opening up a ticket on your account via https://cloudsupport.digitalocean.com/s/createticket.

Received β€” 14 June 2024 ⏭ DigitalOcean Status - Incident History

Managed Databases Control Plane

Jun 15, 02:56 UTC
Resolved - Our Engineering team has confirmed that this incident has been fully resolved.

We appreciate your patience throughout this process and if you continue to experience problems, please open a ticket with our support team for further review.

Jun 15, 00:11 UTC
Monitoring - Our Engineering team has implemented a fix to fully resolve the issue with our Managed Databases services. At this time, users should no longer face issues creating, resizing, forking, or updating trusted sources for PostgreSQL, MySQL, Redis, and Kafka clusters, in any region, as well as Mongo clusters in NYC regions.

We are monitoring the situation closely and will post an update as soon as we confirm the issue is fully resolved.

Jun 14, 22:48 UTC
Update - Our Engineering team has been able to partially mitigate impact from this incident. Creating, resizing, forking, and updates to trusted sources of PostgreSQL, MySQL, Redis, and Kafka clusters, in all regions, is now working as expected. Operations previously attempted on clusters, such as firewall rule attempts, will need to be resubmitted or have a relevant event (i.e. k8s node changes, tag events, etc.) occur from now on. Users should start to see recovery and be able to initiate new operations normally.

Operations for Mongo clusters remain affected at this time.

Jun 14, 19:22 UTC
Update - Our Engineering team continues work to address the root cause of this issue and remediate customer impact.

We apologize for the continued interruption to the Managed Databases Control Plane and will provide another update as soon as we have more information.

Jun 14, 17:39 UTC
Identified - During the course of investigation, our Engineering team has identified additional operations are impacted by this incident.

Users may experience errors with creating, resizing, or updating trusted sources for PostgreSQL, MySQL, Redis, and Kafka clusters, in all regions, as well as Mongo clusters in NYC regions.

Due to updates to trusted sources being impacted, connections to Database clusters from newly added trusted sources will fail. This includes new Managed Kubernetes nodes, Droplets, and Apps using Databases.

Our team is working to deploy a fix and we will provide another update soon.

Jun 14, 17:27 UTC
Investigating - Our Engineering team is investigating an issue impacting new Managed Database creations. During this time, users may receive failures while creating new Managed Database clusters across all regions. We apologize for the inconvenience and will share an update once we have more information.

Received β€” 13 June 2024 ⏭ DigitalOcean Status - Incident History

Cloud Control Panel Page Permissions

Jun 12, 20:23 UTC
Resolved - Our Engineering team identified and resolved an issue impacting multiple pages in the Cloud Control Panel.

From 16:12 - 16:26 UTC, users may have experienced access denied errors while loading pages associated with different services on the Cloud Control Panel. The impacted pages included Managed Database, Block Storage Volumes, Reserved IPs, Domains, Settings, App Platform, Applications & API, Container Registry, Functions, Managed Kubernetes and Monitoring.

Our Engineering team was able to take quick action to mitigate the impact and resolve the issue.

Thank you for your patience, and we apologize for any inconvenience. If you continue to experience any issues, please open a Support ticket for further analysis.

Received β€” 12 June 2024 ⏭ DigitalOcean Status - Incident History

Cloud Control Panel

Jun 12, 19:40 UTC
Resolved - Our Engineering team identified and resolved an issue impacting multiple pages in the Cloud Control Panel.

From 16:12 - 16:26 UTC, users may have experienced access denied errors while loading pages associated with different services on the Cloud Control Panel. The impacted pages included Managed Database, Block Storage Volumes, Reserved IPs, Domains, Settings, App Platform, Applications & API, Container Registry, Functions, Managed Kubernetes and Monitoring.

Our Engineering team was able to take quick action to mitigate the impact and resolve the issue.

Thank you for your patience, and we apologize for any inconvenience. If you continue to experience any issues, please open a Support ticket for further analysis.

New User Sign Ups

Jun 12, 16:41 UTC
Resolved - From 13:45 - 16:04 UTC, users were unable to sign up for new DigitalOcean accounts at https://cloud.digitalocean.com and existing users were unable to update credit card information within the Cloud Control Panel, due to an issue with a billing verification key. This issue is now fully resolved and users should be able to interact with the Cloud Control Panel normally.

We apologize for the inconvenience and invite users to retry any failed attempts. If you continue to experience issues or have any questions, please reach out to Support from within your account.

Jun 12, 16:07 UTC
Monitoring - Our Engineering team has completed implementation of the fix to resolve the issue with new user sign ups and attempts to update credit card details. At this time, services should be functioning as expected. If you were attempting to sign up or change your existing credit card information, please retry.

We're monitoring the situation and will post a final update once we confirm this is fully resolved.

Jun 12, 15:29 UTC
Identified - Our Engineering team has identified the root cause of the issue with new user sign ups and is working to implement a fix. During the course of investigation, the team also identified that existing users are unable to update their existing credit card information.

We will post another update once the fix has been implemented.

Jun 12, 15:15 UTC
Investigating - Our Engineering team is investigating an issue with users attempting to sign up for new cloud.digitalocean.com accounts. At this time, users attempting to sign up are unable to do so, due to an issue with finalizing credit card information. Existing users are unaffected and should be able to sign in normally.

We apologize for the inconvenience and will share an update once we have more information.

Received β€” 11 June 2024 ⏭ DigitalOcean Status - Incident History

Load Balancers associated with Managed Kubernetes Clusters and App Platform

Jun 11, 09:59 UTC
Resolved - Our Engineering team has identified and resolved an issue that impacted the Load Balancers associated with Kubernetes Clusters for a brief duration.

From 05:25 - 07:54 UTC, a subset of users may have experienced issues with accessing and connecting to Load Balancers associated with Kubernetes Clusters and may have noticed failed health checks for the Load Balancers.

Also from 7:50 - 8:40 UTC, our Engineering team observed an issue with the App Platform. A subset of users may have faced issues with connecting to outbound dependencies from the App platform.

Swift action was taken by our Engineering team that restored service and all services are operating correctly.

We apologize for the inconvenience. If you have any questions or continue to experience issues, please reach out via a Support ticket on your account.

Received β€” 10 June 2024 ⏭ DigitalOcean Status - Incident History

NYC1 Network Maintenance

Jun 10, 18:51 UTC
Completed - NYC1 Network Maintenance previously scheduled at 2024-06-11 will be rescheduling it for a later date. We will notify you via email and a new maintenance post on status.digitalocean.com with the updated date and timings for this maintenance.

We apologize for any inconvenience and thank you for your understanding. You may find the initial maintenance notice along with a description of any expected impact related to this work included at the bottom of this message.

If you have questions or concerns about this maintenance, please reach out to us by opening up a ticket on your account.

Jun 9, 14:32 UTC
Scheduled - Start: 2024-06-11 10:00 UTC
End: 2024-06-11 12:00 UTC

During the above window, our Networking team will be performing maintenance on core switches in our NYC1 datacenter as part of network upgrades.

Expected Impact:

We anticipate brief interruptions in network traffic from Droplets and Droplet-based services. This impact would be for a duration of 5 to 10 seconds and could occur a few times within the maintenance window. We will endeavor to keep any such impact to a minimum.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Received β€” 8 June 2024 ⏭ DigitalOcean Status - Incident History

SFO Network Maintenance

Jun 14, 14:00 UTC
Completed - The scheduled maintenance has been completed.

Jun 14, 12:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jun 7, 14:04 UTC
Scheduled - Start: 2024-06-14 12:00 UTC
End: 2024-06-14 14:00 UTC

During the above window, our Networking team will be making changes to core networking infrastructure, to improve performance and scalability in the SFO1, SFO2, and SFO3 regions.

Expected Impact:

Our team will be closely monitoring and we do not expect any disruption to customer workloads.

If an unexpected issue arises, affected Droplets and Droplet-based services in the SFO[1-3] regions could experience increased latency or disruption in public network traffic for a short period of time. If an unexpected issue does occur, our team is prepared to rollback to a safe state to quickly restore public networking. We expect that traffic would be impacted for a period of 5-10 minutes at most in that case.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Received β€” 7 June 2024 ⏭ DigitalOcean Status - Incident History

SFO2 Power Feed Maintenance

Jun 7, 19:11 UTC
Completed - Our datacenter provider has informed us that they are not able to perform this maintenance during the communicated window and will be rescheduling it for a later date. We apologize for any confusion from the previous update, but this maintenance did not begin and there was no impact to any services. We will notify you via email and a new maintenance post on status.digitalocean.com with the updated date and timings for this maintenance.

If you have any questions or continue to experience issues, please reach out via a Support ticket on your account.

Jun 7, 11:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jun 7, 02:44 UTC
Scheduled - Start: 2024-06-07 11:00 UTC
End: 2024-06-07 21:00 UTC

During the above window, our data center team will be performing maintenance on redundant upstream electrical systems in our SFO2 region.

Expected impact:

We don’t expect to see any impact to customer resources as the electrical systems are redundant and our team will isolate one unit at a time during the maintenance. In the highly unlikely event that we lose power to both systems simultaneously, affected Droplets and Droplet-based services would experience downtime for a brief period of time. We will endeavor to keep this to a minimum should that situation arise.

If you have any questions or concerns related to this maintenance, please send us a ticket from your cloud support page https://cloudsupport.digitalocean.com/s/createticket.

Received β€” 5 June 2024 ⏭ DigitalOcean Status - Incident History

Cloud Control Panel and API

Jun 5, 15:30 UTC
Resolved - From 12:10 - 12:41 UTC our Engineering team identified and resolved an issue impacting the Cloud Control Panel and API across multiple regions.

During this time, users may have experienced trouble while viewing the IP address on their newly deployed droplets. Our Engineering team was able to take quick action to mitigate the impact and resolve the issue. All services are now functioning as expected.

Thank you for your patience, and we apologize for any inconvenience. If you continue to experience any issues, please open a Support ticket for further analysis.

❌