Is anyone still using railway after the breach?
I’m just wondering if people still are with them and if they switched. Would love to know where you guys switched if u did
[link] [comments]
I’m just wondering if people still are with them and if they switched. Would love to know where you guys switched if u did
Evaluating CD tools to automate our end to end delivery (build, test, packaging, and deployments). Curious if the standard GHA + Argo/Flux combo is still what most teams stick with, or if there are dedicated delivery/orchestration tools actually worth looking at in 2026.
| Grafana Labs shares how they scaled canary tokens across their Kubernetes workloads, with automated rotation, more precise detection and less maintenance. A practical look at the tradeoffs between coverage, granularity and management overhead. [link] [comments] |
I'm looking for some architectural advice on building a lightweight, self-hosted job execution platform on a single VM (8C/32GB).
I want something with semantics similar to Google Cloud Run / Cloud Run Jobs:
Current constraints:
Technologies I'm considering
Questions
I'm particularly interested in hearing from people who have actually run Nomad in production, especially for single-node deployments.
Thanks!
With the recent high-stakes security exploits, I think as DevOps engineers it's our responsibility to make the local dev environment secure for devs.
What do you think?
What are you actively doing about it (like adopting devcontainers)
I could use some advice on a situation at work.
A few colleagues in ops have, almost overnight, started to fully trust their LLM agent because it has gotten so good lately and "doesn't make mistakes anymore". They built their own harness around it and log what it does (using Grok and Google models), but the agent still runs with root access to our entire production infrastructure. We have no clear team rules or policy yet, and they don't see the danger themselves.
A typical example: whenever the agent needs to check something on a server, they just hand it full access. But looking at info only needs read-only access. The permission should match the task, not the most convenient option.
My concerns:
Logging is detection, not prevention. It shows what broke afterwards, but doesn't stop anything.
Root bypasses the harness. An agent with root can in theory edit the harness, the logs and the backups, so those logs are not a reliable audit trail.
No least privilege. One wrong command or misread context can touch everything.
Prompt injection via logs, tickets or web pages the agent reads.
Model updates. Providers change models without us deciding when, so "no mistakes in recent months" says little about the next version.
Data going to third parties. With root, the agent can read .env files, keys and customer data, which then end up with the model provider (a GDPR question too).
"It has gone fine so far" only means the failure hasn't happened yet.
I don't want to ban anything. I want them to see the risk themselves and set up proper guardrails together with me: a read-only user by default, scoped credentials, just-in-time access, human approval for write actions, staging first, and no access to backups and logs.
Questions for you:
How do I explain the difference between "we log it" and "it can't do it in the first place"?
How have you convinced overconfident colleagues about AI tooling risks without a confrontation?
Do you have horror stories or post-mortems I can share? Concrete incidents tend to work better than theory.
Which guardrails or policy examples have worked well for you?
How do you raise this with management without coming across as "anti-AI"?
Hi,
Im now in the beginning of devops journey and I'm not really a reader person but I want to spend my free time in something useful so any suggestions for a book would help in my career and will benift me.
In case any of you use OpenSSH on musl without PAM:
TL;DR: Accounts that should never take a password at all (root, your key-only deploy user, ldap users, the default user on alpine's aws and azure images) take any password longer than 256 characters.
We use several external APIs and I’m curious how often other teams out there keep performance tests realistic and structured while accounting for third party dependencies. How do you balance realistic dependency behavior with controlled and repeatable tests? Things are going well overall (but they could always be better) and I want to make sure my team and I aren’t missing something that could be beneficial.
Sharing my views , seniors can correct me.
Python
(It will give max advantage i believe, it's programming language as well so I can say I know programming as well, If I want to switch to data science, ai ml or any other field then it can help me, I know basic of it and find it interesting)
Go
(It will help me in understanding Kubernetes in a better way, again programming language so It will be amazing to have skill, don't know much where it will help me but yes, I haven't learned this ever so I will be excited for this)
Shell/bash
(Core skill listed in almost every devops job but again I feel like it will be truly limited to automation and I don't find it very easy to learn, very weirds and long braces, dots and what not )
TLDR; working as a junior in a big team, just doing ops task and not improving enough. how should i spent my free time and navigate my career?
hi all, i know this is probably a recurring discussion, ive tried searching for similar one but i still am not satisfied with the answers.
im a freshgrad with previously 1yoe remote work as devops engineer in a team that only consist of 8 softeng (im the only devops there), honestly its great because i feel like im owning things and can improve things left and right. but, i lack mentorship and feedback there (also low salary lololol) and feel like theres no growth
now, thankfully im in a better and mature company where the devops team itself has more than 50 people. im only a junior here and still in a probation phase, so my day to day are mostly just exploring our services and doing nothing.
what should i do? do i just do thing slowly? i feel like im left behind in this era of AI where people are building things and i just feel like im not enough if all i do is working on ops task and not owning anything. i do learn things on my free time but thats all, feel like its never enough. thanks!
Hey fellow DevOps folks! I've been wrestling with secrets management recently. I set up a tool that generates API keys, but every time I think I've got it right, I find myself second-guessing whether they’re genuinely secure! 😅 There was this one time I accidentally committed sensitive info to Git—ugh, the panic! Luckily, I caught it fast, but it got me thinking: What strategies or tools do you all lean on for managing secrets? Do you prefer something like HashiCorp Vault, or are you more into environment variables? Let’s share some tips and tricks! What’s been your experience with it?
I'm working in a startup. Recently there was a layoff as well. Around 10% was removed.
The thing is this. Within like 8 months, our microservice count is trippled. But the same DevOps team. Even with the layoffs, new developers are hired. Requirements are coming very fast and it's difficult to work on Platform Building, Service Provisioning, Ops work all at once.
We have like unlimited AI. But eventhough it takes time to analyse the requirements, review the designs and implement the solutions.
Currently I'm just asking the AI to raise the PR and just rushing through it and applying the changes.
The problem is that, it's feel like I'm just tired and doing a labour work.
I'm not actively learning anything because of the fast pace and high workload.
Should I look for a new job?
Is other places like same due to the AI?
Update: Thanks redditors. I see that I'm not alone. Seems everywhere is like this.
Just went back to Railway's status page after the Oct 21–23 outage. The main “Railway” uptime chart now says 100.00% for the last 90 days.
Couldn't access the dashboard, couldn't build anything. Apparently that's 100% uptime. Their own incident updates said builds were paused across every region, and Wayback saved the outage announcement.
The October 30 snapshot shows that same “Railway” chart at 97.10%. Now it's 100.00%. The outage is still inside both 90-day windows.
Support & Community used to sit inside that group. Now it's listed separately, still at 97.10%, while “Railway” gets to show a perfect 100%. Very convenient.
The dashboard and API charts were already showing 100% in the old snapshot, despite their own updates saying those services were unavailable. So those numbers never reflected the outage either.
Yes, the incident is still under History. But if I have to dig through incident logs to find the downtime behind your 100% uptime chart, what the hell is the chart for?
I've been speaking to recruiters recently about roles in my area.
Almost every set of interviews now has a live coding portion to it.
I've got an interview next week where I've been told I will be asked to live code. When asked what sort of coding, I was told - it could be writing Terraform, Python, writing a helm chart, kubernetes manifests. It will be random on the day.
I've individually done a lot of these. I've been writing Terraform for years, and I do Kubernetes manifests every single day.
However, I might not touch Terraform for months whilst I work on one project. Python might be even longer.
If tasked with a Terraform project, I'll look at some recent work I did, get a feel for it again, and I'm productive pretty quickly.
But just throw me into an interview, and have multiple engineers ask me to live code, and I feel like I would turn to jelly. I can't remember off the top of my head how to do a module for Terraform - give me a previous one I created, I can take it as a template and start rewriting it, looking up functions if I need new things.
Same for Kubernetes. I will take a deployment file and a svc I did for something in the past, copy and paste it into a new file and edit it to fit my new needs. Writing the document from scratch without documentation... I can't remember that for shit. Who does it afresh on every new service?
We often talk about securing the software supply chain in terms of application dependencies. For instance, pinning package versions, maintaining lockfiles, and auditing third-party libraries.
But what about the code that runs our CI/CD pipelines?
GitHub Actions workflows can execute third-party actions, and the security of those dependencies matters just as much when building and deploying software.
The introduction of workflow dependency lockfiles is an interesting step towards making CI/CD workflows more reproducible and secure.
I wrote about this recently, exploring why workflow lockfiles matter and how they fit into the broader software supply-chain security picture.
Read the article: https://amitmerchant.com/ci-pipeline-lockfile/
Curious how others are approaching dependency security in their CI/CD pipelines. Are you already pinning your actions and enforcing similar controls?
I wish to trim my subscriptions to ones worth thewatching time . Who do you watch for K8s, internal platforms, and lately LLM infra (gateways, routing, cost/observability)?
Deep-dives over hype preferred.
Hello fellow devops engineers,
I would like to improve my networking knowledge do you have some suggestion for any udemy or youtube course to become better at networking.
I am not a noob but i feel i am not good with networking. Some suggestion would be helpful
I’m an ML engineer. Last month I got a Slack from marketing asking why “their dashboard” stopped updating. Nobody on my team knew it existed. It was a Lovable app hitting our Supabase with a service key, deployed to Vercel under someone’s personal account, and the person who built it had left.
That got me wondering how common this is. So, for people on platform/eng/IT teams:
Not selling anything. Trying to understand whether this is a real problem or just my company
Hello. i am Starting to learn IT and i still have time to choose my carreer choice. is it wise to try and learn devops or cloud engineering because of AI? im scared that by the time im done with my degree these jobs wont exist.