With the recent high-stakes security exploits, I think as DevOps engineers it's our responsibility to make the local dev environment secure for devs.
What do you think?
What are you actively doing about it (like adopting devcontainers)
I could use some advice on a situation at work.
A few colleagues in ops have, almost overnight, started to fully trust their LLM agent because it has gotten so good lately and "doesn't make mistakes anymore". They built their own harness around it and log what it does (using Grok and Google models), but the agent still runs with root access to our entire production infrastructure. We have no clear team rules or policy yet, and they don't see the danger themselves.
A typical example: whenever the agent needs to check something on a server, they just hand it full access. But looking at info only needs read-only access. The permission should match the task, not the most convenient option.
My concerns:
Logging is detection, not prevention. It shows what broke afterwards, but doesn't stop anything.
Root bypasses the harness. An agent with root can in theory edit the harness, the logs and the backups, so those logs are not a reliable audit trail.
No least privilege. One wrong command or misread context can touch everything.
Prompt injection via logs, tickets or web pages the agent reads.
Model updates. Providers change models without us deciding when, so "no mistakes in recent months" says little about the next version.
Data going to third parties. With root, the agent can read .env files, keys and customer data, which then end up with the model provider (a GDPR question too).
"It has gone fine so far" only means the failure hasn't happened yet.
I don't want to ban anything. I want them to see the risk themselves and set up proper guardrails together with me: a read-only user by default, scoped credentials, just-in-time access, human approval for write actions, staging first, and no access to backups and logs.
Questions for you:
How do I explain the difference between "we log it" and "it can't do it in the first place"?
How have you convinced overconfident colleagues about AI tooling risks without a confrontation?
Do you have horror stories or post-mortems I can share? Concrete incidents tend to work better than theory.
Which guardrails or policy examples have worked well for you?
How do you raise this with management without coming across as "anti-AI"?
Hi,
Im now in the beginning of devops journey and I'm not really a reader person but I want to spend my free time in something useful so any suggestions for a book would help in my career and will benift me.
In case any of you use OpenSSH on musl without PAM:
TL;DR: Accounts that should never take a password at all (root, your key-only deploy user, ldap users, the default user on alpine's aws and azure images) take any password longer than 256 characters.
We use several external APIs and I’m curious how often other teams out there keep performance tests realistic and structured while accounting for third party dependencies. How do you balance realistic dependency behavior with controlled and repeatable tests? Things are going well overall (but they could always be better) and I want to make sure my team and I aren’t missing something that could be beneficial.
Sharing my views , seniors can correct me.
Python
(It will give max advantage i believe, it's programming language as well so I can say I know programming as well, If I want to switch to data science, ai ml or any other field then it can help me, I know basic of it and find it interesting)
Go
(It will help me in understanding Kubernetes in a better way, again programming language so It will be amazing to have skill, don't know much where it will help me but yes, I haven't learned this ever so I will be excited for this)
Shell/bash
(Core skill listed in almost every devops job but again I feel like it will be truly limited to automation and I don't find it very easy to learn, very weirds and long braces, dots and what not )
TLDR; working as a junior in a big team, just doing ops task and not improving enough. how should i spent my free time and navigate my career?
hi all, i know this is probably a recurring discussion, ive tried searching for similar one but i still am not satisfied with the answers.
im a freshgrad with previously 1yoe remote work as devops engineer in a team that only consist of 8 softeng (im the only devops there), honestly its great because i feel like im owning things and can improve things left and right. but, i lack mentorship and feedback there (also low salary lololol) and feel like theres no growth
now, thankfully im in a better and mature company where the devops team itself has more than 50 people. im only a junior here and still in a probation phase, so my day to day are mostly just exploring our services and doing nothing.
what should i do? do i just do thing slowly? i feel like im left behind in this era of AI where people are building things and i just feel like im not enough if all i do is working on ops task and not owning anything. i do learn things on my free time but thats all, feel like its never enough. thanks!
Hey fellow DevOps folks! I've been wrestling with secrets management recently. I set up a tool that generates API keys, but every time I think I've got it right, I find myself second-guessing whether they’re genuinely secure! 😅 There was this one time I accidentally committed sensitive info to Git—ugh, the panic! Luckily, I caught it fast, but it got me thinking: What strategies or tools do you all lean on for managing secrets? Do you prefer something like HashiCorp Vault, or are you more into environment variables? Let’s share some tips and tricks! What’s been your experience with it?
I'm working in a startup. Recently there was a layoff as well. Around 10% was removed.
The thing is this. Within like 8 months, our microservice count is trippled. But the same DevOps team. Even with the layoffs, new developers are hired. Requirements are coming very fast and it's difficult to work on Platform Building, Service Provisioning, Ops work all at once.
We have like unlimited AI. But eventhough it takes time to analyse the requirements, review the designs and implement the solutions.
Currently I'm just asking the AI to raise the PR and just rushing through it and applying the changes.
The problem is that, it's feel like I'm just tired and doing a labour work.
I'm not actively learning anything because of the fast pace and high workload.
Should I look for a new job?
Is other places like same due to the AI?
Update: Thanks redditors. I see that I'm not alone. Seems everywhere is like this.
Just went back to Railway's status page after the Oct 21–23 outage. The main “Railway” uptime chart now says 100.00% for the last 90 days.
Couldn't access the dashboard, couldn't build anything. Apparently that's 100% uptime. Their own incident updates said builds were paused across every region, and Wayback saved the outage announcement.
The October 30 snapshot shows that same “Railway” chart at 97.10%. Now it's 100.00%. The outage is still inside both 90-day windows.
Support & Community used to sit inside that group. Now it's listed separately, still at 97.10%, while “Railway” gets to show a perfect 100%. Very convenient.
The dashboard and API charts were already showing 100% in the old snapshot, despite their own updates saying those services were unavailable. So those numbers never reflected the outage either.
Yes, the incident is still under History. But if I have to dig through incident logs to find the downtime behind your 100% uptime chart, what the hell is the chart for?
I've been speaking to recruiters recently about roles in my area.
Almost every set of interviews now has a live coding portion to it.
I've got an interview next week where I've been told I will be asked to live code. When asked what sort of coding, I was told - it could be writing Terraform, Python, writing a helm chart, kubernetes manifests. It will be random on the day.
I've individually done a lot of these. I've been writing Terraform for years, and I do Kubernetes manifests every single day.
However, I might not touch Terraform for months whilst I work on one project. Python might be even longer.
If tasked with a Terraform project, I'll look at some recent work I did, get a feel for it again, and I'm productive pretty quickly.
But just throw me into an interview, and have multiple engineers ask me to live code, and I feel like I would turn to jelly. I can't remember off the top of my head how to do a module for Terraform - give me a previous one I created, I can take it as a template and start rewriting it, looking up functions if I need new things.
Same for Kubernetes. I will take a deployment file and a svc I did for something in the past, copy and paste it into a new file and edit it to fit my new needs. Writing the document from scratch without documentation... I can't remember that for shit. Who does it afresh on every new service?
We often talk about securing the software supply chain in terms of application dependencies. For instance, pinning package versions, maintaining lockfiles, and auditing third-party libraries.
But what about the code that runs our CI/CD pipelines?
GitHub Actions workflows can execute third-party actions, and the security of those dependencies matters just as much when building and deploying software.
The introduction of workflow dependency lockfiles is an interesting step towards making CI/CD workflows more reproducible and secure.
I wrote about this recently, exploring why workflow lockfiles matter and how they fit into the broader software supply-chain security picture.
Read the article: https://amitmerchant.com/ci-pipeline-lockfile/
Curious how others are approaching dependency security in their CI/CD pipelines. Are you already pinning your actions and enforcing similar controls?
I wish to trim my subscriptions to ones worth thewatching time . Who do you watch for K8s, internal platforms, and lately LLM infra (gateways, routing, cost/observability)?
Deep-dives over hype preferred.
Hello fellow devops engineers,
I would like to improve my networking knowledge do you have some suggestion for any udemy or youtube course to become better at networking.
I am not a noob but i feel i am not good with networking. Some suggestion would be helpful
I’m an ML engineer. Last month I got a Slack from marketing asking why “their dashboard” stopped updating. Nobody on my team knew it existed. It was a Lovable app hitting our Supabase with a service key, deployed to Vercel under someone’s personal account, and the person who built it had left.
That got me wondering how common this is. So, for people on platform/eng/IT teams:
Not selling anything. Trying to understand whether this is a real problem or just my company
Hello. i am Starting to learn IT and i still have time to choose my carreer choice. is it wise to try and learn devops or cloud engineering because of AI? im scared that by the time im done with my degree these jobs wont exist.
Was reading about the malicious Claude code extension that show up in Cusor marketplace. The pary that got me was it trying to add an SSH key and open a reverse tunnel l, from something that just look like another editor extension
We talk a lot abt checking npm package and dependencies, but IDE extension feal like one of those thing people install once and forget abt it. Especially with auto updates enbled
Those things make me wonder how much access we are giving these extension without really thinking abt it. Are people actually restricting extension on work machines or is it mostly up to each dev? Curious if anyone team changed anything after this 🤔
Source:https://cursor.com/blog/claude-code-extension-security-notic
I manage a lot of repos, that are changed by multiple devs and every few months it turns into a mess: hundreds of merged branches nobody deleted, PRs that went quiet a year ago, repos nobody has touched since a project ended. GitHub only lets me deal with it one repo at a time, so mostly I just... don't.
Curious how others handle this:
- Do you run a script (gh CLI, an Action), use a tool, or just ignore it?
- What's the most annoying part?
- What would you never trust a tool to do automatically?
Especially interested in hearing from people managing 30+ repos or a team org.
I'm about three-quarters of the way migrating my application infrastructure from ArgoCD to Flux, and man, is it difficult. I had to use both Claude and ChatGPT pretty extensively to parse the absolute mountain of error logs I was getting and correct the pile of bugs that I had inadvertently introduced while setting it up the first time around.
I'm too far into this process to back out, but just in case I finish this process and then realize that I actually didn't want to migrate, which platform is actually used more commonly in the industry, so I should actually spend more energy learning it?
Note, even though I do mention that I used two AI models quite a bit, I only used them for parsing logs and deciding what to do. All of the code is actually handwritten.
I've been voluntold to analyse our Datadog costs because our bills keep going up, and we're paying extra for new features if we go over the committed spend. It's a bit mismanaged across the org. Naturally, I need billing data to do the analysis (and do some MI outside of Datadog).
These endpoints all return 403 Forbidden:
Login/api/v2/usage/historical_cost/api/v2/usage/estimated_cost/api/v2/cost_by_tag/monthly_cost_attributionI'm an Admin with all relevant billing and usage permissions. Other usage endpoints work fine, and I can see usage data in the UI.
We buy Datadog through a reseller, and I suspect our organisation is a child org under the reseller's parent org. I think that explains why I can't access historical cost data through the API.
Our reseller has gone quiet for quite some time. I've contacted Datadog Sales and Support (I have no idea why we have a direct connection via a shared Slack channel even though we're not direct customers), but they've basically ghosted me.
We're paying for the service, our bills are increasing, and I'm trying to understand where the money goes. Why is accessing our own cost data such a nightmare?
Has anyone dealt with anything similar?
