❌

Normal view

Binance hack and a DevOps laptop

Binance hack and a DevOps laptop

Binance's initial investigation suggests that the security incident may involve a compromised DevOps engineer’s laptop, which allowed an attacker to gain access to a service container and submit unauthorized withdrawal transactions. We have found no evidence of any wallet private key leak. We will share a detailed technical post-mortem once the investigation is complete and the facts are fully verified.

The estimated total impact sits at around $6.3B. We have worked with blockchain partners to freeze affected assets, with ~$3 billion frozen on-chain so far.

Binance will cover all losses from the SAFU fund and the company’s funds. User funds are safe. We will also publish an updated Proof of Reserves once the remediation is complete.

As an immediate precaution, Binance has:

- temporarily suspended deposits and withdrawals while security checks are completed;

- revoked the affected address authorisation;

- engaged law enforcement and blockchain intelligence partners to trace and recover funds.

Our teams are working around the clock to resolve this issue safely and thoroughly. We are working closely with our law enforcement partners to confirm the identity of these criminals and bring them to justice.

Please rely only on official Binance channels for updates. Please feel free to contact Binance Customer Support if you have any further questions.

submitted by /u/sinunmango to r/devops
[link] [comments]

Dealing with Colleague's AI-generated Code Reviews

As the title says, a new colleague recently joined our team, and he uses AI for everything. Not a single Slack message of his comes from his fingertips unfiltered.

The real pain point has been PR reviews. Our team is small, so reviews are hard to come by. I expected that having another teammate would make things smoother since we could share the review load, but his reviews don't add value, and they actually slow down our development cycle more than before.

I like to break my changes into small, digestible chunks that are easy to review. The problem is that he runs Opus 5.5 on xhigh against every change, and it nitpicks things that aren't even in the PR. It could be a two-line documentation change, and he'll comment that some existing code elsewhere in the same file, outside the scope of the change, needs to be modified. He then marks the pull request as "Changes requested" and doesn't re-review it for days on end.

How would you go about discussing this kindly with him?

submitted by /u/RoseSec_ to r/devops
[link] [comments]

I landed a "devops" role as a junior

30K~ DAU, I have full ownership over an infra under AWS/multiple on premise servers.

we use ECS, terraform, instead of ansible I use a deploy.sh that i give to my boxes with terraform, no graceful deploys. we have our own orchestrator (K8S like) for stateful containers that we have to spin up and down very often (median 1 min of uptime per container) with unsafe code coming from AI/users.

I took the place of someone who left that company and explained me nothing, we are 2 devs, nobody understands how this entire infra works expect me. have been adding observability a lot using cloudwatch, multiple alerts so I page myself at the right times, it's stable but we need to scale to at least 5x (an objective the hierarchy gave me). it currently costs 3.5k/month in aws/on premise servers. btw it's not a classical rest api where you run stateless apis and that's it.

I just discovered how redis worked under a distributed system like this, it's quite good :) I have been a postgresql guy but redis has been a saver for our threat model.

I often read here that you should put everything : EKS + K8S, ansible, terraform, make HA, grafana etc.. but since I manage this entire infra I have been resilient to all of this. If I refactor everything right now they have nobody to do the backend (which is also what I do here) -> "startup" role where one guy has multiple roles.

for the terraform it's all AI, I have been doing a lot of modifications and have created a terraform plan under ci cd so I can review myself, I removed a lot of "destruction" like the database, i versioned the s3, versioned the terraform state, put it to a private s3 etc...

there have been big big security issues (xss, csrf, cookie stealing, untrusted code that owns the entire box, shared passwords, untrusted s3 location so you could overwrite other users...) that were there before I got here

nothing rly interesting maybe for you guys in this experience but I have learned a lot, it has been a crescendo experience where i'm getting to understand more and more which tools to put there because I am the one who owns it 100%, and have to also do features (backend) so not much time either.

submitted by /u/CounterStrike17 to r/devops
[link] [comments]

How do you contain your agents' accesses?

Hey !

I guess most of us have a lot of admin accesses to tools like AWS, k8s clusters, Datadog, GitHub etc.

How do you contain your agents to read-only or curated permissions? Without restraining your own accesses.

I do have some sort of hook as a guard + a setup on pi.dev that copy my config and extract the read-only accesses only to do so, but it's never completely opaque.

submitted by /u/Juloblairot to r/devops
[link] [comments]

Kestrix: CI/CD pipelines defined in HCL outside the repositories, with DAG-based triggers

Kestrix: CI/CD pipelines defined in HCL outside the repositories, with DAG-based triggers

Hey r/devops,

I built Kestrix (https://gitlab.com/kestrix/kestrix) for myself, to keep my DevOps projects separate from my coding projects. It is a repo-agnostic CI/CD control plane written in Go. Repositories are inputs, and the pipeline is defined separately in HCL (pipeline.tf) instead of in a .gitlab-ci.yml or workflow file inside each repository.

What a pipeline looks like

project "platform_services" { default_image = "golang:1.25-alpine" ci_repo "core" { path = "services/core" } ci_repo "api" { path = "api/v1" } ci_task "unit_tests" { trigger = "repo.core.changed || repo.api.changed" execute { repo = "core" script = "go test ./..." } } ci_task "build_container" { trigger = "repo.core.changed" depends_on = ["unit_tests"] execute { repo = "core" script = "docker build -t app:latest ." } } ci_task "deploy_staging" { trigger = "repo.core.changed" depends_on = ["build_container"] execute { repo = "core" script = "tofu apply -auto-approve" } } ci_task "gate_production" { trigger = "repo.core.changed" manual = true depends_on = ["deploy_staging"] execute { repo = "core" script = "echo approved" } } } 

I checked this example with kestrix validate.

How it behaves

  • Trigger resolution: Kestrix parses dependencies, rejects cycles, and evaluates each trigger against the repositories that changed. If only api changed, unit_tests runs and the tasks triggered only by core are skipped.
  • Single-task reruns: If one integration test dies on a network timeout, you rerun that node from the UI and only that task executes.
  • Manual gates: manual = true pauses a task until an operator approves it.
  • Remote runners: They make only outbound HTTPS requests, so they need no inbound ports. In Docker mode, containers run with --cap-drop ALL, --security-opt no-new-privileges, and a PID limit (default 256).
  • Terraform / OpenTofu: terraform-provider-kestrix manages Kestrix projects (kestrix_project).

License

The source is public under PolyForm Noncommercial 1.0.0. It is free for non-commercial use, and commercial use needs a paid plan (from $3/month). It is not an OSI open-source license.

Limits

One serve process per database is supported. Multi-instance coordination is not implemented.

Do you think this will catch on? Am I the only one who prefers to keep DevOps projects separate from strictly software development projects?

submitted by /u/Draqqun to r/devops
[link] [comments]
❌