upvote

points

by Thaxll16 hours ago |
comments
upvoteby rubiquity16 hours ago|
next
[-]
etcd has had numerous liveness and safety bugs, with one happening as recently as December of 2025. Would you consider that a correct implementation?

You may be interested in knowing that the largest managed Kubernetes service in the world (AWS EKS) ripped out etcd for in favor of their homegrown consensus service for large scale EKS clusters: https://aws.amazon.com/blogs/containers/under-the-hood-amazo...

reply
upvoteby fl0ki15 hours ago|
parent|
next
[-]
etcd is some of the most amateur code I've ever seen, despite being one of the oldest and presumably most mature "infrastructure" projects written in Go.

goBGP is arguably even worse.

I don't have a third place in mind that's even worth mentioning relative to these two.

reply
upvoteby alfons_foobar13 hours ago|
parent|
[-]
just curious, what problems do you see with gobgp?

(I have only a rather basic familiarity with go, but was considering gobgp for an infra project...)

reply
upvoteby iscoelho9 hours ago|
parent|
[-]
Not speaking to their code, but to start GoBGP has the worst performance of any BGP daemon by a large margin [1].

[1] https://elegantnetwork.github.io/posts/comparing-open-source...

reply
upvoteby preisschild3 hours ago|
parent|
[-]
Garbage collected languages like Go will always have worse performance than lower level languages like C (frr and bird are implemented in C).

Gobgp is great if you want to embed it directly into a Go app though. Talos Linux has done that recently.

reply
upvoteby tsss23 minutes ago|
parent|
[-]
No they won't. They are generally faster in throughput than any non-gc application that isn't heavily hand optimized. Their problems are higher memory usage and unpredictable latency, not speed.
reply
upvoteby camkego15 hours ago|
parent|
prev|
next
[-]
Sorry to pile on, but yeah, I wanted to use etcd during 2021 and 2022, around v3.5, but etcd had serious issues including silent data corruption. If you are curious, ask gemini flash "there were a number of etcd releases years ago where it seems a new wave of developers came in and started breaking everything"
reply
upvoteby Thaxll15 hours ago|
parent|
prev|
[-]
I'd like to know what you base your statement on that the Raft implementations in etcd or CockroachDB are incorrect. Your original paper does not mention those implementations, so where does that claim come from?
reply
upvoteby gyesxnuibh11 hours ago|
parent|
[-]
Having run a fleet of 100s of etcd clusters for 10000s of rps, and the fact that upstream runs tests similar to antithesis and recently partnered with antithesis [0], and jepsen has tested it long ago as well [1]. Etcd's raft algorithm is fine. Someone even did a TLA+ proof on it in the last couple years[2]. Yes there was a correctness issue a few years ago but otherwise the person you're replying to doesn't know what they're talking about. Also those bugs have nothing to do with the raft implementation, but instead the state machine implemented on top.

0: https://etcd.io/blog/2025/autonomus_testing_with_antithesis/

1: https://jepsen.io/analyses/etcd-3.4.3

2: https://github.com/etcd-io/raft/pull/113

reply
upvoteby dnautics1 hours ago|
parent|
next
[-]
doesnt raft have a problem that it assumes no hysteresis? and that in general you can construct a latency graph that deterministically causes a permanent lock in the leadership election phase?
reply
upvoteby cobbzilla9 hours ago|
parent|
prev|
next
[-]
Is the correctness of its implementation of the algorithm unaffected by bugs in this state machine? Maybe I missed something.
reply
upvoteby gyesxnuibh8 hours ago|
parent|
[-]
The raft algorithm works and if you implemented a less complex state machine (like using a simpler kv store that doesn't need global event ordering via revisions and watches) it would work. That's what antithesis said they did to test the raft algorithms in the other article linked
reply
upvoteby iscoelho8 hours ago|
parent|
prev|
[-]
"there was a correction issue" is downplaying it. Etcd is truly the worst example of Raft.

Etcd corruption and loss of quorum is extremely common in practice and the GitHub issues sit for years. The design is simple, the performance is modest, yet it still has still never been reliable, despite being marketed as so. I can't speak to whether this is specifically due to their Raft implementation, but I'd argue the entire codebase is over-engineered and questionable.

reply
upvoteby AlphaSite7 hours ago|
parent|
next
[-]
My beef with etcd is that its neither performant nor reliable.

Its very much {reliable, performant, flexible} pick none.

reply
upvoteby gyesxnuibh8 hours ago|
parent|
prev|
[-]
Their lock, leader election, sessions, and leases are all awful and I'd never recommend anyone to use those. But as a strongly consistent kv store and if you need the watch mechanics, its useful. It has its place and that's mostly being used by kubernetes.
reply
upvoteby ablob11 hours ago|
prev|
next
[-]
Surely the King is doing it, so that must be the correct way. Look, the King even wears clothes and is totally not naked at all.

That the world runs on Kubernetes is no qualitative statement about the correctness of its Raft implementation. You can say that it's clearly good enough to not matter most of the time, but that is a different statement. No matter who you look at, they're just cooking with gas like you do, and they can make mistakes in just the same way.

Now; I'm only attacking your argument. I do neither know nor particularly care about the correctness of that implementation itself. There's been better refutations of the claim you replied to in other answers anyway.

reply
upvoteby steve19777 hours ago|
parent|
[-]
It's not even a qualitative statement about Kubernetes.
reply
upvoteby otabdeveloper42 hours ago|
parent|
[-]
Kubernetes doesn't solve any technical problem, so the language it's written in is irrelevant.
reply
upvoteby Yasuraka1 hours ago|
parent|
next
[-]
I have a bunch of volumes that I'd like to get automatically attached and mounted to nodes on which their respective workload runs (which are automatically scheduled) who automatically fetch and mount their config files and secrets from a HA DB on demand. I also need some internal loadbalancing and integrations with something like certbot for all of my web workloads. Id also like to make sure that I get metrics and logs from every workload in some form.

Thats basically it for starters, what non-technical solution do you propose?

reply
upvoteby otabdeveloper41 hours ago|
parent|
[-]
If k8s works for you, then go for it.

But k8s was invented so that "dev" and "ops" teams could play nicely when they are different org structures, with their own budgets, deadlines, etc.

If it solves some technical issue then that's incidental.

reply
upvoteby steve19771 hours ago|
parent|
prev|
[-]
What I meant is that the fact that the world is running on Kubernetes is not a qualitative statement about Kubernetes.
reply
upvoteby literalAardvark5 hours ago|
prev|
next
[-]
etcd is notoriously unreliable and one of the biggest problems in k8s.

I didn't know Go just isn't a good language for it, but now that I know I'm no longer surprised at etcd being problematic.

reply
upvoteby Manfrednotfunny26 minutes ago|
parent|
next
[-]
I'm running big k8s cluster and my small ones at home for 8 years, never had an etcd issue.

Could you elaborate?

reply
upvoteby preisschild3 hours ago|
parent|
prev|
[-]
Having maintained multiple etcd clusters for self-managed kubernetes the last few years I disagree with the "problematic" characterization.

Sure it may not be the best fit in a scenario where you want a cluster spanned over the entire globe (thats why GKE uses paxos-based Spanner instead of it) , but even spanned across an entire continent (in europe via glass fiber) it works quite well for me. Its one of the least problematic parts of the stack.

reply
upvoteby formerly_proven15 hours ago|
prev|
next
[-]
Now there's three of them https://github.com/hashicorp/raft
reply
upvoteby aksss16 hours ago|
prev|
[-]
That's not remotely what he's saying at all.
reply
upvoteby volkk16 hours ago|
parent|
next
[-]
> As another example, Go still has not yielded a correct implementation of Raft or Paxos

> are you saying this implementation is wrong?

> That's not remotely what he's saying at all.

I'm v confused by this thread

reply
upvoteby quietbritishjim16 hours ago|
parent|
prev|
next
[-]
I don't care for Go myself (especially its concurrency model, which is a total dinosaur in a world where we have structured concurrency) so I'm not saying this to support my favourite language, but:

That is literally what the comment says.

reply
upvoteby rubiquity16 hours ago|
parent|
prev|
[-]
Thank you. I don't know why this is so complicated.
reply