upvote

points

by andersmurphy1 days ago |
comments
upvoteby zbentley14 hours ago|
next
[-]
There's more process-based concurrency than you'd expect in shops that use those languages.

Cron jobs might need to coordinate with webservers. Even heavily threaded webservers might have some subprocesses/forking to manage connection pools and hot reloads and whatnot. Suid programs are process-separated from non-suid programs. Plenty of places are in the "permanent middle" of a migration from e.g. Java 7 to Java 11 and migrate by splitting traffic to multiple copies of the same app running on different versions of the runtime.

If you're heavily using SQLite for your DB already, you probably are reluctant to replace those situations with multiple servers coordinating around a central DB.

Nit:

> languages that only have process based concurrency python/JS/TS/ruby

Not true. There are tons and tons of threaded Python web frameworks/server harnesses, and there were even before GIL-removal efforts started. Just because gunicorn/multiprocessing are popular doesn't mean there aren't loads of huge deployments running threads (and not suffering for it much, because most web stacks are IO bound). Ruby's similar, though threads are less heavily-used than in Python. JS/TS as well: https://nodejs.org/api/worker_threads.html

reply
upvoteby russellthehippo19 hours ago|
prev|
next
[-]
I actually hadn’t thought about it this way. The killer app I was imagining was 1ms reactivity without SQL polling and messaging atomic with business commits, plus “one db” and no daemon.

But this is actually a great main benefit as well.

reply
upvoteby infogulch1 days ago|
prev|
[-]
He mentions Litestream, maybe this also works for litestream read-only replicas which may be in completely different locations?
reply
upvoteby russellthehippo19 hours ago|
parent|
[-]
Whoa I really hadn’t considered this. Do a litestream read replica, trigger across machines with S3 as the broker essentially. But you’re still stuck with the litestream sync interval. Maybe interesting for cross server notify?
reply
upvoteby infogulch19 hours ago|
parent|
[-]
I guess the idea is to have all writes go through a central server with local read replicas for improved read perf. The default litestream sync interval is 1s. I bet many use-cases would be satisfied with a few seconds delay for cross-region notifications.
reply
upvoteby russellthehippo17 hours ago|
parent|
[-]
It's good for pubsub but not for claim/ack workflow unless you do If-None-Match CAS semantics on a separate filesystem which, actually, yeah that's probably fine. Feels heavy on S3 ops. But! you do save on inter-AZ networking, the Warpstream hypothesis.
reply
upvoteby ncruces17 hours ago|
parent|
[-]
Claims kill this, IMO.

Unless you have a single "reader", you don't mind the delay, and don't worry about redoing a bunch of notifications after a crash (and so, can delay claims significantly), concurrency will kill this.

reply
upvoteby vrajat6 hours ago|
parent|
next
[-]
I wrote a simple queue implementation after reading the Turbopuffer blog on queues on S3. In my implementation, I wrote complete sqlite files to S3 on every enqueue/dequeue/act. it used the previous E-Tag for Compare-And-Set.

The experiment and back-of-the-envelope calculations show that it can only support ~ 5 jobs/sec. The only major factor to increase throughput is to increase the size of group commits.

I dont think shipping CDC instead of whole sqlite files will change the calculations as the number of writes mattered in this experiment.

So yes, the number of writes (min. of 3) can support very low throughputs.

reply
upvoteby russellthehippo17 hours ago|
parent|
prev|
[-]
exactly. then you're building distributed locking and it's probably time for a different tool
reply