Different workflows should probably go in different buckets or "topics" for clarity. Since it's distributed, the system must guarantee that the log items are stored in the same ordering ("offsets") among the nodes.
Not a bad way to do things.
One reason why a "logs are all you need" solution may fail: untrusted-log-as-injection[1].
Check those SBOM, and don't forget to include their CICD pipelines[2].
[1] https://news.ycombinator.com/item?id=48315440
[2] https://github.com/jqwik-team/jqwik/issues/708#issuecomment-...
A distributed WAL (to survive a machine death) would also probably be something I'd want, and … something I'm not sure you're getting directly from SQLite.
I am joining a new project and need to know to what extent Kafka is still a part of the future for new big data projects. It doesn't seem like there are alternatives at the high end but instead the question is when other technologies (that are easier to manage, require less compute, etc.) max out.
"Sockets are all you need for durable workflows" and then finally "Kernel primitives are all you need for durable workflows."
But seriously, part of being a professional is using the right tool for the job.