upvote
They probably asked an AI to run the tests against their DB. And the AI said "yeah I ran it, all green!" (except with a lot more words and perfect grammar, and maybe a few section headings and emoji.)
reply
Seems to me that “we ran the Jepsen test” is just something an LLM thought would sound good on a DB’s website.
reply
Assuming the LLMs did actually run the tests, I think this points to the flaw in TDD and LLMs following goals blindly.

Tests are usually not comprehensive. The tests for SQL engines for example aren't intended to provide full code coverage, because they are intended to be re-usable across different codebases. LLMs can therefore easily over-fit to those tests, solving problems in the dumbest ways rather than solving them in the ways humans would.

The human's goal is to produce a working database engine, the LLM's goal is to pass the test suite or implement the spec it has been given.

reply