See: autocodebench
https://github.com/Tencent-Hunyuan/AutoCodeBenchmark/tree/ma...
Specific studies, as the one quoted, are a long way from original real world problems.
LLMs absolutely understand and write good Elixir. I've done complex OTP and distributed work in tandem with Sonnet/Opus and they understand it well and happily keep up. All the Elixir constructs distinct from ruby are well applied: pipes, multiple function clauses, pattern matching, etc.
I can say that anecdotally, CC/Codex are significantly more accurate and faster working with our 250K lines of Elixir than our 25K lines of JS (though not typescript).
I'd layer in a few more
* Largely stable and unchanged language through out its whole existance
* Authorship is largely senior engineers so the code you train on is high quality
* Relatively low number of abstractions in comparisson to other languages. Meaning there's less ways to do one thing.
* Functional Programming style pushes down hidden state, which lowers the complexity when understanding how a slice of a system works, and the likelyhood you introduce a bug
(And Elixir's relationship to Ruby is pretty overstated, IMO. There's definitely inspiration, but the OO-FP jump is a makes the differences pretty extreme)
> Elixir's relationship to Ruby is pretty overstated
Perhaps I am actually am over thinking this. Elixir has probably diverged enough from Ruby (e.g. defmodule, pipe operators, :atom syntax) for LLMs to notice the difference between the two. But it does open the question, though, how does an LLM actually recognise the difference in code blocks in its training data.
There are probably many more programming languages where similarities exist.
That surprises me :)
From my time doing Ruby (admittedly a few years back), I found libraries were very well documented and tested. But put into context of then (not now), documentation and testing weren't that popular amongst other programming languages. Ruby was definitely one of the drivers for the general adaption of TDD principles, for example.
I used to frequently find myself reading the source code of popular libraries or prying into them at runtime. There's also no central place or format for documentation in ruby. Yes rubydoc.info exists, but it's sort of an afterthought. Sidekiq uses a github wiki, Nokogiri has a dedicated site, Rails has a dedicated site, Ruby itself has yet another site. Some use RDoc, some don't. Or look at Devise https://rubydoc.info/github/heartcombo/devise/main/frames, there's simply nothing documented for most of the classes, and good luck finding in the docs where `before_action :authenticate_user!` comes from.