The rest looks fairly nice but there are a couple of things I would do differently: I would not have the tests for NULL, use signed integers for indices and dimensions, use a flexible array member to integrate the data into the vector type directly, and omit the capacity field (as long as benchmarking does not show it is really needed). I would also use variably modified types for bounds checking, and with C23 the include guards become largely unnecessary.
All identifiers that begin with an underscore are reserved for use as identifiers with file scope in both the ordinary and tag name spaces.
btw, I went down the micrograd path with numpy-primitives all the way to building a PyTorch clone that can pre-train and post-train LLMs (https://github.com/workofart/ml-by-hand). My learning focus was on the math/calculus <-> high-level APIs, instead of efficiency. I'm glad to see more people tackling this problem from different angles.
> Reference counting buys correctness and composability, but at a cost.
> Disadvantage #1: you must balance every reference. Each value_create, value_retain, and each operation's implicit retain of its operands has to be matched by a value_release. Forget one and you leak; do one too many and you free memory that is still in use. The training examples in examples/ are verbose precisely because they are scrupulous about this in their error paths; that verbosity is the price of leak-free C.
I’d put it this way: though the idea of ref counting felt very natural at the beginning for my use case, at some point I realized there were probably better techniques to achieve the same goal. I found myself multiple times writing nonsense like:
sum = value_add(v1, v2) mul = value_mul(sum, v3) [...] value_release(sum) value_release(mul)
so that later I could release the sum. When you only have one intermediate value it’s still acceptable, but at 3-4 it starts getting cumbersome.
After asking for feedback, someone rightfully pointed out that the better and faster approach for an autograd engine is using an arena allocator. My reason for saying “rightfully” is that arenas are ideal when you have many “objects” that have the same lifespan, such as the values involved in the forward/backward pass. RC is better when you have a lot of “objects” with independent lifespans.