That phrasing and analogy stuck in my mind, of looking at the space of all possible programs as a resource to be explored for valuable nuggets of algorithms. Your description of interpolating between two functions gives me a similar perspective, of seeing algorithms not only as discrete and separate objects/processes, but "slices" of a larger space, the continuum of computation.
What the article is describing seems to me like "slices of semantic space", not just similar on the surface, but it's actually talking about the same space explored using different tools and lenses.
I’ve been mulling it over the past couple of weeks. I’m not sure what to make of it.
I'm a fan of Michael Levin's work, a biologist and philosopher riding that edge between genius and mad science. In the interview he mentions how behavioral psychology may be a useful tool in studying the behavior of algorithms and other phenomena typically not associated with having a mind of its own.
I agree about the small piece of the pie, but I can't really see it as a sorted collection.
The essence of many dimensions is that you can head off in another direction that doesn't impact the relationship of other dimensions. It seems common to consider a latent space as a encoding of meaning. It is certainly a mapping of relationships, and I think there's some pretty good philosophy arguing that a set of relationships is synonymous with meaning.
A long distance view of LLMs is embeddings encode meaning, Attention finds relevant meanings, and the perceptron does the thinking on things that have meaning and are relevant to each other. The transformer is those things stacked to turn input latents into output latent with a different meaning relative to the input. Stack enough transformer layers to get lots of thinking about lots of meanings.
I'm not entirely convinced that embeddings are doing exactly the things that the ideas like the King - man + woman = Queen examples suggest. It seems to me that the number of dimensions are too low for that to allow a good combination of ideas.
I have wondered how things would look if you considered, instead of cosine difference have something like min(cosine_difference(dimension_mask_0)... cosine_difference(dimension_mask_n)).
The idea would be instead of dimensions being pure encodings of some group of meaning some dimensions are expendable.
Like if you had W,X,Y,Z dimensions and you wanted to encode the relationship between items if they all had identical circular dials with a thousand concepts written around the rim. and the dial faces were around WX,WY,WZ, XY, XZ, YZ, you could link any two concepts with any two dials.
In higher dimensions the combination of relationships possible become astronomical. and to me it seems intuitively more expressive for relationships than the weighting of some dimensions representing a vector that encodes a magnitude of a single concept.