upvote
On the SDK front, you tried just having an agent reimplement the model you're interested in and just use the weights? I've taken to treating off the shelf implementations as reference implementations anyway, because I can often squeeze out significantly better performance for my configuration and use case by having Codex hammer at it for a few hours.
reply
Hugging face is working on something like this where well known models get fused into a single implementation.
reply
Could you share the prompts and workflow? I’ve tried this too, but with mixed success when it comes to creating custom tile kernels.

I would really appreciate your input!

reply
It's been pretty ad hoc, but my prompts are nothing special. Things I generally do:

1. Top end model on high/xhigh thinking (last time I did it it was Sol xhigh I think)

2. Make sure it creates some representative fixtures of different sizes and sets up a good testing, profiling and benchmarking loop that doesn't require my input.

3. Make sure it has access to reference implementation code

Edit: Oh and one obvious pitfall that for some reason I still have to remind even smart models of from time to time: make sure it knows not to try to parallelize its benchmark runs. I've occasionally had an agent struggle to figure out absolutely nonsensical data because it tried to run multiple tests on the same compute hardware simultaneously.

reply
The problem with the open standards is that their dev UX is absolutely horrible. You can't neglect usability, and then be surprised that there are no users.
reply
This is at the core of the matter for me, and my knowledge is too weak to understand why this is the case. I don't enjoy the idea of relying on Nvidia's stack for GPU compute, but the alternatives I've tried (e.g. Vulkan compute) are higher friction to use. I am trying to reconcile why; Nvidia shouldn't have this moat.

My software is labeled "CPU only unless using an nVidia GPU". I would prefer to strikethrough "nVidia". Incidentally, this means no more Mac support.

reply
Vulkan compute is not really designed or intended to be a CUDA competitor, its feature set is much more restricted, and Vulkan host side code is much more verbose than CUDA. OpenCL or SYCL are much closer in features to CUDA. I found that when using SYCL on Nvidia, debugging symbols etc can be passed through and you can use tools like NSight Compute to profile it as if it were CUDA.
reply
I tried getting LLMs to add proper Vulkan support to ik_llama.cpp, which have very good support for CUDA and CPU. The models do an admirable job; they don't care much about poor DX.

Few problems I noticed:

* coopmat2 from nvidia is the classic embrace, extend, extinguish. No point to ask the models to translate from CUDA to coopmat2. Instead, the models can understand the existing CUDA and CPU kernels, and adapt them accordingly to non-nvidia devices.

* However, the standard API is also lacking. The models struggled to make prompt processing compute-bound on strix halo when the graph is complex. Upfront standard API might just be an evolution dead end.

reply
coopmat2 can be implemented by anybody non-nvidia. It's not EEE when the regular coopmat extension is not good enough to get good performance
reply
On the other hand, despite my complain about the standard API, the models were able to come up with cooptmat1 kernels that run dsv4 flash faster than whatever the guys at antirez/ds4 can come up with using rocm, on a strix halo, with the added benefit that I can also pair the strix halo with an egpu to drastically speed things up.
reply
From what I can tell, coopmat2 can get to about 75~90% of cuda performance on a single device, and there is no good way to do direct communication across devices. It is fair to say that nobody would replace cuda with coopmat2? That looks like a EEE project that can assigned to a couple of nvidia engineers, to fragment the ecosystem.
reply
yeh - exactly.

Sucks like important libraries like Alphafold are locked into CUDA. Its ridiculous for researchers.

reply
Commercially it's better to have AMD support CUDA which can help break Nvidia soft monopoly.
reply
I would too, but sadly that's Khronos' job to organize, and they've had trouble getting American vendors to work together.

It's likely that CUDA will continue dominating until they put aside their differences. The current MLX/MPS/ROCm ecosystems are too fractured to threaten Nvidia.

reply
Maybe a not-Khronos org should try
reply
Wdym American Vendors?

Intel uses SPIRV iirc

reply
> Intel uses SPIRV iirc

They're migrating away from SPIR-V to their own, Intel PISA: https://discourse.llvm.org/t/rfc-upstreaming-the-pisa-backen...

reply
To my knowledge, SPIR-V on Intel will stay, and be it only because it’s part of the OpenCL and Vulkan standards.
reply
Yeah talking about the (vendor-preferred) compute part here

Vulkan's SPIR-V dialect is substantially different from the OpenCL one, notably with the former having structured control flow. They're incompatible between each other.

reply
Yes, unfortunately. Otherwise we could just implement all of SYCL and OpenCL on top of Vulkan and live happily ever after.
reply
I'm talking about holistic efforts like OpenCL, and standards that would be equivalent to Nvidia's "Compute Capability" versioning.

The basic underlying tech can be agreed on, but Apple/AMD/Intel all have different GPU priorities that limit their ability to agree on a CUDA-adjacent hardware platform.

reply
Ultimately AMD missed the most important boat with their woeful support for GPU compute. 10 years ago they needed to go all in and offer something to compete with CUDA - whether that was internal or something standard from Khronos. They just couldn't commit to it as a business, didn't have the vision.

Intel missed for a different reason - they didn't invest in the product space at all for decades. In terms of units they had the most market share (millions of Intel Integrated Graphics chips), but it wasn't seen as important. Bare minimum to render Windows and Office UI, nothing more.

reply
What do you mean by holistic? SYCL is an open versioned standard that allows for vendor specific extensions. The problem is not that there isn’t a proper standard, the problem is that many hardware vendors - or software developers simply don’t want to adopt it.

Intel (via Codeplay) was handing it out on a silver platter - Nvidia on SYCL, full top chain, and people still wouldn’t want it.

reply
Isn't OneAPI a good example of the problem, alongside Mojo/ONNX/TensorRT? The industry doesn't need a fifteenth competing standard. They need hardware buy-in.

By holistic, I mean hardware architecture cooperation. Nvidia can hold onto their lead forever if GPU designers fight over what a GPGPU hardware baseline looks like. The current ecosystem fragmentation is not competitive, and future fragmentation probably wouldn't work either. I think the fastest way to kill Nvidia would be a hardware consortium.

reply
The problem with OneAPI is naming. It leads people to believe that is another competing standard where in fact is is simply just an implementation of a standard compliant SYCL compiler. If it just had been named “Intel SYCL compiler”, similar to the existing and accepted Intel OpenCL compiler, it would have been easier.

What would you expect the hardware consortium to coordinate on? Unified ISA?

reply
oneAPI is effectively an Intel-only platform not a standard.

Yes they have implementations on top of CUDA but they're maintained by... Intel. They didn't get buy-in for cross-vendor collaboration

reply
They were maintained by Codeplay - paid for my Intel. Nvidia can make contributions anytime they want, and here is the problem: Nvidia does not want to. Until each vendor starts pitching in with contributing their backend to an open standard, you will have to rely on others doing it for them.
reply
it will only get worse now when hughingface was bough by nvidia, not immediately, but in span of year or years...
reply
It's impossible to have an open standard. Hardware accelerators are nothing alike and have different perf characteristics. Each kernel is tuned to the hardware. The idea of writing a performant kernel in opencl is a fantasy.

Source: worked at a bunch of accelerator companies in the kernels or equivalent team. They're nothing alike.

reply
For a performant portable language, we’d have to go to a higher level where you describe what to do and leave the how to do to the compiler. It would then need to be able to adjust memory layout, access patterns, data type choice to the underlying hardware. I’m not sure if this is possible to do reliably - the closest we have right now may in fact be highly detailed plain English descriptions of the algorithms fed to an LLM prompted to produce assembly.
reply
It's not possible to do reliably. None of the models in current use today use any esoteric math. It's extremely easy to implement the math behind both the inference and learning of all modern models.
reply
Not everything is AI and dot products of massive vectors, there are still applications that do other maths on GPUs

My thinking was rather that most of our current programming languages put memory layout fully into the programmer’s responsibility - I can think off hand of a language where the compiler makes performance decisions like whether your structure are SoA, AoS or SoAoS, what alignment, padding, strides and float types to use.

Automatic decisions about when to use cooperative loads through shared local mem versus gathers from global mem and hardware caches are also something that such a hypothetical compiler would have to make.

reply
I mean if you don't care about perf, opencl does what you want, and exists today.

As for ai and matrix vector performance... I mean matrices are extremely useful across many domains. The hands off language that exists today is called blas. That's fine but won't lead to the best perf on any GPU today.

SoA and AoS data layouts are not even a worthwhile point of contention. Same with shared v global mem. Today's hardware has specialized memory depending on the operation. The hardware on these processors is so specialized as to make anything but first class support for the feature be completely pointless. If you look at Nvidia code that's open source even they will special case a lot of their chips. Literally if you want the best perf you write the kernel exactly for the exact chip. That's intra vendor .. you can only imagine inter vendor

reply