I would really appreciate your input!
1. Top end model on high/xhigh thinking (last time I did it it was Sol xhigh I think)
2. Make sure it creates some representative fixtures of different sizes and sets up a good testing, profiling and benchmarking loop that doesn't require my input.
3. Make sure it has access to reference implementation code
Edit: Oh and one obvious pitfall that for some reason I still have to remind even smart models of from time to time: make sure it knows not to try to parallelize its benchmark runs. I've occasionally had an agent struggle to figure out absolutely nonsensical data because it tried to run multiple tests on the same compute hardware simultaneously.
My software is labeled "CPU only unless using an nVidia GPU". I would prefer to strikethrough "nVidia". Incidentally, this means no more Mac support.
Few problems I noticed:
* coopmat2 from nvidia is the classic embrace, extend, extinguish. No point to ask the models to translate from CUDA to coopmat2. Instead, the models can understand the existing CUDA and CPU kernels, and adapt them accordingly to non-nvidia devices.
* However, the standard API is also lacking. The models struggled to make prompt processing compute-bound on strix halo when the graph is complex. Upfront standard API might just be an evolution dead end.
Sucks like important libraries like Alphafold are locked into CUDA. Its ridiculous for researchers.
It's likely that CUDA will continue dominating until they put aside their differences. The current MLX/MPS/ROCm ecosystems are too fractured to threaten Nvidia.
Intel uses SPIRV iirc
They're migrating away from SPIR-V to their own, Intel PISA: https://discourse.llvm.org/t/rfc-upstreaming-the-pisa-backen...
Vulkan's SPIR-V dialect is substantially different from the OpenCL one, notably with the former having structured control flow. They're incompatible between each other.
The basic underlying tech can be agreed on, but Apple/AMD/Intel all have different GPU priorities that limit their ability to agree on a CUDA-adjacent hardware platform.
Intel missed for a different reason - they didn't invest in the product space at all for decades. In terms of units they had the most market share (millions of Intel Integrated Graphics chips), but it wasn't seen as important. Bare minimum to render Windows and Office UI, nothing more.
Intel (via Codeplay) was handing it out on a silver platter - Nvidia on SYCL, full top chain, and people still wouldn’t want it.
By holistic, I mean hardware architecture cooperation. Nvidia can hold onto their lead forever if GPU designers fight over what a GPGPU hardware baseline looks like. The current ecosystem fragmentation is not competitive, and future fragmentation probably wouldn't work either. I think the fastest way to kill Nvidia would be a hardware consortium.
What would you expect the hardware consortium to coordinate on? Unified ISA?
Yes they have implementations on top of CUDA but they're maintained by... Intel. They didn't get buy-in for cross-vendor collaboration
Source: worked at a bunch of accelerator companies in the kernels or equivalent team. They're nothing alike.
My thinking was rather that most of our current programming languages put memory layout fully into the programmer’s responsibility - I can think off hand of a language where the compiler makes performance decisions like whether your structure are SoA, AoS or SoAoS, what alignment, padding, strides and float types to use.
Automatic decisions about when to use cooperative loads through shared local mem versus gathers from global mem and hardware caches are also something that such a hypothetical compiler would have to make.
As for ai and matrix vector performance... I mean matrices are extremely useful across many domains. The hands off language that exists today is called blas. That's fine but won't lead to the best perf on any GPU today.
SoA and AoS data layouts are not even a worthwhile point of contention. Same with shared v global mem. Today's hardware has specialized memory depending on the operation. The hardware on these processors is so specialized as to make anything but first class support for the feature be completely pointless. If you look at Nvidia code that's open source even they will special case a lot of their chips. Literally if you want the best perf you write the kernel exactly for the exact chip. That's intra vendor .. you can only imagine inter vendor