Right, that makes sense. There's a ton of overhead in the setup though, I'm assuming that's because everything runs on the CPU by default. Is this then an OS design choice? Thanks for your input.
replyThere can be overhead, though GPU memory + drivers are some of the most optimized software in the world.
replySome OS/hardware have separate GPU memory (Nvidia), some have unified memory (Macos, modern consoles), but moving memory is usually not the problem.
Things don't "run on CPU by default", it's up to the programmer where to run it. But the CPU and GPU paradigm of "running a program" are very different, since on GPU you're running SIMD (i.e. running your code on arrays) or else there is no point to use a GPU.
For me, if-branching is the hardest to wrap my head around. "If"s are so natural to CPU programming that it's second nature to any developer but on GPU, it's expensive.
replyinstead of iterating over an array and if-branching to one of two paths for each item, conceive of your conditionals as logic operations which test for set membership (i.e. "is this array item a member of the set i care about?") and with this array mask of "do the thing or don't do the thing" flags, you can mutate your data 0 or 1 times.
replythe key for me is thinking in terms of sets—much of your conditional logic is probably about sorting things into sets anyway.