The reason 2D grids work well is because you can split tensor axes along whatever dim you want across PEs (batch, sequence, hidden, etc.). And your per PE code is relatively simple. It's of the type : get data from neighbor, compute my subtensor, send to neighbor. And you can implement broadcasts and reductions along any axis.
replyYou have to route around disabled subchips though.
replyThe EDA tools that are used to design chips do not support diagonal wires so that kind of rules out hexagonal grids. With only horizontal and vertical wires you end up with rectangular/square grids or rings.
replyDo you need diagonal wires for a topological hexagonal tiling?
replyWouldn't a rectangular brick layout be topologically a hexagonal tiling
If that is so, would it be entirely to decrease search space for layout algorithms? I believe there isn't anything in the fabrication process that couldn't model diagonals.
replyI could see that square packing is more simple, but by shifting elements per row I don't see why it couldn't propagate at odd angles.
That's also my assumption. Place and route has to work out an NP hard problem on billions of wires.
replyAre the next generation of Cerebras chips designed by the current generation? Maybe it won't be recognizable as a particular layout but rather a chaotic but functional vibe layout.
replyA few years old now, but their co-founder whiteboards how their chips work https://www.youtube.com/watch?v=7GV_OdqzmIU
reply