Hey, this is a super cool project. It's great to see a lot of the IPFS stuff resurfacing again.

A few questions:

1.) How does this handle privacy? If you're distributing compute this way then all actors in the compute graph will also know the sequence being computed.

2.) Any safeguards against malicious actors poisoning model activations?

What is the incentive for me to join the public mesh? Do you have any fairness guarantees, e.g. if I contribute 1/8th of the VRAM required to run a particular model, do I get at least 1/16th of the inference share, or anything similar to this?

> A model gets partitioned by layer ranges into stages: layers 0 to 15 on one node, 16 to 31 on the next, and so on down the pipeline.

Numbers in this example are arbitrary. How does it actually work? What if the model’s number of layers is 33, or 34?

Is there a document explaining all constraints of this implementation?

This is super impressive, We have a lab with lots of different epycs and different models - to bring them together this way is amazing. Well done!

I have never really delved into kv cache implementation, do they run effectively separate caches per layer?

If so I can see it all dividing nicely, computation and data size wise and the only slowdown would be in search layer waiting for it's turn. If you pipelined it you could run multiple queries.

Is anyone doing best-of-n with a n stage pipeline running each query offset by one?

Is it a fully custom inference engine or are you reusing parts of an existing stack? (llama.CPP, vLLM, etc.)

The obvious burning question is how performance looks over different network conditions on some standard models. Have you done much benchmarking? Is it mainly latency affected or is overall throughput less than the capacity of the GPUs due to being distributed?

I wonder how security is done in this engine, since it's accepting input from anyone. llama.cpp's RPC layer seems to says that you shouldn't run it in public (I assume because it is lower level and may result in RCE on your GPU)

Would this benefit from integrating with the Colibri project announced here just days ago?

How does this differ from exo?

Curious about: does it have fault tolerance if one of the machines goes down mid-inference? Can it dynamically reroute, or does it just retry?

[flagged]