Thanks for taking the time and test it.

Key management is a critical part of the story, as j2kun has pointed out before. In this demo, we're intentionally caching the same key material for every visitor so we can showcase the actual FHE computation without making everyone wait through client-side key generation and upload. Even as a one-time cost, having each user generate their own keys in the browser and transfer them to the server introduces noticeable setup time, which we felt would get in the way of the experience for a demo. We actually implemented it that way at first; setup wasn't a matter of minutes, but we changed course after worrying about the experience for visitors on mobile data.

This demo is focused on demonstrating the computation itself, not a production-grade key management flow. Supporting per-user keys and a more realistic trust model is definitely on our roadmap for future demos.

IMO a big unanswered question in FHE (which is my field so I care about the answer) is key management. If you have a service with lots of different users, and each user requires 100 MiB or more in evaluation key material, simply loading that key material from disk onto the GPU for each user's request adds a nontrivial overhead on the order of magnitude of the FHE computation itself. Not to mention the storage cost (for a 1B-user service, each requiring 1 GiB of evaluation key material, requires 1 EiB of storage, which is a hard sell).

So it seems kind of reasonable to me that if you want to showcase your fast inference for its own sake, you can sidestep the remaining engineering problems around key management by giving the user pre-computed key material corresponding to hot-loaded evaluation keys.