upvote
My harness actually has 3 distinct components: the LLM gateway, the agent session manager, and the tool server, and they can be run on different machines. The most obvious use case is the LLM gateway being a central router for all provider queries with multiple machines running the agent sessions.

The agent session manager talks to the LLM gateway over HTTP using a standard internal protocol, then the gateway has adapters for each upstream provider and translates between their external API format and the standard internal one.

The tool server is a separate jar running as a separate server process with its own systemd unit and the agent session manager talks to the tool server over a beanstalkd queue. The primary motivation for having 2 processes was total separation for unprivileged agent tool call execution (ie. the tool server systemd unit runs as a user with no sudo whereas the agent session manager can execute as root, so the user can run scripts through the interface that do things like install packages and access secrets, but the agent can't do that.

The tool server could just as easily be talking to beanstalkd over a Cloudflare tunnel or ssh port forward, though, and a session could be routed to different machines, allowing execution of tool calls in different environments within the same context window/agent session.

reply
[flagged]
reply