I'm not sure what all is needed to make that work for people.

Back in 2023 I started my own C# LLM library for doing tool calls and structured output, and over the years it has morphed bigger and bigger, and that is the backbone of almost all of my LLM-based projects.

I've never released it, but its easy to understand, and simple to add your own tools:

  [AIDescription("Get current weather for a location")]
  static string GetWeather(
    [AIDescription("The city name")] string city,
    [AIDescription("The country name")] string country,
    [AIDescription("Temperature unit", ["C", "F"])] string unit = "C")
  {
    // make some API call to a weather API and return a string to the LLM
    return $"The weather in {city}, {country} is 22°{unit} and sunny";
  }
  
  var chat = client.StartConversation("You are a helpful assistant with access to weather data.");
  var response = await chat.SendAsync<string>("What's the weather in London?", GetWeather);

My issue currently is KV Cache, because I can't keep enough parallel caches running (4 is where I'm at), so TTFT (is that the initialism?) can be long when I have a particularly large scene (basically more than 2 NPCs).

But my harness does let me offload to any OpenAI compatible endpoint, I just prefer local cuz free.