upvote
Show HN: TERMy – A fast terminal assistant that does not use LLMs

(github.com)

106 points

by gioscarab20 hours ago |
30 comments
upvoteby zem7 minutes ago|
next
[-]
what I would love to see along those lines is something that can answer "what packages do I have installed to do task $foo"; I keep installing things that I use for one thing and then forget about when I need to do the same task some months or years later.
reply
upvoteby Alpha30311 hours ago|
prev|
next
[-]
Very interesting project, I like it. Just wanted to clarify though the sentiment analysis is just the count of stripped words and used to tag things with the emoji? I was initially expecting it to be a part of the actual command construction process (even though I couldn't figure out how that would be relevant) given how it was listed.
reply
upvoteby publlus_enigma4 hours ago|
prev|
next
[-]
This is a cool project; I think it's wonderful that traditional NLP methods are being used, rather than reaching straight for an LLM.

One significant advantage of not using a local LLM is the significantly simplified dependency stack.

reply
upvoteby nateb20229 hours ago|
prev|
next
[-]
cf. https://github.com/TellinaTool/nl2bash

paper: https://arxiv.org/abs/1802.08979

reply
upvoteby gioscarab8 hours ago|
parent|
[-]
Thank you very much for the link.

WOW! With that dataset the capabilities of TERMy could be vastly extended!

Thank you.

reply
upvoteby vegnus10 hours ago|
prev|
next
[-]
If you could get Termy to code, you'd be a rich man
reply
upvoteby dmos627 hours ago|
prev|
next
[-]
It would make sense to have this integrate with a self-learning routine for an agent: e.g. at night it looks through what it did and writes NPC-Forge recipes. Tomorrow it can answer queries (which he turned added to NPC-Forge) without an LLM. Of course this implies a branching where a query is either processed by NPC-Forge or an LLM, depending on some measure of confidence that NPC-Forge can answer it well.
reply
upvoteby gioscarab6 hours ago|
parent|
[-]
Yeah would be a nice experiment, if you are interested to contribute to NPC-Forge please open an issue, I would be happy to discuss about that.
reply
upvoteby gioscarab11 hours ago|
prev|
next
[-]
Hi, I am the creator, feel free to ask any questions :)

What do you think about it?

reply
upvoteby gurjeet10 hours ago|
parent|
next
[-]
I haven't evaluated it yet, but I love the fact that the output is (at least claimed to be) deterministic. I can't trust an LLM to do the right thing after I deploy it to production, because their output is non-deterministic by design.

TERMy (or is it the NPC-forge) seems to be worth a try.

reply
upvoteby piterrro9 hours ago|
parent|
next
[-]
You can get determinostic output (mostly) by setting the temperature to zero. Using couple of other tricks you can get close to 100% of determinism with LLMs.
reply
upvoteby jdiff9 hours ago|
parent|
[-]
That's reproducible, I wouldn't call it deterministic. Small, semantically meaningless changes in the input can still result in wildly different output.
reply
upvoteby asQuirreL8 hours ago|
parent|
[-]
That's the definition of a chaotic system (small change in initial conditions results in large, seemingly -- but not actually -- random changes in output), but it's still deterministic (same input results in same output).
reply
upvoteby kouteiheika10 hours ago|
parent|
prev|
[-]
> because their output is non-deterministic by design.

It isn't. At least not by design, even though in practice it often can be. If you do greedy decoding (or use a preset seed) and deterministically compute everything (e.g. only use integer math) then it will be 100% always deterministic.

reply
upvoteby kennywinker9 hours ago|
parent|
[-]
That’s true, but not true-true. Sure, every time you prompt “what is the weather in kansas” you’ll get the same output, but if you prompt “what is the weather in kansas right now” you’ll get a different output, and then “what is the weather in kansas today” gets a different output. Language being language, there are infinite ways to say things, so there are infinite variations in what the llm can output in response to very similar prompts.

This tool has a finite amount of outputs for an infinite amount of inputs. Which is different from an llm based tool.

reply
upvoteby skeledrew8 hours ago|
parent|
[-]
I think the point being made is that given a particular input string, you can get a deterministic output string back from the LLM.
reply
upvoteby kennywinker6 hours ago|
parent|
[-]
Yes i think I acknowledged that, but is that useful for making a tool that can be trusted to safely run shell commands when asked arbitrary questions? No. It’s not.
reply
upvoteby kouteiheika10 hours ago|
parent|
prev|
next
[-]
> Models like ornith:9b, mistral:7b or cogito:14b can get the job done sometimes, but they are not fast and reliable enough for general use, specially if you have only 4GB of VRAM.

Have you considered/tried using a model that's, well, more appropriate size-wise for an use case like this? These are relatively big. Something like FunctionGemma [1] finetuned for a given set of tasks would be a lot more speedy.

[1] https://blog.google/innovation-and-ai/technology/developers-...

reply
upvoteby coder5439 hours ago|
parent|
next
[-]
FunctionGemma never worked well for me (without fine tuning). Liquid has released 230M and 350M models that work far, far better in my testing: https://huggingface.co/LiquidAI/LFM2.5-230M

I really look forward to a hypothetical LFM3-230M, because LFM2.5-230M is so close to being usable, while FunctionGemma is miles away from being usable.

But, yes, still tangential to TERMy.

reply
upvoteby gioscarab10 hours ago|
parent|
prev|
next
[-]
I tried functiongemma, it is for sure faster than those models, the problem is that is not reliable enough for a terminal assistant. I would say that no LLM is good for a terminal assistant, if you take into account the operational cost and the risk of damage. Even if it fails only 1 time out of 10 becomes useless. That's why I developed FlintParser!
reply
upvoteby kennywinker9 hours ago|
parent|
prev|
[-]
https://github.com/ThorOdinson246/whatisit-nl2sh uses a finetune of Qwen2.5-Coder-1.5B-Instruct. It works pretty well, tho it will misunderstand things from time to time
reply
upvoteby utopiah10 hours ago|
parent|
prev|
next
[-]
What dataset does step 5 rely on? Is it from your own terminal history, man pages, scrapped dataset from e.g. StackOverflow, sth else?
reply
upvoteby gioscarab10 hours ago|
parent|
[-]
The dataset is here: https://github.com/gioblu/NPC-Forge/tree/main/npcs/termy/dat...

I hope the community will help me to enhance it :) it is just a proof of concept for now

reply
upvoteby 9 hours ago|
parent|
[-]
deleted
reply
upvoteby registereduser110 hours ago|
parent|
prev|
next
[-]
Cool project! How does it differ from warp terminals ai mode where you can ask it questions and it responds back
reply
upvoteby gioscarab10 hours ago|
parent|
[-]
Warp uses LLMs so it is slow and prone to hallucination. Using very colloquial terms TERMy is more or less a calculator that knows english :) so it can run on your CPU and respond instantly! The difference is that it can only answer predetermined responses (with optional arguments) this makes it useless if you need to generate text, but makes it safe and predictable for a use case like a terminal assistant.
reply
upvoteby mpalmer10 hours ago|
parent|
prev|
[-]
At first blush, it is a really persuasive compromise between full-on LLM inference and boring old fuzzy history search!

I really like it, this flavor of specialization gives the user a win on privacy and speed. Seems like the right idea for such a tool.

reply
upvoteby mbil8 hours ago|
prev|
next
[-]
It's kind of antithetical to the tool's deterministic positioning, but have you considered making TERMy leverage an LLM for unseen or low-confidence queries, and then generate the config and update itself to make future similar queries deterministic?
reply
upvoteby gioscarab8 hours ago|
parent|
[-]
This is such a nice idea! I could add a fallback towards LLMs, it was present but I removed it. Would you be interested to help me implement the auto-update? I must admit, the LLMs are very useful for this kind of work. I think that TERMY's design is now feasible BECAUSE OF the availability of LLMs. They make the dataset development feasible.
reply
upvoteby indigodaddy9 hours ago|
prev|
next
[-]
So is this kind of like a super-powered tealdeer ?
reply
upvoteby gioscarab9 hours ago|
parent|
[-]
tealdeer just shows you a cheatsheet, termy can effectively take a prompt and execute a command, example:

$ termy create file test.txt and write Hello

TERMy | template match | Confidence: 100.00%

Thinking: Ok, I am asked to create the file test.txt.

echo 'Hello' > 'test.txt' && termy_set_context 'active_file' 'test.txt'

Description: Writes Hello in file test.txt.

Response: Affirmative

Now that I think about it, I should let TERMy use tldr...

reply
upvoteby nirmeet01101112 minutes ago|
prev|
[-]
[flagged]
reply