upvote

points

by jedbrooke15 hours ago |
comments
upvoteby tomashubelbauer12 hours ago|
next
[-]
If you're on macOS you can try the built in LLM which I think is similar in size. There's a project called Apfel that wraps it in a CLI. Also Chrome ships with a web API called Prompt API that gives you offline access to Gemini Nano which can do both text and images at the input. Also tiny. I've integrated these into my workflows where a tiny but non zero amount of reasoning is needed in between the otherwise fully deterministic steps.
reply
upvoteby jedbrooke4 hours ago|
parent|
next
[-]
looks like the macOS one is Tahoe only. I’ve been putting of upgrading to tahoe but this might be enough to tempt me
reply
upvoteby stogot6 hours ago|
parent|
prev|
[-]
What kind of reasoning makes this worthwhile?
reply
upvoteby tomashubelbauer5 hours ago|
parent|
[-]
I have a personal, fully offline and local version of Windows Recall basically, but good, made using macOS built-in OCR and LLM. The reasoning requirements are tiny (just interpret the screen based on the OCR, do rolling de-duplication and summarization), but they are non-zero. The tool is valuable to me and it being dep-free and fully offline and local just gives me a good feeling.
reply
upvoteby skinfaxi5 hours ago|
parent|
[-]
Would you ever consider writing up or sharing your setup?
reply
upvoteby tomashubelbauer4 hours ago|
parent|
[-]
The ingredients are:

1. Bun.Cron API to run a script every minute

2. Bun.$ (Bun Shell) to execute the macOS command to take a screenshot (I do this for all connected screens at that moment)

3. Bun.Image to downscale everything to 1x in case some of the screenshots are 2x

4. Bun Shell again to run a JXA AppleScript thing to use the Vision Framework or whatever it is called to OCR the image into a file

5. Bun Shell to run the Swift compiler in the one-off eval mode with inline Swift helper that runs the Foundation Models Framework built-in LLM with a system prompt that tells it what the OCR said and instructs it to glean what may be on the screen (can't do this with JXA because the models are not exposed with ObjC APIs)

6. For each screenshot, continuously, take the previous day summary file and the last OCR/context results and produce a new summary of the day

I plan on adding extra information from the OS like the currently opened windows, currently focused window, time of day etc. into the mix, but so far it hasn't been needed. It produces reports of a good enough quality for me.

I `grep` these daily summaries whenever I need to recall a link I saw or a find what channel a message I spotted was in or take another look at that one tab I already closed, maybe re-open it by its OCR'd URL etc.

reply
upvoteby golph12 hours ago|
prev|
next
[-]
I actually tried building a harness around their constraints, just to find out if it was possible, but the combination of small context window, no tool calls and just small model, made me understand, that it’s not going to work.

If you find a way to do it, I’d love to hear it!

reply
upvoteby haellsigh8 hours ago|
prev|
next
[-]
I added it in my oh-my-pi configuration before (it's OpenAI compatible), but Llama 3 8B is just absolutely unusable for anything coding related. It is very fast and the latency is very good however.
reply
upvoteby venusenvy475 hours ago|
prev|
next
[-]
I tried the site and can't find any information about what it is. What is it?
reply
upvoteby npilk5 hours ago|
parent|
[-]
They make custom chips with a model's weights and parameters "hard-coded" which allows for much, much faster inference.
reply
upvoteby rbinv9 hours ago|
prev|
[-]
Codex offers a -spark model that runs on Cerebras. Not quite 17k tok/s, but _very_ fast nonetheless. Worth a look.
reply