If we care about self-hosting and open source and decentralization because we believe it leads to a better world, and not simply because we want to be control freaks and snobs, then we need to learn how to build shit that is actually convenient and usable and understandable to normal people.
I think it's possible. I think many people would want this, and care, if we could package it up in a digestible way.
Windows never ever ever makes you open a terminal to do anything, that’s scary to non-tech people. Windows “just works” and most people just want to minimize their computer use for work and go watch tiktok on their phone.
I remember trying to install apps and it was annoying for all the same reasons self hosting is these days. It was niche, only hobbyists did it, "nobody cared" and the presumption was that for normal non-nerds, phones were exclusively for calls and texting.
There are some spectacular successes in terms of the actual self hosted software (immich and home assistant) but the overall experience is still N95 not iPhone.
The part where self hosting falls down isnt the apps it's config, networking, reliability and backups.
If I tell my mom to download and install Ollama on her Mac, do you think she's gonna have any clue which model to download from all of the 5000 models she can choose from? Do you think she understands which ones will even work well on her machine? What if she downloads one that's highly recommended online but it doesn't work on her 64 GB MacBook (hypothetical example)? Because it's too too large/slow...
and can Ollama, out of the box, do actual work on her computer, similar to the ChatGPT app with its "Work" mode, and dozens of built in connectors (google, etc)?
you call this easy to use for a normal person? You just made their point. The ChatGPT and Claude apps that my mom can download, set the standard for her. Her standard of good/easy is not what you and I as nerds find easy. you need to really put yourself in the shoes of a tech illiterate person. we haven't even touched on the fact that Claude and ChatGPT ship with easy to use iOS and Android apps that sync up perfectly with her Mac.
The consumer market for LLMs is small. compared to the business market.
Before that number is high, can we realistically expect non-techies to use self-hosting?
Also, none of this is intuitive and a lot of popular closed stuff is harder to use (and vastly disliked by the very same normal users) than some popular open source stuff.
I'd even mention 'normies' want self hosted Word and Excel and regularly reject the web-based versions of them. Normies don't want streaming games but a gaming PC or console. Normies may buy streaming services like Netflix but get upset when their favorite shows are removed, so they actually don't like netflix. Normies complain about censorship on twitter and reddit and discord and would love an alternative they control, but may be unaware that's an option and seem stuck on those services and hate them.
Normies want self-hosted all the time too. Their ability to understand and do this is what's lacking. So that's a conversation on its own, but normies aren't actually happy. Form what I can tell most people hate 50% of the stuff they use right now and most of those complaints revolve around the cloud-hosted/lack of control/privacy and other issues self-hosting or federated services or FOSS apps could arguably solve for them.
This is a big conversation but imho tends to boil down to who has the marketing and sway to get you to install or use something, not that typical end users are savvy judges of software and useability working purely on merit and informed facts. Does your grandma actually like Facebook, or is she just lonely and its the lowest friction way to not be lonely? Do you really like Chrome or Word or is it just easy to get and popular? Is the big mac the best burger or is just easy to get and popular?
I mean...sure...but how convenient was that Office 365 outage? How convenient is it to find that a feature you actually depended on is gone because the company that runs your software decided to get rid of it?
It is always a back-and-forth. It is not as if centralized computing is a new idea, in fact it was the original way to use computers and people found that issues of control mattered and switched to PCs; the pendulum swung back because people realized that there are advantages to centralization (e.g. not having a failed hard drive destroy high-value documents). The cycle will probably continue for a long time to come because the advantages and disadvantages of both approaches are fundamental.
One outage in a blue moon? Bloody very convenient. People have more outage days per year on their own laptops.
It’s not so much about being snobs as it is about not having time and energy for a second job. The level of quality you demand simply requires a level of attention, investment and involvement that can’t be provided without working full-time on it, for profit.
Telling a bunch of hobbyists that they should provide service of a quality equivalent to that of a trillion-dollar corporation if they are serious about it is gaslighting.
I'll go purify myself now.
That's a mirror that hurts to look at that you just held in front of my face. Guilty on both counts, even though I try to convince myself I'm not
Why? Because Limewire was easy enough to install that even grandma could do it. There is not a single way to self-host even a simple html website without learning half a dozen software skills. People will self-host when it is click-and-play.
[1] https://marquettewire.org/3777966/tribune/marquee/the-life-a...
1. Set up port forwarding on your router
2. Open a terminal
3. Run `python -m http.server 8000` in the directory with the html file
I'm not sure you could really count any of those 3 as software skills, but even if you count all 3 it's still half as many as you said it would take.
A local AI can be as simple as a program that runs in your computer [0]. Any machine with 12GB+ of VRAM can run the same model I'm using. I has a chat interface, can see images, control your browser, generate artifacts, work on projects and workspaces. You name it.
They don't now, but they might soon.
It used to be that bad behavior on the part of our service providers was annoying because you'd get ads and spam. Nowadays the leaked data gets aggregated into apps that help government spooks abduct your neighbors. The effort remains constant, the threats it helps you avoid are getting more dire. Eventually those lines will cross.
In terms of self-hosting vs remote-hosting regular software… I mean, a lot of regular software is fundamentally about communication (social media and office suites) so that goes on the server. LLMs don’t benefit much from being remotely hosted, other than that it makes the flops run in somebody else’s machine. But that will be hard to fund in a universe where the upper-limit to what can be charged is defined by how much additional cost “good AI hardware” adds to the price of your phone. We’ll see I guess.
I told a remote model to set up strata on my local machine. With the best options for my hardware. If someone can figure out using remote models, it's trivial to bootstrap local intelligence assuming available hardware. The hardest part was knowing that strata exists.
Ninja edit: I've even had my agent tell me about suboptimal hardware configurations and told me what to do to fix it (XMP settings, having hardware in suboptimal slots). Cognitive load is simply a non-issue. The only issue is time, but in the grand scheme of things, it's a trivial amount of time to get up and running.
While this doesn't remove the upfront cost (which could be fixed with basic financing like Apple does) and physical manipulation (e.g. replacement on failure, etc.) it does lower the barrier.
Finally, there's another path I find interesting and interested to think more of which is sharing hardware locally, from your house to your neighborhood, etc. We've done this in the late 90s - early 00s with Internet and it worked really well. I think it could work great for compute.
I see a lot of opportunities in this direction, actually and most are actually enabled by the latest improvements in AI, so most of the "old" thinking doesn't apply anymore.
By that reasoning you'd also conclude that no one cares about affordable housing or having a good work life balance or having your own lake made out of solid gold.
Think of who uses local AI, it’s either
-power users upset with the limits —-cost of intelligence is going down
-people and corps that want stabiltiy —-a massive self own by the labs, this is justified
-American labs seemingly snitching on their users (wtf?) —-huge incentive for people to use Chinese inference, which is cheaper along with other benefits
Regardless, we’re talking about a subset of a subset of people, 1% of 1%. The big bet is that everyone will become AI power users. I think it’s accurate, forget openAI, fuck Anthropic, let’s look at us, the people. Entire workflows like coding have been buried, there’s no going back to how things were before. Is it reasonable to say that will happen to the other work streams? There’s no indication not, and it’s a total gamble if it’s yes
I really do not care if everyone does it my way. What matters is that the choice exists for those that want more ownership, freedom, and privacy.
That said, most next gen laptops will be shipping with capable inference hardware more than most consumers will ever need. Not using the GPU people have in front of them would be silly, but people are silly.
More broadly, those prompts and the responses have value. There's a whole shadow economy in China of aggregating queries across Claude (etc) and then using that as training data. It's like your entire search history is for sale.
Clould infra has typically been opaque to the provider. You run an MySQL instance on an Amazon VM and Amazon doesn't have easy access to the unencrypted traffic and what you store. Cloud LLMs don't operate that way.
AI hardware is improving at a rapid pace. By 2030 I believe a significant amount of LLM usage will be local for these and other reasons.