How do we make room for their use as a cognitive-prosthesis (if you will allow such framing) without losing humane-ness?
As a more nuanced case, someone could dictate a long rambling stream of thoughts full of contradictions to an LLM to transcribe and summarize, and then they could reflect on the summary and write the real thing themselves. That's like talking to someone about a subject and writing about it afterwards, and also not something I would really take issue with.
That's my partial answer to the cognitive prosthesis question: keep it off to the side as a tool to help you do the work. There are still many caveats here, and the more complex the demands you make of the LLM, the greater the risk of some kind of break down. For example, continuing the example in the previous paragraph, if I did have the rambling conversation with an actual person who helped me refine and clarify my thoughts, I have some kind of mental model for the dialogue partner that can help me correct for some biases. With an LLM, anything resembling a mental model I could have would be very far off the mark, so I would need to be aware that I can't treat its output the same way I would treat something written by a person.
(By the way, for myself I have a stricter rule: I would only use an LLM in a way that made me a better person independently of the LLM and not dependent on the LLM, and thus I don't use them at all. That's what I actually recommend, but I don't expect everyone to adopt that rule.)
My main problem with having the LLM just do your writing is that it's simply a misrepresentation. It's wrong to claim you wrote something unless you chose the words. I really empathize with the struggle of expressing oneself clearly, but there's just no getting around this. And to be clear this isn't something pedantic or just a technicality, since the act of formulating a thought in an actual human language with valid syntax does require a level of care and attention that simply is not there unless you do it yourself.
Really? I seriously believe that people that think this way underestimate their self, especially if they're going to be sharing something valuable.
> How do we make room for their use as a cognitive-prosthesis (if you will allow such framing) without losing humane-ness?
As many others have expressed, we'd love to read the prompt (and the model's chain of thought if you have it). This scenario is like talking through a person translating things to each other, except from the recipient's perspective, the translator is absent from the conversation entirely.
Additionally, disclosing LLM usage is low hanging fruit to differentiate yourself from the 100x other people who "do not review and vouch for things written on their behalf by LLMs". If the LLM converted something unreadable (whether due to a language barrier or incoherence) to text relevant to people on the other end, and THEN they disregard it anyway, the fault is not on you for them dismissing the writing too early.
Just watch an interview with a famous author. Or compare a legal brief to what court actually sounds like.
If you're using an LLM to tidy something up, that's one thing. If the LLM is your voice and is supplanting your knowledge, I might as well cut you out and talk to the LLM directly.
Jekill and Hyde vibes
I'm in the process of writing a pretty long essay that has taken a few months. I've written the first draft, and I'm almost done with the second draft, which incurred substantial revisions. At the end of the process, maybe no sentence will be something I would actually utter in person, but the writing is still something I created, and as such, it is a representation of who I am as a person. Had I used an LLM to do any of the writing, that would no longer be the case, and the writing would at best represent me as a person when the LLM is at my side, and at worst (and most likely) not really represent me at all.