upvote

points

by Dfol7 hours ago |
comments
upvoteby simonw6 hours ago|
next
[-]
I don't understand.

System prompts are part of the software that customers pay to access.

Complaining about that is a bit like complaining that your Netflix subscription includes paying to execute the compiled code that Netflix wrote that serves you video streams from their servers.

Actually there is a difference: If Anthropic deleted a large chunk of that system prompt I guess you might get like a 1% increase in how much Opus 5 you can use via their chat allowance for your paid subscription.

Is that really something worth being frustrated by?

reply
upvoteby TZubiri6 hours ago|
prev|
[-]
Cached.

they are the first part of the input and it contains no user dependent variables, so the model is in a known state that it can reuse across all users, it does not need to recompute all that inference

reply
upvoteby cubefox6 hours ago|
parent|
[-]
Unless they are using a linear architecture, the compute cost still scales O(n²) for n tokens, and nemory cost scales O(n).
reply
upvoteby TZubiri4 hours ago|
parent|
[-]
>the compute cost still scales O(n²) for n tokens,

That is never the cost, it's a common misconception.

Cost scales linearly per tokens. Unless you are sending one token at a time and avoiding using the same machine or cache.

Just look at api charges, they are charged by token, not by token squared.

reply
upvoteby cubefox3 hours ago|
parent|
[-]
Which seems to contradict the usual consensus that purely linear architectures are not sufficiently capable and unsuited for frontier models.
reply