upvote

points

by onion2k9 hours ago |
comments
upvoteby michaelt5 hours ago|
next
[-]
> highlighting 5m lines of code at once is unusual

Speak for yourself, I often accidentally open 50MB JSON files, crashing my text editor as it tries to figure out the syntax highlighting :)

reply
upvoteby silon424 hours ago|
parent|
[-]
50MB in one line (with syntax highlighting) is a good test of a text editor.
reply
upvoteby avaer8 hours ago|
prev|
[-]
The use case isn't highlighting a huge amount of code, but local re-entrant highlighting.

You generally never need to highlight a whole file, only the part you're looking at. Which by definition must be fuzzy since you don't have the full source, the part you see will not parse as a complete program so you need to guess. This is actually one of the nastier parts of writing a highlighter, and it must be done for every language.

The lesson is that if you're going to be fuzzy you might as well be learned too and the results are pretty good.

This is more useful than it appears at first glance. It would be just as useful running on the CPU, which I'm sure it can.

reply
upvoteby umpalumpaaa7 hours ago|
parent|
next
[-]
Is that really the use case for the related packages mentioned on their website? highlightjs and prismjs are typically used to highlight code snippets on websites… which are at most a few hundred lines long. And are usually highlighted as a whole
reply
upvoteby tough6 hours ago|
parent|
[-]
you have to bundle and ship per language syntax dictionaries tho
reply
upvoteby normie30007 hours ago|
parent|
prev|
next
[-]
> the part you see will not parse as a complete program so you need to guess

Presumably this could also be a useful trait when live highlighting of files when editing, as in-progress typing is likely to be unparseable sometimes.

reply
upvoteby silon424 hours ago|
parent|
prev|
[-]
No need to guess... For basic system highlighting, one can store the lexer context periodically (maybe per line) and invalidate it if edits are made before it. For more advanced stuff, you need a more complete parser anyway and do it asynchronously (LSP...).
reply
upvoteby vidarh3 hours ago|
parent|
[-]
That is a good optimization, but you still need to figure out the starting state at a given location, and for quite a lot of languages that is not solvable in the local case.

E.g. if you're looking at a page of Ruby, you can not in the general case know if it's inside or outside a quoted string, as the quote character can be any arbitrary character (and I really mean any. "% x " excluding the double-quotes is a valid Ruby quoted string where the quote character is space but you can e.g. pick a unicode codepoint you're not going to use anywhere else in the file)

So unless you parse from the start of the file, you're left with fuzzy matching, and that can be made good enough the vast majority of time.

reply