upvote
Show HN: Chonky – a neural text semantic chunking goes multilingual

(huggingface.co)

42 points

by hessdalenlight2 days ago |
5 comments
upvoteby kamranjon2 days ago|
next
[-]
This is interesting! I once trained a t5 model by removing newlines from Wikipedia text and it worked surprisingly well / at the time the context length was the biggest issue.

Another, not so easy to solve issue was conversational dialogue type data, which wasn’t super well represented in the training data.

I’ve always wanted to come back to working on the problem again, because I think it’s very interesting and we will have a bunch of unstructured text as a result of STT models like whisper that do a great job of transcribing/translating but generally don’t format anything.

reply
upvoteby nvdnadj921 days ago|
parent|
[-]
In case you need conversational data for the experiment you want to try, I developed an open-source cli tool [1] that create transcripts from voice chats on discord. Feel free to try it out!

[1] https://github.com/naveedn/audio-transcriber

reply
upvoteby CjHuber1 days ago|
prev|
next
[-]
Took me a minute to realize this is not about Chonkie. I would be interested in how this compares to the other's semantic chunking approach
reply
upvoteby freakynit14 hours ago|
prev|
next
[-]
This is absolutely useless. Tried a few examples yesterday using hf demo. Fcking retarded af.

It literally splitted the text in-between of related texts while at the same time kept unrelated texts together, even though the embedding limit was far off.

I genuinely wanted this to work. I mean this. But nop. This shit did not work at all.

RAG is still fcked because if chunking issues. GraphRAG doesn't work correctly either unless you are willing to throw a lot of money during ingestion time.

reply
upvoteby TZubiri1 days ago|
prev|
[-]
That example looks terribly useless. Maybe there's an actually useful application you had in mind? I don't know say

Chonk("Hey I forgot my password, this is Tom from X Company") = ("Hey", "I forgot my password", "this is Tom from X Company")

Even then it doesn't quite look helpful.

reply