We were also surprised at first. The reason the models don't do so well is that they need to find information across 90k pages. When they are pointed to the right location they tend to do much better. And with these treasury documents grepping / keyword searching is almost impossible because everything appears thousands of times.

And thank you, we also love the traceability, it's one of the aspects that we have prioritized. Models will never be perfect so rather than building the best model harness we went for the best human harness haha.

Tbh it's been a while since I've looked at notebooklm so I expect it would have gotten better over time. One thing where I found it lacking in the past was the structure we could get out (which gives the traceability) - for example a deep dive on one the underlying data for this corpus: https://demo.parsewise.ai/projects/39bee9d8-d722-4b23-8894-e...

And yes, we're really excited whenever new open weights models come out that push quality, price, latency. We're finding that throughput is a big obstacle so I'm looking forward to more of this running locally, but it will be a while..