Indexing your corpus as you go makes retrieval a lot faster, and then the agent can dig into the specific file if it needs something more.
Having an inverted index (as with FTS5) is useful in that, for a basic single-term lookup, you reduce a sequential scan, O(N), down to O(log N). For small N, the performance difference might not be meaningful. Performance gap widens with more sophisticated queries (boolean operators, ranking, etc).
i can see for public facing deployments agent memory like this could result in faster roundtrips.