Start small, hit issues, fix them, add features, iterate, just like any other software.
There's also a handful of smaller open source agentic tools out there which you can start from, or just join their community, rather than writing your own.
ML related stuff isnt going to matter a ton since for most cases an LLM inference is you making an API call
web scraping is probably the most similar thing