Not without having a degraded git experience like shallow clones, or using hacks like LFS or Xet, and then you're back at the initial problem of depending on "something else besides your repo".

as far as I can tell, fossil doesn't support that. but I could be wrong.