It bloats your repo, both with the actual code, and the large diffs when you update it.
You have to manually track new versions, without something to tell you if new versions are available, or if your version has known security vulnerabilities.
If the dependency has it's own dependencies, you have to vendor those too recursively. And if multiple dependencies have the same transitive dependency, it is up to you to deduplicate them, and make sure you have a version compatible with all dependents.
Etc.