There are three issues that arise

- structural: There are height and size limits on things you can build without a permit

- trespassing: flinging objects onto others people property is illegal

- endangerment: any activity that can endanger other people is illegal

All of these three depend on how rural you live I guess. So in a suburban neighborhood I would assume they get involved very quickly.

I suggest having a BBQ with your neighbors. That should minimize the pre-launch problems. If you use a steel ball bearing like he originally thought, that bullet will be the start of your legal problems.

Surely that depends on the side of your back yard?