On an unrelated note, I'm a little surprised there is no good open source web audio tracker (like Renoise but for the web) out there yet...
Unless part of your fun is keeping it so very trim, of course!
When I started developing I was a little frustrated with how bloated the web felt back then so I took that direction, it's much better today though and it's no longer an issue, but I still find it fun to impose these constraints and try to work within or around them (there's this fascinating concept of constrained creativity)
That said the web offers such great techniques to maintain this. Passive loading of plugins for example could keep things snappy and light and load things when you need them.
If you want the perspective of a prospective long term user: I'd be very comfortable using your app even at tens of megabytes. You could probably keep your initial load pretty light but pull in larger modules as needed. There are certain effects and audio layering I often use in Audacity that would keep me there, but your modern interface and browser access are huge selling points. If your vision includes moving to a bigger editor I guarantee you you'll find a huge base who wouldn't even notice megabytes of code.