upvote
Making the data available is non-trivial no matter who does it. There were many, many exabytes of this data a decade ago and countless petabytes of new data is generated every day. How to manage this has been an open question for a long time.

In practice access has always been restricted to this "public" data because there is not remotely enough bandwidth, either network or storage, to give everyone a copy of the data that may want it. They often don't even have enough capacity for internal users. A lot of it also sits in offline or near-line archival formats due to the scale of the data.

This is a general problem for data models of this type. You need to compute on the data in-situ or it won't scale but virtually no one can build that type of infrastructure at the scale required for these data models. It would also be extremely expensive to build and operate.

reply