I don't think you have encountered the problems that this class of formats solves. Try looking up columnar storage formats, the pros and cons are pretty well defined these days. It is not meant for video decoding, indeed.