As it is programs have to guess the following;
A) encoding. If ANSI which code page? If unicode which encoding? In the case of utf-16 big or little endian?
B) line endings? CR? LF? CRLF? I guess no-one uses LFCR... right?
C) number formats? 100.000 or 100,000? Date formats? is that mm-dd-yyyy or dd-mm-yyyy?
D) what human-language is it in?
E) CSV? Don't get me started...
Yes. Text is a lot easier to load, parse, make guesses about than say XLS. Yes all of the above things can be "guessed" to a greater or lesser extent.
Yes BOM exists to at least try solving the encoding question. Of course most files don't use them. Lots of tools don't support them. And they don't solve any of the other issues.
But sure, ASCII files in English with US date formats, and Windows line endings....no problems at all...
The line endings problem really isn't a problem. Pretty much every text editor out there can handle different line endings.
I don’t think number nor date formats are relevant here. For example you could have that same problem entering text into MS Word. That’s really more of an issue if you want to use text as a database rather than a document format, which isn’t really something that even plain text advocates would generally recommend.
As for human language detection, that’s a much easier problem to solve than decoding a proprietary binary blob.
CSV definitely has its warts. But it’s not like that’s the only plain text option for serialising data. (JSON, jsonlines, YAML, XML, etc). Or you could use the actual ASCII codes reserved for records, if you really wanted something that didn’t require quoting and escaping in plain text. It’s actually a pity nobody does this.
Basically it's a text-based language for embedding semantic metadata into some kind of underlying text stream. Is this interesting to you?
“Windows” newlines are also the standard in many communication protocols like HTTP and SMTP. That’s not because of Windows or DOS, it’s because it was the standard for teletypes which needed bot CR and LF. It’s arguably systems like Unix that deviated from that standard.
I agree that beyond ASCII there is a slope from well-supported to less-supported and quirky to problematic areas of Unicode. For example, HN filters many Unicode text elements like combining characters (Zalgo text) and emojis, and there is no specification to point at what it supports.
Even within ASCII, most control characters don’t have a portable meaning. So it’s really just the printable subset of ASCII, and strictly speaking not even that, given that there are regional variants of ASCII, such as the Japanese one where backslash becomes the Yen sign.
Fortunately they’re easy to test for and most languages have standard libraries that make this painless.
And in practice, still a rather large number of them, going back to the 1970's - https://en.wikipedia.org/wiki/Extended_ASCII