Do you mean Han Unification? I.e. that conceptually equivalent characters which are written differently in Japan and China received only a single unicode code-point, and are rendered the Chinese way by default on most computers?
I think some characters got different code points, while others were merged. And apparently the Japanese complained bitterly over the ones that were merged. If you had read any articles about that, this is probably what you have in mind right now.
And I'm also a bit tilted by the ones that had different code points, because when processing CJK text now we have to deal with characters that are (in my native Cantonese) essentially the same, looks similar (to my undiscriminating eyes), yet having different code points so that things like text search sometimes don't work.
Of course I'm not "blaming" the Japanese, if anything the simplified vs traditional Chinese thing is much more of a practical problem, and the conflicting code points I deal with on a routine basis are more of a Hong Kong vs Taiwan thing, but I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode (which I think really is some kind of Aristotelian essentialism...) mostly due to vocal objections from the Japanese.
I have no idea what "Aristotelian essentialism" is supposed to mean, or if you are saying that the unification was that.
> And apparently the Japanese complained bitterly over the ones that were merged.
and
> but I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode [...] mostly due to vocal objections from the Japanese.
Seems to contradict each other.
[1] https://en.wikipedia.org/wiki/Ideographic_Research_Group
Page 55:
""" there are characters with no glyphs. glyphs that can correspond to a number of different characters according to context. Glyphs that correspond to multiple characters at the same time (with weightings assigned to each), and even more possibilities.
The problem of glyphs and characters is so complex that it has gone beyond the realm of computer specialists and has come to be of interest even to philosophers. For example, the Japanese philosopher Shigeki Moro, who has worked with ideographic characters in Buddhist documents, goes so far in his article Surface or Essence: Beyond Character Model Set [274] as to say that Unicode's approach is Aristotelian essentialist and to recommend supplanting it by an approach inspired by Jacques Derrida's theory of writing [114, 115]. The reader interested in the philosophical aspects of the issue is invited to consult [165,156], in addition to the works cited above. """
I think "essentialist" is probably a good description of the philosophy of how Unicode defines characters as opposed to fonts and glyphs, so I adopted it.
(Arguably even in English we run into the fact that there is not one "A" - there are many "A"s that sound and act completely different, and we collapsed them into one representation.)
Which is always fun when you think that clearly nobody thinks of "o or o with a leg" (O vs Q).
What would that philosophy be about? Sounds apocryphal. Unicode has never done "unification" like that for other languages/scripts?
i/ı/i, ö/ø/ø̈/oͤ: Same same, different codepoints.
Search and sorting is a mess everywhere. Depending on your locale, ö sorts either after o or after z. Sometimes it's semantically and phonetically equivalent to o wrt search but moreoften not. https://en.wikipedia.org/wiki/%C3%96
Conceptually we long debated unifying everything and in an ideal perfect world we would have done it. The reason was one primary goal for a new standard was to make it easily parseable and having unique rather than repeated codes was key to that. Sadly in the end we did not unify everything only to get buy-in from all major countries to support. That’s even why you see the roman/asciii characters repeated within Unicode itself—like as romaji. This was all well in good until we came to CJK and the number characters with semantical overlap was huge that this was more seriously considered—infact we started investigating this at Xerox before even thinking about Unicode and that work predated and influenced and leveraged the work done later.
Japanese users are angry when system chooses Chinese fonts over Japanese, Chinese users apparently suffer with a font mishmash of Chinese and Japanese all the time like a mid-word capitalization. The official sanctioned solution is to just commit a genocide and nuke the offending language out of the system or to attach IVS to every Japanese characters which is like zero padding every single characters by a byte or two.
It's really putting everyone in pain and making absolutely no one happy, which sounds like what a good compromise tends to be, but LLMs seem to be struggling with there being multiple completely separate language families sharing codepoints but not character shapes or syntax. I've seen a smaller LLM model use made up verbs spilling over from the other language. I believe ML guys don't like datasets that do that kind of things. Same apply to image models.
> As far as I understood it, the result was an incoherent mess.
There has to be a reason why it happened(as to why Unicode suddenly started insisting it has to fit inside a 2^16 total chars or whatever).
Cross language search seems like a hack to me. Searching in Chinese should find Chinese words and searching in Japanese should find Japanese words. Being able to search in Japanese and get Chinese results is not what most users want, unless they don't have a proper keyboard.
I think you underestimate the similarities. For instance Japanese names are not translated to Chinese. Theyre just read with Chinese pronunciations. So Chinese will regularly interact with Japanese content (im guess it happens the other way around too, but i dont have the personal experience)
And then some examples of non-unified Chinese chars: https://en.wikipedia.org/wiki/Han_unification#Examples_of_so...
It looks like the CJK unified space is over 20,000 characters, so that's a real technical magnitude distinction compared to Latin and Cyrillic. "Asian languages be damned" seems like a bad faith read, compared to "Java and Windows char is 16 bits and that will never change realistically" (and in fact they still haven't, even in 2026 things which rely on UTF16 instead of UCS2 are still commonly bugged unfortunately)
Plus, these characters were only added 20+ years later, and in the supplemental planes, not the BMP.
The fullwidth/halfwidth stuff is a bit of a mess, but you could argue that these are actually CJK characters that just happen to resemble Latin characters (much like how Greek and Cyrillic both happen to have letters that look a lot like the Latin "A"), since they only exist for compatibility with older CJK encodings. This wouldn't be a very good argument though :)
Note that Han Unification has a history predating Unicode, and the Unicode work very much followed on from that.
The most notable is CCCII, developed in Taiwan in the early 80s, and then standardised in various places.
That's not to say that it hasn't been controversial (it has!), nor to say that it hasn't caused problems (it has!), but it's also unfair to say that it's just Unicode doing its own special thing.