Web lists-archives.com

Re: utf




> On Mon, Apr 02, 2018 at 09:39:05AM +0200, Andre Majorel wrote:
> >I wouldn't say that. UTF-8 breaks a number of assumptions. For
> >instance,
> >1) every character has the same size,
> >2) every byte sequence is a valid character,
> >3) the equality or inequality of two characters comes down to
> >  the equality or inequality of the bytes they encode to.

I am sure you do not realize that none of these assumptions are really
met by any encoding and none of these assumptions actually bring
something. They are just rehashed poor arguments to rationalize a fear
of change by people afraid their long-earned knowledge will become
obsolete. I do not know if you fit in that category. Odds are you have
just been misinformed after having normal trouble during the transition.

Darac Marjal (2018-04-03):
> If these things matter to you, it's better to convert from UTF-8 to Unicode,
> first.

"Convert to Unicode" does not mean anything. Unicode is not a format,
and therefore you cannot convert something to it.

Unicode is a catalog of "infolinguistic entities". I do not say
characters, because they are not all. Most of Unicode is characters, but
not all. As is stands, the principle of Unicode is that any "pure" text
can be represented as a sequence of Unicode code points.

For storing in a file or sending on the network, this sequence of code
points must be converted into a sequence of octets. UTF-8 is by far the
best choice for that, because it has many interesting properties. Other
encodings suffer from being incomplete, being incompatible with ASCII,
being sensible to endianness problems, being subject to corruption or
all of the above.

>	 I tend to think of Unicode as an arbitrarily large code page. Each
> character maps to a number, but that number could be 1, 1000 or 500_000
> (Unicode seems to be growing without might end in sight).

The twentieth century just called, it wanks the "code page" idiom back.

>							    Internally, you
> might store those code points as Integers or QUad Words or whatever you
> like. Only once you're ready to transfer the text to another process (print
> on screen, save to a file, stream across a network), do you convert the
> Unicode back into UTF-8.

Internally, this is a reasonable choice in some cases, but not actually
that useful in most. Your discourse is based on the assumption that
accessing a single Unicode code point in the string would be useful.
Most of the time, it is not. Remember that an Unicode code point is not
necessarily a character.

You need to decide the data structure based on the treatments you intend
to perform on the text. And actually, most of the time using an array of
octets in UTF-8 is the best choice for internal representation too.

> Basically, you consider UTF-8 to be a transfer-only format (like Base64). If
> you want to do anything non-trivial with it, decode it into Unicode.

No, definitely not. If somebody wants to do anything non-trivial with
text, then either they already know what they are doing better than
this, and do not need that advice, or they do not and they will get it
wrong.

Use a library. And use whatever text format that library use.

The problem does not come from UTF-8 or Unicode or anything
computer-related, the problem comes from the principle of written human
text: writing systems are insanely complex.

Regards,

-- 
  Nicolas George

Attachment: signature.asc
Description: Digital signature