Character Encoding and Unicode

Prerequisites: completion of Number Systems and Bit Manipulation; Python 3 for the hands-on steps.

Sourcing & version notes: verified against the Unicode Standard, Version 17.0 (released September 9, 2025 — the current version as of this writing; Version 18.0 is tentatively planned for September 2026, and per Unicode's own beta-review schedule this roadmap should be re-checked against it once released) and RFC 3629 (UTF-8, the IETF standard that has defined UTF-8's exact byte-level encoding rules, unchanged, since 2003). Unicode 17.0 brings the total number of encoded characters to 159,801; the specific character count is the only thing that changes meaningfully version to version — the encoding mechanics this project covers (code points, UTF-8/16/32, normalization) are stable, decades-old parts of the standard.

The single most important distinction in this entire project, and the one most confusion about "encoding issues" traces back to: a code point is an abstract number the Unicode Standard assigns to a character (U+0041 is the code point for the letter "A"; U+1F600 is 😀). An encoding is a concrete rule for representing that abstract number as actual bytes in memory or on disk. The same code point can be encoded completely differently depending on which encoding you choose — UTF-8, UTF-16, and UTF-32 all represent U+0041 differently at the byte level, even though it's unambiguously "the letter A" in all three.

This project starts from ASCII (where code point and byte value were, for decades of computing convenience, the same number) through the mess that followed when the world needed more than 128 characters, to Unicode's actual solution, ending with the specific, mechanical byte patterns that make UTF-8 the encoding almost everything uses today.