HowToCodePage

Character sets: ASCII, Latin-1 and Windows-1252

ASCII as a 7-bit standard, how the high 128 positions became a collision zone between Latin-1 and Windows-1252, and why legacy text breaks when it moves between systems.

ASCII is only 7 bits

ASCII defines 128 code positions: 0-31 are control codes, 32-126 are printable, and 127 is delete. Anything above 127 is outside the standard, which is exactly where the trouble starts.

RangeContentsExamples
0x00-0x1FControl codesTab 09, LF 0A, CR 0D, ESC 1B
0x20-0x3FPunctuation and digitsSpace, !, 0-9
0x40-0x5FUppercase and symbolsA-Z, [, backslash, ]
0x60-0x7ELowercase and symbolsa-z, {, |, }, ~
0x7FDeleteOne code point
0x80-0xFFNot ASCIIInterpretation depends entirely on the encoding
b"A".decode("ascii")        # 'A'
chr(65)                     # 'A'
ord("A")                    # 65

b"\x80".decode("ascii")     # UnicodeDecodeError — outside the 7-bit range
b"\x80".decode("latin-1")   # '\x80' — Latin-1 maps every byte to a code point

The high half became a collision zone

Latin-1 (ISO-8859-1) fills 0x80-0xFF with Western European characters. Microsoft's Windows-1252 filled the same range with different glyphs, most famously putting curly quotes and the euro sign in positions Latin-1 leaves undefined.

ByteLatin-1Windows-1252Consequence
0x80Undefined controlEuro signA euro in 1252 becomes a control character in Latin-1
0x91Undefined controlLeft single quoteCurly quote renders as a box or a control code
0x92Undefined controlRight single quoteThe classic apostrophe corruption
0x93/0x94Undefined controlLeft/right double quoteQuotes turn into garbled pairs
0xE9e acutee acuteThe two agree here, which hides the problem
0xF7Division signDivision signAgreement again
# the same bytes, three different stories
raw = b"caf\xe9 \x92quoted\x92 \x80 10"

raw.decode("latin-1")        # 'café ’quoted’ € 10'
raw.decode("cp1252")         # 'café ’quoted’ € 10'  — the intended text
raw.decode("utf-8", "replace")  # 'caf��quoted...' — bytes are not valid UTF-8

Both encodings agree for most accented letters, so a file can look correct in the middle and wrong only where the bytes differ — which is why these bugs survive casual inspection.

Living with legacy text

  • Detect before decoding: try UTF-8 first, and fall back to a legacy page only when it fails.
  • Never decode as Latin-1 by default for user input — it never raises, so corruption is silent.
  • Convert once at the boundary, and store UTF-8 internally.
  • Record the source encoding alongside the data if you cannot convert immediately.
  • The Windows-1252 range 0x80-0x9F is the highest-risk zone; check it explicitly in tests.
def decode_legacy(raw: bytes) -> str:
    for enc in ("utf-8", "cp1252", "latin-1"):
        try:
            return raw.decode(enc)
        except UnicodeDecodeError:
            continue
    raise ValueError("no candidate encoding matched")
⚠️
The errors="replace" argument hides data loss behind a placeholder. Use it for display, never for storage — a decoded value that contains U+FFFD should be rejected or quarantined, not written back to the database.

FAQ

Is Latin-1 the same as ISO-8859-1?
Effectively yes for the byte to code point mapping. The IANA label latin-1 is treated as ISO-8859-1, but browsers map it to Windows-1252 for HTML, which is a documented historical inconsistency.
Why does my file open fine in one editor and break in another?
The editors guess differently. One assumes UTF-8, another uses the system code page. Add an explicit encoding declaration or BOM so the guess is unnecessary.

Bytes, bits, hex and number bases Mojibake: diagnosing and fixing broken text

Last refreshed 2026-09-18.