Bytes, code points, and text boundaries

The idea

Lua strings are byte sequences and can contain arbitrary bytes. The length operator counts bytes, while ordinary string indexing functions use byte positions. A visually short Unicode string can therefore occupy several bytes. State whether an operation concerns storage size, Unicode code points, or user-perceived characters before choosing its indexing rules.

Lua 5.4’s utf8 library helps traverse valid UTF-8 sequences and count code points. A code point is still not necessarily one displayed character: combining marks can join a base character, and other display sequences may contain several code points. Validate encoding when the input contract requires UTF-8, and avoid string.sub boundaries that split a multibyte sequence. Do not promise grapheme-aware behavior from a byte-based helper.

local text=utf8.char(0xE9)
print(#text)
print(utf8.len(text))
print(utf8.codepoint(text))

2
1
233

The code point U+00E9 uses two UTF-8 bytes but remains one code point with numeric value 233.