Byte Order Mark
A Byte Order Mark (BOM) is a strategically-placed U+FEFF (ZERO WIDTH NO-BREAK SPACE) character at the beginning of a Unicode text file, or other block of Unicode text.
Contents |
Discussion
There are two main schools of thought as to its purpose:
- Its purpose is to identify the endianness of a file whose encoding is otherwise already known (particularly useful with UTF-16).
- Its purpose is more general: to help computer programs guess the encoding of a file, even if they have no external information about what its encoding might be. Thus, the term "byte order mark" is something of a misnomer.
The idea of a BOM is undeniably a hack, but its benefits sometimes outweigh its drawbacks.
To make false positives less likely, the U+FFFE code point is permanently reserved, and will never be a meaningful code point.
Other usage of the U+FEFF character is deprecated, and U+2060 WORD JOINER is suggested instead.
Byte patterns of common BOMs
A file beginning with bytes 0xFE 0xFF is probably encoded in UTF-16 with big-endian byte order.
0xFF 0xFE suggests UTF-16 with little-endian byte order.
0xEF 0xBB 0xBF suggests UTF-8.
UTF-8
Whether UTF-8 files should ever use a BOM is a contentious issue. A good case can be made for either side of the argument. But note that if you need to read files written by third-party applications, that ship has sailed: existing UTF-8 files often do use a BOM.
