Plain text - Revision history

265 993 303: Unicode as of 17.0 still does not include U+0A00 or U+0A0D

2025-09-13T09:59:20Z

Unicode as of 17.0 still does not include U+0A00 or U+0A0D

265 993 303: Unicode as of 16.0 still does not include U+0A00 or U+0A0D, so the heuristic still works

2024-09-10T21:22:35Z

Unicode as of 16.0 still does not include U+0A00 or U+0A0D, so the heuristic still works

Dexvertbot: Added sample files

2023-12-28T14:55:35Z

Added sample files

265 993 303 at 18:24, 13 September 2023

2023-09-13T18:24:21Z

265 993 303: 0xFEFF is byte order mark

2023-08-23T05:34:39Z

0xFEFF is byte order mark

Foxtrot: /* Identification */

2023-06-20T12:25:10Z

‎Identification

265 993 303 at 10:45, 20 June 2023

2023-06-20T10:45:33Z

Dan Tobias at 00:24, 2 June 2019

2019-06-02T00:24:12Z

Jsummers at 15:41, 6 April 2018

2018-04-06T15:41:24Z

Dan Tobias: /* Extension */

2017-03-17T23:58:44Z

‎Extension

@@ Line 27: / Line 27: @@
 [[UTF-32]] text files are arrays of 32-bit integers representing Unicode code points and are usually detected by starting with the ''Byte Order Mark'' (BOM) consisting of the bytes <code>FF FE 00 00</code> (for little endian <code>0x0000FEFF</code>) or <code>00 00 FE FF</code> (for big endian <code>0x0000FEFF</code>). In some cases UTF-32 files may occur without the BOM, however, only <code>0x00000000</code>—<code>0x0000D7FF</code> and <code>0x0000E000</code>—<code>0x0010FFFF</code> are valid ranges for dwords; <code>0x0000D800</code>—<code>0x0000DFFF</code> and <code>0x00110000</code>—<code>0xFFFFFFFF</code> are invalid.
-[[UTF-16]] text files are arrays of 16-bit integers representing code units and are usually detected by starting with the byte order mark (BOM) consisting of the bytes <code>FF FE</code> (for little endian <code>0xFEFF</code>) or <code>FE FF</code> (for big endian <code>0xFEFF</code>). However, in some cases UTF-16 files may occur without the BOM, in which case, detection is not guaranteed to be reliable, but the line feed (<code>0x000A</code>) in its byte reversal (<code>0x0A00</code>) is not in ''Unicode 16.0'', and null bytes are unlikely to occur in other text encodings, so the presence of word-aligned <code>00 0A</code> or <code>0A 00</code> can rule out 8-bit encodings and one of the endianness and therefore may be used for UTF-16 detection. On the other hand, the bytes <code>0D 0A</code> in little endian form <code>U+0A0D</code> which is not in ''Unicode 16.0'' either but it is a common newline in 8-bit encodings. The detection of [[UCS-2]] text works similarly, since UCS-2 is the precursor of UTF-16, as UTF-16 introduced surrogate pairs formed by <code>0xD800</code>—<code>0xDBFF</code> followed by <code>0xDC00</code>—<code>0xDFFF</code>, with other combinations of <code>0xD800</code>—<code>0xDFFF</code> being invalid.
+[[UTF-16]] text files are arrays of 16-bit integers representing code units and are usually detected by starting with the byte order mark (BOM) consisting of the bytes <code>FF FE</code> (for little endian <code>0xFEFF</code>) or <code>FE FF</code> (for big endian <code>0xFEFF</code>). However, in some cases UTF-16 files may occur without the BOM, in which case, detection is not guaranteed to be reliable, but the line feed (<code>0x000A</code>) in its byte reversal (<code>0x0A00</code>) is not in ''Unicode 17.0'', and null bytes are unlikely to occur in other text encodings, so the presence of word-aligned <code>00 0A</code> or <code>0A 00</code> can rule out 8-bit encodings and one of the endianness and therefore may be used for UTF-16 detection. On the other hand, the bytes <code>0D 0A</code> in little endian form <code>U+0A0D</code> which is not in ''Unicode 17.0'' either but it is a common newline in 8-bit encodings. The detection of [[UCS-2]] text works similarly, since UCS-2 is the precursor of UTF-16, as UTF-16 introduced surrogate pairs formed by <code>0xD800</code>—<code>0xDBFF</code> followed by <code>0xDC00</code>—<code>0xDFFF</code>, with other combinations of <code>0xD800</code>—<code>0xDFFF</code> being invalid.
 [[ASCII|ASCII-only]] text files may be detected by verifying that the file has all <code>0x01</code>—<code>0x7F</code> bytes. <code>0x80</code>—<code>0xFF</code> are not used in ASCII encoding, and null characters by <code>0x00</code> are not typically found in plain text; null bytes are much more likely to be in UTF-16 or UTF-32 text.

@@ Line 27: / Line 27: @@
 [[UTF-32]] text files are arrays of 32-bit integers representing Unicode code points and are usually detected by starting with the ''Byte Order Mark'' (BOM) consisting of the bytes <code>FF FE 00 00</code> (for little endian <code>0x0000FEFF</code>) or <code>00 00 FE FF</code> (for big endian <code>0x0000FEFF</code>). In some cases UTF-32 files may occur without the BOM, however, only <code>0x00000000</code>—<code>0x0000D7FF</code> and <code>0x0000E000</code>—<code>0x0010FFFF</code> are valid ranges for dwords; <code>0x0000D800</code>—<code>0x0000DFFF</code> and <code>0x00110000</code>—<code>0xFFFFFFFF</code> are invalid.
-[[UTF-16]] text files are arrays of 16-bit integers representing code units and are usually detected by starting with the byte order mark (BOM) consisting of the bytes <code>FF FE</code> (for little endian <code>0xFEFF</code>) or <code>FE FF</code> (for big endian <code>0xFEFF</code>). However, in some cases UTF-16 files may occur without the BOM, in which case, detection is not guaranteed to be reliable, but the line feed (<code>0x000A</code>) in its byte reversal (<code>0x0A00</code>) is not in ''Unicode 15.1'', and null bytes are unlikely to occur in other text encodings, so the presence of word-aligned <code>00 0A</code> or <code>0A 00</code> can rule out 8-bit encodings and one of the endianness and therefore may be used for UTF-16 detection. On the other hand, the bytes <code>0D 0A</code> in little endian form <code>U+0A0D</code> which is not in ''Unicode 15.1'' either but it is a common newline in 8-bit encodings. The detection of [[UCS-2]] text works similarly, since UCS-2 is the precursor of UTF-16, as UTF-16 introduced surrogate pairs formed by <code>0xD800</code>—<code>0xDBFF</code> followed by <code>0xDC00</code>—<code>0xDFFF</code>, with other combinations of <code>0xD800</code>—<code>0xDFFF</code> being invalid.
+[[UTF-16]] text files are arrays of 16-bit integers representing code units and are usually detected by starting with the byte order mark (BOM) consisting of the bytes <code>FF FE</code> (for little endian <code>0xFEFF</code>) or <code>FE FF</code> (for big endian <code>0xFEFF</code>). However, in some cases UTF-16 files may occur without the BOM, in which case, detection is not guaranteed to be reliable, but the line feed (<code>0x000A</code>) in its byte reversal (<code>0x0A00</code>) is not in ''Unicode 16.0'', and null bytes are unlikely to occur in other text encodings, so the presence of word-aligned <code>00 0A</code> or <code>0A 00</code> can rule out 8-bit encodings and one of the endianness and therefore may be used for UTF-16 detection. On the other hand, the bytes <code>0D 0A</code> in little endian form <code>U+0A0D</code> which is not in ''Unicode 16.0'' either but it is a common newline in 8-bit encodings. The detection of [[UCS-2]] text works similarly, since UCS-2 is the precursor of UTF-16, as UTF-16 introduced surrogate pairs formed by <code>0xD800</code>—<code>0xDBFF</code> followed by <code>0xDC00</code>—<code>0xDFFF</code>, with other combinations of <code>0xD800</code>—<code>0xDFFF</code> being invalid.
 [[ASCII|ASCII-only]] text files may be detected by verifying that the file has all <code>0x01</code>—<code>0x7F</code> bytes. <code>0x80</code>—<code>0xFF</code> are not used in ASCII encoding, and null characters by <code>0x00</code> are not typically found in plain text; null bytes are much more likely to be in UTF-16 or UTF-32 text.
-[[UTF-8]] text files may be detected by presence of any bytes from <code>0x80</code>—<code>0xFF</code> (to avoid UTF-8 for ASCII-only files), absence of null bytes (if UTF-16 and UTF-32 haven't been ruled out yet), and verifying that the file is valid UTF-8. UTF-8 has many error cases; the only valid bit patterns are <code>0xxxxxxx</code> (where x forms <code>0x00</code>—<code>0x7F</code>), <code>110xxxxx</code> <code>10xxxxxx</code> (where x forms <code>0x0080</code>—<code>0x07FF</code>, but not <code>0x00</code>—<code>0x7F</code>), <code>1110xxxx</code> <code>10xxxxxx</code> <code>10xxxxxx</code> (where x forms <code>0x0800</code>—<code>0xD7FF</code> <code>0xE000</code>—<code>0xFFFF</code>, but not <code>0x0000</code>—<code>0x07FF</code> or <code>0xD800</code>—<code>0xDFFF</code>), and <code>11110xxx</code> <code>10xxxxxx</code> <code>10xxxxxx</code> <code>10xxxxxx</code> (where x forms <code>0x10000</code>—<code>0x10FFFF</code>, but not <code>0x0000</code>—<code>0xFFFF</code> or <code>0x110000</code>—<code>0x1FFFFF</code>). UTF-8 text files may also start with the UTF-8 byte order mark (EF BB BF), but should still be verified for validity.
+[[UTF-8]] text files may be detected by presence of any bytes from <code>0x80</code>—<code>0xFF</code> (to avoid processing ASCII-only files as UTF-8), absence of null bytes (if UTF-16 and UTF-32 haven't been ruled out yet), and verifying that the file is valid UTF-8. UTF-8 has many error cases; the only valid bit patterns are <code>0xxxxxxx</code> (where x forms <code>0x00</code>—<code>0x7F</code>), <code>110xxxxx</code> <code>10xxxxxx</code> (where x forms <code>0x0080</code>—<code>0x07FF</code>, but not <code>0x00</code>—<code>0x7F</code>), <code>1110xxxx</code> <code>10xxxxxx</code> <code>10xxxxxx</code> (where x forms <code>0x0800</code>—<code>0xD7FF</code> <code>0xE000</code>—<code>0xFFFF</code>, but not <code>0x0000</code>—<code>0x07FF</code> or <code>0xD800</code>—<code>0xDFFF</code>), and <code>11110xxx</code> <code>10xxxxxx</code> <code>10xxxxxx</code> <code>10xxxxxx</code> (where x forms <code>0x10000</code>—<code>0x10FFFF</code>, but not <code>0x0000</code>—<code>0xFFFF</code> or <code>0x110000</code>—<code>0x1FFFFF</code>). UTF-8 text files may also start with the UTF-8 byte order mark (EF BB BF), but should still be verified for validity.
 When a file is known to be a plain text file but [[UTF-32]], [[UTF-16]], [[ASCII]], and [[UTF-8]] were already ruled out, only 8-bit encodings or mixed single byte/double byte encodings (such as [[JIS|Shift JIS]]) remain. In this case, the only thing left (other than applying complex heuristics) is to use the regional or system text encoding, such as [[Windows 1252|CP1252]], [[Windows 1250|CP1250]], [[CP437]], [[CP852]], etc..

@@ Line 40: / Line 40: @@
 == Software ==
 * [http://textract.readthedocs.org/en/latest/ Textract: extract text from various document formats]
 == Links and References ==

@@ Line 25: / Line 25: @@
 == Identification ==
-[[UTF-32]] text files are usually detected by starting with the ''Byte Order Mark'' (BOM) consisting of the bytes <code>FF FE 00 00</code> (for little endian <code>0x0000FEFF</code>) or <code>00 00 FE FF</code> (for big endian <code>0x0000FEFF</code>). In some cases UTF-32 files may occur without the BOM, however, only <code>0x00000000</code>—<code>0x0000D7FF</code> and <code>0x0000E000</code>—<code>0x0010FFFF</code> are valid ranges for dwords; <code>0x0000D800</code>—<code>0x0000DFFF</code> and <code>0x00110000</code>—<code>0xFFFFFFFF</code> are invalid.
+[[UTF-32]] text files are arrays of 32-bit integers representing Unicode code points and are usually detected by starting with the ''Byte Order Mark'' (BOM) consisting of the bytes <code>FF FE 00 00</code> (for little endian <code>0x0000FEFF</code>) or <code>00 00 FE FF</code> (for big endian <code>0x0000FEFF</code>). In some cases UTF-32 files may occur without the BOM, however, only <code>0x00000000</code>—<code>0x0000D7FF</code> and <code>0x0000E000</code>—<code>0x0010FFFF</code> are valid ranges for dwords; <code>0x0000D800</code>—<code>0x0000DFFF</code> and <code>0x00110000</code>—<code>0xFFFFFFFF</code> are invalid.
-[[UTF-16]] text files are usually detected by starting with the byte order mark (BOM) consisting of the bytes <code>FF FE</code> (for little endian <code>0xFEFF</code>) or FE FF (for big endian <code>0xFEFF</code>). However, in some cases UTF-16 files may occur without the BOM, in which case, detection is not guaranteed to be reliable, but the line feed (<code>0x000A</code>) in its byte reversal (<code>0x0A00</code>) is not in ''Unicode 15.0'', and null bytes are unlikely to occur in other text encodings, so the presence of word-aligned <code>00 0A</code> or <code>0A 00</code> can rule out 8-bit encodings and one of the endianness and therefore may be used for UTF-16 detection. On the other hand, the bytes <code>0D 0A</code> in little endian form <code>U+0A0D</code> which is not in ''Unicode 15.0'' either but it is a common newline in 8-bit encodings. The detection of [[UCS-2]] text works similarly, since UCS-2 is the precursor of UTF-16, as UTF-16 introduced surrogate pairs formed by <code>0xD800</code>—<code>0xDBFF</code> followed by <code>0xDC00</code>—<code>0xDFFF</code>, with other combinations of <code>0xD800</code>—<code>0xDFFF</code> being invalid.
+[[UTF-16]] text files are arrays of 16-bit integers representing code units and are usually detected by starting with the byte order mark (BOM) consisting of the bytes <code>FF FE</code> (for little endian <code>0xFEFF</code>) or <code>FE FF</code> (for big endian <code>0xFEFF</code>). However, in some cases UTF-16 files may occur without the BOM, in which case, detection is not guaranteed to be reliable, but the line feed (<code>0x000A</code>) in its byte reversal (<code>0x0A00</code>) is not in ''Unicode 15.1'', and null bytes are unlikely to occur in other text encodings, so the presence of word-aligned <code>00 0A</code> or <code>0A 00</code> can rule out 8-bit encodings and one of the endianness and therefore may be used for UTF-16 detection. On the other hand, the bytes <code>0D 0A</code> in little endian form <code>U+0A0D</code> which is not in ''Unicode 15.1'' either but it is a common newline in 8-bit encodings. The detection of [[UCS-2]] text works similarly, since UCS-2 is the precursor of UTF-16, as UTF-16 introduced surrogate pairs formed by <code>0xD800</code>—<code>0xDBFF</code> followed by <code>0xDC00</code>—<code>0xDFFF</code>, with other combinations of <code>0xD800</code>—<code>0xDFFF</code> being invalid.
-[[ASCII|ASCII-only]] text files may be detected by verifying that the file has all <code>0x01</code>—<code>0x7F</code> bytes.
+[[ASCII|ASCII-only]] text files may be detected by verifying that the file has all <code>0x01</code>—<code>0x7F</code> bytes. <code>0x80</code>—<code>0xFF</code> are not used in ASCII encoding, and null characters by <code>0x00</code> are not typically found in plain text; null bytes are much more likely to be in UTF-16 or UTF-32 text.
-[[UTF-8]] text files may be detected by presence of any bytes from <code>0x80</code>—<code>0xFF</code>, absence of null bytes (if UTF-16 hasn't been ruled out yet), or verifying that the file is valid UTF-8. UTF-8 has many error cases; the only valid bit patterns are <code>0xxxxxxx</code> (where x forms <code>0x00</code>—<code>0x7F</code>), <code>110xxxxx</code> <code>10xxxxxx</code> (where x forms <code>0x0080</code>—<code>0x07FF</code>, but not <code>0x00</code>—<code>0x7F</code>), <code>1110xxxx</code> <code>10xxxxxx</code> <code>10xxxxxx</code> (where x forms <code>0x0800</code>—<code>0xD7FF</code> <code>0xE000</code>—<code>0xFFFF</code>, but not <code>0x0000</code>—<code>0x07FF</code> or <code>0xD800</code>—<code>0xDFFF</code>), and <code>11110xxx</code> <code>10xxxxxx</code> <code>10xxxxxx</code> <code>10xxxxxx</code> (where x forms <code>0x10000</code>—<code>0x10FFFF</code>, but not <code>0x0000</code>—<code>0xFFFF</code> or <code>0x110000</code>—<code>0x1FFFFF</code>). UTF-8 text files may also start with the UTF-8 byte order mark (EF BB BF).
+[[UTF-8]] text files may be detected by presence of any bytes from <code>0x80</code>—<code>0xFF</code> (to avoid UTF-8 for ASCII-only files), absence of null bytes (if UTF-16 and UTF-32 haven't been ruled out yet), and verifying that the file is valid UTF-8. UTF-8 has many error cases; the only valid bit patterns are <code>0xxxxxxx</code> (where x forms <code>0x00</code>—<code>0x7F</code>), <code>110xxxxx</code> <code>10xxxxxx</code> (where x forms <code>0x0080</code>—<code>0x07FF</code>, but not <code>0x00</code>—<code>0x7F</code>), <code>1110xxxx</code> <code>10xxxxxx</code> <code>10xxxxxx</code> (where x forms <code>0x0800</code>—<code>0xD7FF</code> <code>0xE000</code>—<code>0xFFFF</code>, but not <code>0x0000</code>—<code>0x07FF</code> or <code>0xD800</code>—<code>0xDFFF</code>), and <code>11110xxx</code> <code>10xxxxxx</code> <code>10xxxxxx</code> <code>10xxxxxx</code> (where x forms <code>0x10000</code>—<code>0x10FFFF</code>, but not <code>0x0000</code>—<code>0xFFFF</code> or <code>0x110000</code>—<code>0x1FFFFF</code>). UTF-8 text files may also start with the UTF-8 byte order mark (EF BB BF), but should still be verified for validity.
 When a file is known to be a plain text file but [[UTF-32]], [[UTF-16]], [[ASCII]], and [[UTF-8]] were already ruled out, only 8-bit encodings or mixed single byte/double byte encodings (such as [[JIS|Shift JIS]]) remain. In this case, the only thing left (other than applying complex heuristics) is to use the regional or system text encoding, such as [[Windows 1252|CP1252]], [[Windows 1250|CP1250]], [[CP437]], [[CP852]], etc..

@@ Line 25: / Line 25: @@
 == Identification ==
-[[UTF-32]] text files are usually detected by starting with the ''Byte Order Mark'' (BOM) consisting of the bytes <code>FF FE 00 00</code> (for little endian <code>0x0000FEFF</code>) or <code>00 00 FE FF</code> (for big endian <code>0x0000FFFE</code>). In some cases UTF-32 files may occur without the BOM, however, only <code>0x00000000</code>—<code>0x0000D7FF</code> and <code>0x0000E000</code>—<code>0x0010FFFF</code> are valid ranges for dwords; <code>0x0000D800</code>—<code>0x0000DFFF</code> and <code>0x00110000</code>—<code>0xFFFFFFFF</code> are invalid.
+[[UTF-32]] text files are usually detected by starting with the ''Byte Order Mark'' (BOM) consisting of the bytes <code>FF FE 00 00</code> (for little endian <code>0x0000FEFF</code>) or <code>00 00 FE FF</code> (for big endian <code>0x0000FEFF</code>). In some cases UTF-32 files may occur without the BOM, however, only <code>0x00000000</code>—<code>0x0000D7FF</code> and <code>0x0000E000</code>—<code>0x0010FFFF</code> are valid ranges for dwords; <code>0x0000D800</code>—<code>0x0000DFFF</code> and <code>0x00110000</code>—<code>0xFFFFFFFF</code> are invalid.
-[[UTF-16]] text files are usually detected by starting with the byte order mark (BOM) consisting of the bytes <code>FF FE</code> (for little endian <code>0xFEFF</code>) or FE FF (for big endian <code>0xFFFE</code>). However, in some cases UTF-16 files may occur without the BOM, in which case, detection is not guaranteed to be reliable, but the line feed (<code>0x000A</code>) in its byte reversal (<code>0x0A00</code>) is not in ''Unicode 15.0'', and null bytes are unlikely to occur in other text encodings, so the presence of word-aligned <code>00 0A</code> or <code>0A 00</code> can rule out 8-bit encodings and one of the endianness and therefore may be used for UTF-16 detection. On the other hand, the bytes <code>0D 0A</code> in little endian form <code>U+0A0D</code> which is not in ''Unicode 15.0'' either but it is a common newline in 8-bit encodings. The detection of [[UCS-2]] text works similarly, since UCS-2 is the precursor of UTF-16, as UTF-16 introduced surrogate pairs formed by <code>0xD800</code>—<code>0xDBFF</code> followed by <code>0xDC00</code>—<code>0xDFFF</code>, with other combinations of <code>0xD800</code>—<code>0xDFFF</code> being invalid.
+[[UTF-16]] text files are usually detected by starting with the byte order mark (BOM) consisting of the bytes <code>FF FE</code> (for little endian <code>0xFEFF</code>) or FE FF (for big endian <code>0xFEFF</code>). However, in some cases UTF-16 files may occur without the BOM, in which case, detection is not guaranteed to be reliable, but the line feed (<code>0x000A</code>) in its byte reversal (<code>0x0A00</code>) is not in ''Unicode 15.0'', and null bytes are unlikely to occur in other text encodings, so the presence of word-aligned <code>00 0A</code> or <code>0A 00</code> can rule out 8-bit encodings and one of the endianness and therefore may be used for UTF-16 detection. On the other hand, the bytes <code>0D 0A</code> in little endian form <code>U+0A0D</code> which is not in ''Unicode 15.0'' either but it is a common newline in 8-bit encodings. The detection of [[UCS-2]] text works similarly, since UCS-2 is the precursor of UTF-16, as UTF-16 introduced surrogate pairs formed by <code>0xD800</code>—<code>0xDBFF</code> followed by <code>0xDC00</code>—<code>0xDFFF</code>, with other combinations of <code>0xD800</code>—<code>0xDFFF</code> being invalid.
 [[ASCII|ASCII-only]] text files may be detected by verifying that the file has all <code>0x01</code>—<code>0x7F</code> bytes.

← Older revision		Revision as of 10:45, 20 June 2023
Line 22:		Line 22:

	The traditional extension for text files is <code>.txt</code>, but lots of other extensions have been used. Occasionally on systems permitting extensions longer than three letters, <code>.text</code> has been used, and <code>.asc</code> for ASCII has also had some use; <code>.doc</code> has also sometimes been used for files "documenting" something (like the manual accompanying a piece of downloaded software), but that went out of common use once that extension became associated with Microsoft Word's [[DOC]] format.		The traditional extension for text files is <code>.txt</code>, but lots of other extensions have been used. Occasionally on systems permitting extensions longer than three letters, <code>.text</code> has been used, and <code>.asc</code> for ASCII has also had some use; <code>.doc</code> has also sometimes been used for files "documenting" something (like the manual accompanying a piece of downloaded software), but that went out of common use once that extension became associated with Microsoft Word's [[DOC]] format.
		+
		+	== Identification ==
		+
		+	UTF-32 text files are usually detected by starting with the byte order mark (BOM) consisting of the bytes FF FE 00 00 (for little endian 0x0000FEFF) or 00 00 FE FF (for big endian 0x0000FFFE). In some cases UTF-32 files may occur without the BOM, however, only 0x00000000—0x0000D7FF and 0x0000E000—0x0010FFFF are valid ranges for dwords; 0x0000D800—0x0000DFFF and 0x00110000—0xFFFFFFFF are invalid.
		+
		+	UTF-16 text files are usually detected by starting with the byte order mark (BOM) consisting of the bytes FF FE (for little endian 0xFEFF) or FE FF (for big endian 0xFFFE). However, in some cases UTF-16 files may occur without the BOM, in which case, detection is not guaranteed to be reliable, but the line feed (0x000A) in its byte reversal (0x0A00) is not in Unicode 15.0, and null bytes are unlikely to occur in other text encodings, so the presence of word-aligned 00 0A or 0A 00 can rule out 8-bit encodings and one of the endianness and therefore may be used for UTF-16 detection. On the other hand, the bytes 0D 0A in little endian form U+0A0D which is not in Unicode 15.0 either but it is a common newline in 8-bit encodings. The detection of UCS-2 text works similarly, since UCS-2 is the precursor of UTF-16, as UTF-16 introduced surrogate pairs formed by 0xD800—0xDBFF followed by 0xDC00—0xDFFF, with other combinations of 0xD800—0xDFFF being invalid.
		+
		+	ASCII only text files may be detected by verifying that the file has all 0x01—0x7F bytes.
		+
		+	UTF-8 text files may be detected by presence of any bytes from 0x80—0xFF, absence of null bytes (if UTF-16 hasn't been ruled out yet), or verifying that the file is valid UTF-8. UTF-8 has many error cases; the only valid bit patterns are 0xxxxxxx (where x forms 0x00—0x7F), 110xxxxx 10xxxxxx (where x forms 0x0080—0x07FF, but not 0x00—0x7F), 1110xxxx 10xxxxxx 10xxxxxx (where x forms 0x0800—0xD7FF 0xE000—0xFFFF, but not 0x0000—0x07FF or 0xD800—0xDFFF), and 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx (where x forms 0x10000—0x10FFFF, but not 0x0000—0xFFFF or 0x110000—0x1FFFFF). UTF-8 text files may also start with the UTF-8 byte order mark (EF BB BF).
		+
		+	When a file is known to be a plain text file but UTF-32, UTF-16, ASCII, and UTF-8 were already ruled out, only 8-bit encodings or mixed single byte/double byte encodings (such as Shift JIS) remain. In this case, the only thing left (other than applying complex heuristics) is to use the regional or system text encoding, such as CP1252, CP1250, CP437, CP852, etc..

	== See also ==		== See also ==

@@ Line 5: / Line 5: @@
 |mimetypes={{mimetype|text/plain}}
 |pronom={{PRONOM|x-fmt/111}}
 }}
 '''Plain text''' files (also known by the extension TXT) consist of characters encoded sequentially in some particular [[character encoding]]. Plain text files contain no formatting information other than white space characters. Some data formats (usually those intended to be human-readable) are based on plain text; see [[Text-based data]] for some structured formats that are stored in plain text (and hence can be opened in a plain text editor if no more specific program is available).

@@ Line 2: / Line 2: @@
 |formattype=electronic
 |subcat=Document
-|extensions={{ext|txt}}, {{ext|text}}, {{ext|doc}}, {{ext|asc}} {{noext}}
+|extensions={{ext|txt}}, {{ext|text}}, {{ext|doc}}, {{ext|asc}}, {{noext}}, many others
 |mimetypes={{mimetype|text/plain}}
 |pronom={{PRONOM|x-fmt/111}}

@@ Line 20: / Line 20: @@
 == Extension ==
-The traditional extension for text files is <code>.txt</code>, but lots of other extensions have been used. Occasionally on systems permitting extensions longer than three letters, <code>.text</code> has been used, and <code>.asc</code> for ASCII has also had some use. <code>.doc</code> has also sometimes been used for files "documenting" something (like the manual accompanying a piece of downloaded software), but that went out of common use once that extension became associated with Microsoft Word's [[DOC]] format.
+The traditional extension for text files is <code>.txt</code>, but lots of other extensions have been used. Occasionally on systems permitting extensions longer than three letters, <code>.text</code> has been used, and <code>.asc</code> for ASCII has also had some use; <code>.doc</code> has also sometimes been used for files "documenting" something (like the manual accompanying a piece of downloaded software), but that went out of common use once that extension became associated with Microsoft Word's [[DOC]] format.
 == See also ==