imported>Lashtear: blanked by request

2012-08-23T10:13:52Z

blanked by request

← Older revision		Revision as of 10:13, 23 August 2012
Line 1:		Line 1:
	As NBT files can be stored with a variety of compression types, it's sometimes useful to be able to transparently detect which is in use. This doesn't help for writes unless you statefully track where the NBT structure came from and how it was compressed, but it can simplify map export type tools.

	~~There are three types of storage currently possible:~~

	* GZip (defined in RFC 1952)
	* Zlib (defined in RFC 1950)
	* Uncompressed

	The two compressed types amount to [[wikipedia:DEFLATE\|DEFLATE]] plus a small header. Most languages have libraries or bindings for these already. It's possible to detect stream type by considering the first two bytes, however.

	~~== GZip ==~~
	~~GZip starts with a two-byte identification header: 0x31 0x8b. If it doesn't have that, it isn't GZip.~~

	~~== Zlib ==~~
	* Zlib starts with a two-byte header that must be decomposed into bits:

	~~{\| class="wikitable"~~
	~~! colspan="17" \| Zlib header bytes~~
	\|-
	~~! Byte offset~~
	~~\| colspan="8" \| 0~~
	~~\| colspan="8" \| 1~~
	\|-
	~~! Byte name~~
	~~\| colspan="8" \| CMF~~
	~~\| colspan="8" \| FLG~~
	\|-
	~~! Bit offset~~
	~~\| 7 \|\| 6 \|\| 5 \|\| 4 \|\| 3 \|\| 2 \|\| 1 \|\| 0 \|\| 7 \|\| 6 \|\| 5 \|\| 4 \|\| 3 \|\| 2 \|\| 1 \|\| 0~~
	\|-
	~~! Bit field name~~
	~~\| colspan="4" \| CINFO~~
	~~\| colspan="4" \| CM~~
	~~\| colspan="2" \| FLEVEL~~
	~~\| colspan="1" \| FDICT~~
	~~\| colspan="5" \| FCHECK~~
	\|}

	~~Just make sure that these fields obey the following invariants:~~
	* <code>CM == 8</code> (compression method is deflate)
	* <code>CINFO <=7</code> (log<sub>2</sub> of window size)
	* The entire header as a big-endian 16-bit word is divisible by 31. (i.e. <code>(CMF*256+FLG) % 31 == 0</code>). The FCHECK field exists solely to allow the header to be set this way.

	~~Technically, other methods beyond DEFLATE could be implemented, but they haven't been added to this format standard. As a result, it's reasonably safe to consider only the DEFLATE case of Zlib.~~

	~~== Uncompressed ==~~

	NBT files are required by the standard to start with a TAG_Compound, which is 0x0a. That's only one byte of identification, but it helps as a basic sanity test. As all known root-tag-names are less than 256 codepoints long, NBT streams should all start with 0x0a 0x00, but longer names are legal.

imported>Lashtear: Created page with "As NBT files can be stored with a variety of compression types, it's sometimes useful to be able to transparently detect which is in use. This doesn't help for writes unless ..."

2012-08-23T09:48:19Z

Created page with "As NBT files can be stored with a variety of compression types, it's sometimes useful to be able to transparently detect which is in use. This doesn't help for writes unless ..."

New page

As NBT files can be stored with a variety of compression types, it's sometimes useful to be able to transparently detect which is in use. This doesn't help for writes unless you statefully track where the NBT structure came from and how it was compressed, but it can simplify map export type tools.

There are three types of storage currently possible:

* GZip (defined in RFC 1952)
* Zlib (defined in RFC 1950)
* Uncompressed

The two compressed types amount to [[wikipedia:DEFLATE|DEFLATE]] plus a small header. Most languages have libraries or bindings for these already. It's possible to detect stream type by considering the first two bytes, however.

== GZip ==
GZip starts with a two-byte identification header: 0x31 0x8b. If it doesn't have that, it isn't GZip.

== Zlib ==
* Zlib starts with a two-byte header that must be decomposed into bits:

{| class="wikitable"
! colspan="17" | Zlib header bytes
|-
! Byte offset
| colspan="8" | 0
| colspan="8" | 1
|-
! Byte name
| colspan="8" | CMF
| colspan="8" | FLG
|-
! Bit offset
| 7 || 6 || 5 || 4 || 3 || 2 || 1 || 0 || 7 || 6 || 5 || 4 || 3 || 2 || 1 || 0
|-
! Bit field name
| colspan="4" | CINFO
| colspan="4" | CM
| colspan="2" | FLEVEL
| colspan="1" | FDICT
| colspan="5" | FCHECK
|}

Just make sure that these fields obey the following invariants:
* <code>CM == 8</code> (compression method is deflate)
* <code>CINFO <=7</code> (log<sub>2</sub> of window size)
* The entire header as a big-endian 16-bit word is divisible by 31. (i.e. <code>(CMF*256+FLG) % 31 == 0</code>). The FCHECK field exists solely to allow the header to be set this way.

Technically, other methods beyond DEFLATE could be implemented, but they haven't been added to this format standard. As a result, it's reasonably safe to consider only the DEFLATE case of Zlib.

== Uncompressed ==

NBT files are required by the standard to start with a TAG_Compound, which is 0x0a. That's only one byte of identification, but it helps as a basic sanity test. As all known root-tag-names are less than 256 codepoints long, NBT streams should all start with 0x0a 0x00, but longer names are legal.

NBT compression - Revision history

imported>Lashtear: blanked by request

imported>Lashtear: Created page with "As NBT files can be stored with a variety of compression types, it's sometimes useful to be able to transparently detect which is in use. This doesn't help for writes unless ..."