Avro - Revision history

Effect2: /* Single-Object Encoding */ Clarified that the fingerprint is of the schema, not (it might be wrongly assumed) of the file

2019-06-23T11:58:29Z

‎Single-Object Encoding: Clarified that the fingerprint is of the schema, not (it might be wrongly assumed) of the file

Effect2: Fixed another link

2019-06-14T06:31:41Z

Fixed another link

Effect2: Fixed link

2019-06-14T06:31:07Z

Fixed link

Effect2: Made magic number formats consistent, and added the omitted byte from the object container magic

2019-06-14T02:38:50Z

Made magic number formats consistent, and added the omitted byte from the object container magic

Effect2: Broke off the section about JSON encoding from the main flow

2019-06-14T00:03:48Z

Broke off the section about JSON encoding from the main flow

Effect2: Created page with "{{FormatInfo |formattype=electronic |subcat=Serialization |extensions={{ext|avsc}}, {{ext|avro}} |released=2009Date of the earliest release, i.e. https://archive.apache.o..."

2019-06-14T00:00:02Z

Created page with "{{FormatInfo |formattype=electronic |subcat=Serialization |extensions={{ext|avsc}}, {{ext|avro}} |released=2009<ref>Date of the earliest release, i.e. https://archive.apache.o..."

New page

{{FormatInfo
|formattype=electronic
|subcat=Serialization
|extensions={{ext|avsc}}, {{ext|avro}}
|released=2009<ref>Date of the earliest release, i.e. https://archive.apache.org/dist/avro/avro-1.0.0/</ref>
}}

'''Avro''', commonly known prefixed by the name of its creator organization as "'''Apache Avro'''", is a [[serialization]] format (and RPC system) used largely within "big data" systems, specifically the Apache Hadoop environment. It has been designed for extreme "flexibility", which seems to actually mean that it is meant to meet several apparently contradictory requirements at once, and, as a result, it has a somewhat confusing structure.

At its basis, Avro is an imitation of [[Protocol Buffers]], with a schema written is [[JSON]] that defines a wire format<ref group="Note">In this case, "wire format" is meant in the generic sense, not Avro's specific usage of it to refer to its network protocol.</ref> for serialized data that essentially consists of the field values converted to binary and concatendated. Instead of this binary encoding, the message itself may also be serialized using JSON.<ref>https://avro.apache.org/docs/current/spec.html#Encodings</ref>. However, this becomes complicated by the fact that its schema is usually stored with the message, which gives rise to Avro's distinctive qualities.

One of the features frequently touted as an advantage of Avro is that it can work without Protocol Buffers-like code generation<ref>[[Wikipedia:Apache Avro]]</ref><ref>http://cloudurable.com/blog/avro/index.html</ref> (although in practice this seems to usually be done, anyhow). This only affects ease of programming, not the data format. (Technically, it allows for messages to be parsed without knowing anything about the schema in advance, but this is only useful in the course of writing something like a debug tool.)

== Extensions ==
Schema files use the extension ".avsc", while encoded files (of any sort) use the extension ".avro".<ref>https://avro.apache.org/docs/current/gettingstartedpython.html</ref>

== Parsing Canonical Form ==
The "parsing canocial form" is used to test for equality of two schemas. It is computed by taking a schema, specified in JSON, and performing a number of transformations on it. If the result of this process is the same for two schemas, they are considered identical.<ref>https://avro.apache.org/docs/current/spec.html#Parsing+Canonical+Form+for+Schemas</ref> The parsing canocial form becomes important when schemas are embedded in files (as is discussed in the next section).

== Embedded Schemas and their Consequences ==
The distinctive feature of Avro is that it almost always stores or transmits the serialized message along with its schema. This is how Avro tries to solve the problem of backwards and forwards compatibility in schema versions (a difficult problem for serialization formats; see e.g. <ref>https://capnproto.org/faq.html#how-do-i-make-a-field-required-like-in-protocol-buffers</ref>).

=== The Wrapper Formats ===
The schema and the message are stored together with two formats: the "single-object encoding" format and the "object container file" format. The former does not actually store the schema in full, only a reference.

==== Single-Object Encoding ====
The "single-object encoding" consists of<ref>https://avro.apache.org/docs/current/spec.html#single_object_encoding</ref>:
* The magic number {{magic|0xC3 0x01}}.
* A 64-bit fingerprint, generated by a variant of the [[Rabin fingerprint|Wikipedia:Rabin fingerprint]] (itself apparently a type of [[CRC]]) called the "[AVRO fingerprinting algorithm|https://avro.apache.org/docs/current/spec.html#schema_fingerprints]".
* The encoded message.
This allows for the schema to be looked up in a database.

==== Object Container File ====
The "object container file", which seems to be more widely-used, is more complicated. It consists of<ref>https://avro.apache.org/docs/current/spec.html#Object+Container+Files</ref>:
* The magic number {{magic|4f 62 6a}}.
* The full JSON schema, and name of the compression algorithm, themselves encoded together as an Avro "map" object.
* 16 random bytes called the "sync marker"
* Several "blocks" of the encoded binary message, with optional compression (the specification calls a compression algorithm a "codec", which is unusual usage outside of media compression).
See [[#Specifications]] for more details.

== Specifications ==
* [https://avro.apache.org/docs/current/spec.html 1.9.0 Specification]

== Links ==
* [https://avro.apache.org/docs/current/ Official website]
* [http://cloudurable.com/blog/avro/index.html Understanding Apache Avro]
* [https://dzone.com/articles/introduction-apache-avro Introduction to Apache Avro]
* [[Wikipedia:Apache Avro|Wikipedia page]]

== Notes ==
<references group="Note" />

== References ==
<references />

[[Category:Apache]]

@@ Line 27: / Line 27: @@
 The "single-object encoding" consists of<ref>https://avro.apache.org/docs/current/spec.html#single_object_encoding</ref>:
 * The magic number {{magic|c3 01}}.
-* A 64-bit fingerprint, generated by a variant of the [[Wikipedia:Rabin fingerprint|Rabin fingerprint]] (itself apparently a type of [[CRC]]) called the "[https://avro.apache.org/docs/current/spec.html#schema_fingerprints AVRO fingerprinting algorithm]".
+* A 64-bit fingerprint of the schema, generated by a variant of the [[Wikipedia:Rabin fingerprint|Rabin fingerprint]] (itself apparently a type of [[CRC]]) called the "[https://avro.apache.org/docs/current/spec.html#schema_fingerprints AVRO fingerprinting algorithm]".
 * The encoded message.
 This allows for the schema to be looked up in a database.

@@ Line 27: / Line 27: @@
 The "single-object encoding" consists of<ref>https://avro.apache.org/docs/current/spec.html#single_object_encoding</ref>:
 * The magic number {{magic|c3 01}}.
-* A 64-bit fingerprint, generated by a variant of the [[Wikipedia:Rabin fingerprint|Rabin fingerprint]] (itself apparently a type of [[CRC]]) called the "[AVRO fingerprinting algorithm|https://avro.apache.org/docs/current/spec.html#schema_fingerprints]".
+* A 64-bit fingerprint, generated by a variant of the [[Wikipedia:Rabin fingerprint|Rabin fingerprint]] (itself apparently a type of [[CRC]]) called the "[https://avro.apache.org/docs/current/spec.html#schema_fingerprints AVRO fingerprinting algorithm]".
 * The encoded message.
 This allows for the schema to be looked up in a database.

@@ Line 26: / Line 26: @@
 ==== Single-Object Encoding ====
 The "single-object encoding" consists of<ref>https://avro.apache.org/docs/current/spec.html#single_object_encoding</ref>:
-* The magic number {{magic|0xC3 0x01}}.
+* The magic number {{magic|c3 01}}.
 * A 64-bit fingerprint, generated by a variant of the [[Rabin fingerprint|Wikipedia:Rabin fingerprint]] (itself apparently a type of [[CRC]]) called the "[AVRO fingerprinting algorithm|https://avro.apache.org/docs/current/spec.html#schema_fingerprints]".
 * The encoded message.
@@ Line 33: / Line 33: @@
 ==== Object Container File ====
 The "object container file", which seems to be more widely-used, is more complicated. It consists of<ref>https://avro.apache.org/docs/current/spec.html#Object+Container+Files</ref>:
-* The magic number {{magic|4f 62 6a}}.
+* The magic number {{magic|4f 62 6a 01}}.
 * The full JSON schema, and name of the compression algorithm, themselves encoded together as an Avro "map" object.
 * 16 random bytes called the "sync marker"

@@ Line 8: / Line 8: @@
 '''Avro''', commonly known prefixed by the name of its creator organization as "'''Apache Avro'''", is a [[serialization]] format (and RPC system) used largely within "big data" systems, specifically the Apache Hadoop environment. It has been designed for extreme "flexibility", which seems to actually mean that it is meant to meet several apparently contradictory requirements at once, and, as a result, it has a somewhat confusing structure.
-At its basis, Avro is an imitation of [[Protocol Buffers]], with a schema written is [[JSON]] that defines a wire format<ref group="Note">In this case, "wire format" is meant in the generic sense, not Avro's specific usage of it to refer to its network protocol.</ref> for serialized data that essentially consists of the field values converted to binary and concatendated. Instead of this binary encoding, the message itself may also be serialized using JSON.<ref>https://avro.apache.org/docs/current/spec.html#Encodings</ref>. However, this becomes complicated by the fact that its schema is usually stored with the message, which gives rise to Avro's distinctive qualities.
+At its basis, Avro is an imitation of [[Protocol Buffers]], with a schema written is [[JSON]] that defines a wire format<ref group="Note">In this case, "wire format" is meant in the generic sense, not Avro's specific usage of it to refer to its network protocol.</ref> for serialized data that essentially consists of the field values converted to binary and concatendated. (Instead of this binary encoding, the message itself may also be serialized using JSON, although this is relatively rare.<ref>https://avro.apache.org/docs/current/spec.html#Encodings</ref>) However, this becomes complicated by the fact that its schema is usually stored with the message, which gives rise to Avro's distinctive qualities.
 One of the features frequently touted as an advantage of Avro is that it can work without Protocol Buffers-like code generation<ref>[[Wikipedia:Apache Avro]]</ref><ref>http://cloudurable.com/blog/avro/index.html</ref> (although in practice this seems to usually be done, anyhow). This only affects ease of programming, not the data format. (Technically, it allows for messages to be parsed without knowing anything about the schema in advance, but this is only useful in the course of writing something like a debug tool.)