In this book, the author is using avro to serialize data before it is being processed by pig. What I don’t understand is, why we need to do it using a serialization framework like avro? Why don’t we just use json as the serialization data format? This way we can just use serializer/deserializer library in a language e.g. data.json in clojure.
1
Rendering data into JSON can be considered a form of serialization, albeit into a very unconstrained format. A lot of the value in a serialization protocol and library is the additional structure it imposes on the wire representation.
JSON used as-is is not able to represent via any built-in way the ‘type’ of objects or inter-object references. In order to represent them, you need to add additional ad-hoc structure yourself like fields encoding type information, object IDs so they can be linked, and so on.
A purpose of a serialization library is to encode these things in a consistent manner onto the chosen transport format (custom binary, JSON, XML, text), such that the important aspects of the data are not lost.
When you stuff a bunch of data into a JSON file, you are adhering to an informal contract by the encoding and interpretation of the producer and consumer. Concepts such as JSON Schema may help structure such an approach, but it’s often easier to use an established serialization format.