Binary encoding of JSON?

[DennisHeimbigner]

Is there a previous discussion about storing the Json in a binary format
such as protobuf? Unidata experience with this for the OPeNDAP protocol
shows that parsing text is a performance bottleneck and that using e.g.
protobuf is much faster.

[alimanfoo]

There isn't a previous discussion that I can remember.

…

On Tue, 15 Jan 2019 at 16:23, Dennis Heimbigner ***@***.***> wrote: Is there a previous discussion about storing the Json in a binary format such as protobuf? Unidata experience with this for the OPeNDAP protocol shows that parsing text is a performance bottleneck and that using e.g. protobuf is much faster. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#390>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QryXR4SE3NZlgdxgNYrS-KEhN6Zrks5vDgCZgaJpZM4aBUHt> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health Big Data Institute Li Ka Shing Centre for Health Information and Discovery University of Oxford Old Road Campus Headington Oxford OX3 7LF United Kingdom Phone: +44 (0)1865 743596 or +44 (0)7866 541624 Email: alimanfoo@googlemail.com Web: http://a <http://purl.org/net/aliman>limanfoo.github.io/ Twitter: @alimanfoo <https://twitter.com/alimanfoo> Please feel free to resend your email and/or contact me by other means if you need an urgent reply.

[jakirkham]

This blog post about what Uber did with JSON is kind of interesting and potentially relevant.

We could consider adding a JSON filter chain much like the on we have for objects. The default could be to simply use json.loads (possibly wrapped in some kind of Codec class). That way users could change this as needed to handle things like binary encoding and compression. This should be pretty flexible for users trying to explore what is best for them. A bonus of this approach might be to provide users a way to handle things like fill_values that are not JSON serializable. ( #216 ) ( #244 ) ( #354 )

[DennisHeimbigner]

Thanks for the pointer. I was actually proposing something different than
they did. I am proposing we write an e.g. protobuf scheme for JSON
and then use that to encode/decode ascii-json <-> binary-encoded-json
The test that would be interesting would be comparing a compression of
ascii-json to the compression of the binary-encoded-json.

[d-v-b]

closing as not planned. changing the decoded representation of the metadata document is a spec question and the stored representation of the JSON document is left to the store, so protobuf is available for stores that support that as a transparent medium for storing JSON (e.g., icechunk)