Unveil

Code

Ways of writing that hide nothing by design — bases, bits, binary formats, floats, Unicode, Morse, QR codes, Roman numerals, timestamps, coordinates, check digits — read and written both ways.

Binary Formats

Reads CBOR (RFC 8949), MessagePack and Protocol Buffers without a schema: the bytes as nested items, written in each format’s notation and drawn as bars over the bytes they take, an item’s head apart from its content. A Protocol Buffers field has no name and no type in its bytes, so a nested message, a text or bytes is a reading, never a fact.

History, how it works, how it was checked, limits

History

JSON writes data as text; these three write it as bytes. MessagePack, by Sadayuki Furuhashi, gives each value a first byte that says what follows. CBOR, first RFC 7049 and now RFC 8949 (Bormann and Hoffman, 2020), packs a major type and a length into its first byte and adds tags and lengths left open. Protocol Buffers, Google’s format, sends only a field’s number and a wire type: what the field is called and what it holds is written in a .proto file that the bytes do not carry.

How it works

The bytes are pasted in hexadecimal or Base64 and read by the format chosen. Each item becomes a node over the bytes it takes: its head (the first byte and any length after it) and its content. The whole is written in the format’s notation: CBOR’s diagnostic notation of RFC 8949 §8, Protoscope’s for Protocol Buffers as the encoding guide writes its examples, and for MessagePack, which has none, JSON with a few extras that are Unveil’s. The figure draws each item as a bar on the bytes’ offsets, its head darker, its parts on the row under it, and under it the same items as an outline; one chosen names its bytes. A Protocol Buffers payload of bytes is given the first reading that holds (a nested message, else text, else bytes) and the others are named beside it.

How it was checked

The tests read every one of the 81 examples of RFC 8949’s Appendix A and hold each to the notation the RFC prints beside it, with the lengths left open and the three float sizes. The encoding guide’s own messages (150 as 08 96 01, “testing”, the message inside a message, the packed 1 2 3) come out as it writes them. MessagePack’s formats are tested on bytes built from the specification’s table, its three timestamps among them; no outside program was run for it. Malformed input is refused where it fails.

The tests hold 27 checks of this tool; 2 rest on outside material:

  • A standard’s own vectors RFC 8949, Appendix A: all 81 examples read into the diagnostic notation the RFC prints (integers to 2^64, bignums, the three float sizes, tags, strings and containers of indefinite length)
  • A standard’s own vectors RFC 8949’s {"a": 1, "b": [2, 3]}: seven items, each over its own bytes, a map’s head one byte and its content the rest, keys and values told apart

Every tool’s checks

Limits

CBOR tags are named only where RFC 8949 itself defines them; IANA’s registry was not read. A Protocol Buffers message cannot be told for certain from its bytes: a short text can pass for a nested message, a varint may be signed, a bool or an enum, and no field has a name here. MessagePack’s application extension types are shown as their bytes. BSON, Avro and Thrift are not read. At most 20,000 items are read.