Losing focus of the original UBJSON goal

UBJSON was originally proposed to solve a problem with existing binary JSON formats:

Attempts to make using JSON faster through binary specifications like BSON, BJSON or Smile exist, but have been rejected from mass-adoption for two reasons:

Custom (Binary-Only) Data Types: Inclusion of custom data types that have no ancillary in the original JSON spec, leaving room for incompatibilities to exist as different implementations of the spec handle the binary-only data types differently.

Complexity: Some specifications provide higher performance or smaller representations at the cost of a much more complex specification, making implementations more difficult which can slow or block adoption. One of the key reasons JSON became as popular as it did was because of its ease of use.

BSON, for example, defines types for binary data, regular expressions, JavaScript code blocks and other constructs that have no equivalent data type in JSON. BJSON defines a binary data type as well, again leaving the door wide open to interpretation that can potentially lead to incompatibilities between two implementations of the spec and Smile, while the closest, defines more complex data constructs and generation/parsing rules in the name of absolute space efficiency. These are not short-comings, just trade-offs the different specs made in order to service specific use-cases.

The existing binary JSON specifications all define incompatibilities or complexities that undo the singular tenant that made JSON so successful: simplicity.

JSON’s simplicity made it accessible to anyone, made implementations in every language available and made explaining it to anyone consuming your data immediate.

Any successful binary JSON specification must carry these properties forward for it to be genuinely helpful to the community at large.

As an evolution of UBJSON, BJData originally fixed some basic quality-of-life issues with UBJSON, namely the lack of unsigned numbers and use of big-endian.

I believe the recent addition of extension types unfortunately bring us back to where the other binary specifications were in terms of complexity and risks reducing the further adoption of BJData as a generic JSON binary format.

It could be argued that other types such as char, half, and even the strongly-typed byte arrays I proposed had already deviated from the original JSON spec, but crucially these types all have an obvious/implicit representation in regular JSON, namely string, float and int array. This is not the case for extension types, where the context is completely lost if we turn them into plain numbers or strings.

I have wondered in hindsight whether it would have been better to stick with U based binary type as while it ended up solving a problem in some languages, it introduced further ambiguity in others e.g. Javascript which has a Uint8Array but no corresponding ByteArray. Extension types are likely to suffer the same fate, where some languages will have first-level support for timestamps, dates and UUIDs, while others will have to use proprietary types that make it more complicated to integrate fully.

Right now BJData is the closest thing we have to a one-to-one binary JSON representation, and I really don't want to lose that. Other than simply removing extension types from the spec, one option could be to branch BJData into two formats (which is ever so slightly better than introducing a new format...).

One would be a basic format that would stay true to the original goal - perhaps removing features such as no-op, N-dim arrays and any features deemed to be adding unnecessary complexity.

The other would be a feature-rich superset that could still parse the basic format but also introduce types that are not directly compatible with JSON.

This way those willing to commit to using "BJData++" exclusively could benefit from advanced features (though even there I would urge caution!), while BJData would remain a solid choice if you only care about JSON.

@nebkat

BJData spec has never set the goal of following UBJSON. It does prefer heavily towards JSON-like simplicity, which UBJSON follows.

@nebkat, the recent Draft-4 release of BJData adds two things:

packed objects {$... (or SoA struct-of-arrays or arrays-of-structs), and
extension type E

the first one has been something I have been hoping to add for a very long time - it mirrors the idea of packed array that I initially introduced to BJData that eventually departs BJData from the original UBJSON. it is also something being extensively proposed in the UBJSON community as well as after BJData starts its own path.

for extension E container, I consider it as the only way to stop BJData from inventing new type markers, and the only path towards creating a format that is both forward and backward compatible. As long as someone adds a new data type marker in the future, all existing parsers will break because it does not know how to handle it, or even how to ignore/skip it. E+typeid+payloadlen+payload provides such a way that parsers today can safely ignore unfamiliar data types, even data from future, yet still be able to read the remaining stream/data.

You don't really need to parse anything enclosed inside the E-marker. You can simply use the payloadlen to skip forward, or leave the payload buffer for other libraries to interpret. It is one of the simplest container that I can think of to provide such forward/backward compatibility. I don't know what else can be simpler, in the spirit of JSON, that can achieve this.

Again, extension types are custom types, if parsers do not want to handle customized data types, just ignore those. Perhaps I can add language to discourage the use of E for canonical data storage. was that what your concern? that people will abuse E and store function-critical data?

BJData spec has never set the goal of following UBJSON.

I've always viewed it as the de-facto continuation of UBJSON after its abandonment but that is fair enough! I wouldn't want to suggest that it should take on that goal, but I hope you can consider the unique position that it is in right now.

packed objects {$... (or SoA struct-of-arrays or arrays-of-structs)

I am also very happy to see this implemented. It doesn't introduce any new types so I don't see it contributing to this problem. While there is some implementation complexity it is IMO completely justified by the increased efficiency. A well written encoder could even dynamically generate the schemas based on arbitrary input data so it's easy to benefit from it without making any BJData specific concessions (vs normal JSON).

extension type E

The beauty of UBJSON/BJData over all the other formats was always the one-to-one mapping to JSON.

You could take any valid BJData and convert it to JSON and back without losing anything except maybe encoding efficiency.
You could write APIs that accept as input and return as output both BJData and JSON and know that it will just work.
You could (for the most part) follow existing language conventions on which language data types to use for which JSON data types when writing parsers.
You were therefore never "locked in" to it as a format, making it easy to adopt in the first place.

Extension types:

Are the only feature that doesn't have a direct representation in JSON, losing the one-to-one guarantee.
Are the only type that encodes a purpose/meaning alongside its value, which is normally handled by a schema.
Introduce (sensible, but) opinionated default data types e.g. 12 byte nanosecond epoch, 8 byte microsecond delta.
Introduce ambiguity in terms of compatibility between different languages (particularly with the reserved extension types).

If the goal was just to provide an arbitrary length custom data type this was already possible with a byte array and application level schemas/conventions.

You don't really need to parse anything enclosed inside the E-marker. You can simply use the payloadlen to skip forward, or leave the payload buffer for other libraries to interpret. It is one of the simplest container that I can think of to provide such forward/backward compatibility. I don't know what else can be simpler, in the spirit of JSON, that can achieve this.

JSON is intentionally limited to a basic set of types to ensure universal compatibility. Standards like GeoJSON are possible by building a schema around those basic building blocks, not by introducing proprietary data types. I think this is a better approach to encoding e.g. complex numbers, especially with the addition of SoA making it 100% efficient.

Again, extension types are custom types, if parsers do not want to handle customized data types, just ignore those. Perhaps I can add language to discourage the use of E for canonical data storage. was that what your concern? that people will abuse E and store function-critical data?

It seems more likely to me that the parser would receive an extension type that it can't ignore precisely because it contains vital information. If the option is there, people will naturally use it for important data, and interoperability will depend on every implementation understanding the same extension semantics.

I use the extensions E type in RPC protocol design.

When alignments are required, every byte counts. With a U8 array there is a 6 to 7-byte header representing the element type and the length: [$U#U. or [$U#u.. But there is no standard way to put the inner data type. I might want to declare a 8-bit schema ID preceding the payload, for example.

The E type allows that to be done compactly. You would write Eu<schemaId>u<len> That's 7 bytes with 1 byte of padding to spare.

Like OP mentioned, we can put raw data after the U8 array header [$U#u.. so that an entire bundle of data after that do not need to be parsed. I found that U8 arrays and extension types work well together when combined:

N[$U #u<u16OuterLen>              // begin packet content
Eu<u16typeId> u<u16InnerLen><pad> // first request
<request payload>
Eu<u16typeId> u<u16InnerLen><pad> // second request
<request payload>

In this configuration, all u16 fields do not cross the 8-byte boundary, assuming N[$U... is 8-byte aligned. My parser simply detects a fast-path pattern with 2 aligned u64 reads and some masking.

I am less focused on theoretical purity and more on the engineering side. IMO, Binary JData spec can have both ways of encapsulating raw bytes.

If the goal is absolute maximum data compression then a custom format is likely more appropriate? Trying to bend a dynamic format into alignment is going to be much more work. As I mentioned in #15 (comment):

We have adopted proprietary binary formats where we have finer control over alignment and maximum efficiency then use BJData within that for fields where we need the dynamic JSON capabilities.

For example:

With a U8 array there is a 6 to 7-byte header representing the element type and the length: [$U#U. or [$U#u..

We could save 1 byte by dropping the # (which is redundant) or create a shorthand marker for [$B# to save 3 bytes, but both at the cost of readability.

@nebkat

@nebkat, sorry for the late reply.

I appreciate your detailed reply. I share the same caution of not complicating BJData and recognize the strength of having a simple binary counterpart of JSON-like data.

I've seen the comment that UBJSON is aimed as the "one-to-one mapping" to JSON, but to be honest, I don't have a strong feeling about this point - perhaps you can help me make a clear delineation where this line should be drawn.

in my view, binary JSON/UBJSON/BJData are already going beyond what JSON can support - strong data types, binary data, optimized/packed storage etc. In a way, I thought that providing what JSON misses and offering more natural/efficient data serialization strategy for applications prioritizing speed and size is the primary drive for most of the binary JSON specs. In this sense, the 1-to-1 mapping is already gone.

JSON users also recognize these limitations, supplementing the lack of type-support and data semantics via JSON schema. It adds some type checking capabilities, but still pretty limited compared to binary JSON, which offers strong type schemas. JSON schema also further extending JSON via its data type formats, for example, the string type supports "format": "time" and "format": "date-time", among other formats.

again, because of my vague understanding of "1-to-1 mapping" needs from some of the users, currently, my main rationale of further growing BJData is mostly based on how far the added feature deviates from the simplicity of JSON - a common principal shared with UBJSON.

For the addition of E marker for custom/extended data types, it indeed overlaps with B - except that 1) B does not have a type ID to label the semantics of the data, and 2) B has to be used with [ to produce a byte-stream (without semantics). If B were not added before E, it could have been implemented as E with a lower ID.

I don't know if it helps - in most of the tools I developed, I give a huge weight to portability and backward-compatibility (and when possible, forward-compatibility). One thing that bothers me for the BJData design is the chance of adding new markers could break all existing parsers. There were no "skipping" mechanism. Adding E offers such a mechanism to lower such risks and potentially ensures the long-term stability of the spec.

@fangq

@fangq No worries, thank you for taking my concerns seriously!

I've seen the comment that UBJSON is aimed as the "one-to-one mapping" to JSON, but to be honest, I don't have a strong feeling about this point - perhaps you can help me make a clear delineation where this line should be drawn.

in my view, binary JSON/UBJSON/BJData are already going beyond what JSON can support - strong data types, binary data, optimized/packed storage etc. In a way, I thought that providing what JSON misses and offering more natural/efficient data serialization strategy for applications prioritizing speed and size is the primary drive for most of the binary JSON specs. In this sense, the 1-to-1 mapping is already gone.

This is how I view the existing features:

Value type markers

Marker	Effective meaning
`Z`	`null`
`T` / `F`	`true` / `false`
`i` `U` `I` `u` `l` `m` `L` `M` `h` `d` `D` `B` `H`	a number
`C` / `S`	a string
`N`	(no equivalent, only for encoding purposes)

Container type markers

Marker	Effective meaning
`[` … `]`	an array
`{` … `}`	an object

Optimized-container modifiers (strong types and counts)

Marker	Effective meaning
`$`	an array/object
`#`	an array/object
`$` + `#` on object	an object
`[$][type][#][[]…` (row-major)	an array (of arrays)
`[$][type][#][[][[]…` (column-major)	an array (of arrays)
`[$ {schema} #`	an array (of objects)
`{$ {schema} #`	an object (of arrays)

Note that I do not distinguish between a U, B, D or H - they are all just numbers. C and S are both just strings, every array/object type is just that, even if it is strongly typed or dimensioned. The reason these markers exist is to provide the most efficient encoding possible for the given value, not to assign any particular meaning to it.

The trouble arises when we allow implementation details of the encoding/decoding process to leak out of the library We allow this to happen because in reality we are trying to optimize not just for the efficiency of the encoded data, but also for its representation in memory before/after decoding.

A strongly typed array of B is still just "an array of numbers" but if we decoded it as a generic list[int] it would be very inefficient, so instead we allow it to decode as a bytearray. However this should not be taken as an invitation to treat it any differently than if it had been decoded as a list[int].

The correct approach would be for an application level schema to normalize either the list[int] or bytearray (or np.ndarray[np.uint8]) to whatever representation it wants, remaining fully compatible with JSON. I back this view up with the fact that a non-strongly typed array of numbers decodes to a generic list[int] regardless of the type used, even if the equivalent strongly typed array might have had a better representation.

Of course the reality is many programs commit fully to BJData, realize there is "free" semantic information available and start using it. You branch logic based on whether you received a bytearray or a list[int] (as opposed to a list[int] vs a string, which would have been fine as it is JSON compatible). From that point on it is a slippery slope, as once you accept you are tied to BJData you can justify introducing any data type that happens to be convenient.

The final nail in the coffin is if we officially proclaim we aren't truly JSON compatible any more, so why bother at all. We are then competing with the likes of Protobuf, FlatBuffers, etc - neither here nor there.

JSON users also recognize these limitations, supplementing the lack of type-support and data semantics via JSON schema. It adds some type checking capabilities, but still pretty limited compared to binary JSON, which offers strong type schemas. JSON schema also further extending JSON via its data type formats, for example, the string type supports "format": "time" and "format": "date-time", among other formats.

If we retain a promise of one-to-one mapping with JSON then JSON schema remains fully usable with BJData. We explicitly state that BJData does not take any responsibility in assigning meaning to the data, it is just there to enable efficient storage and encoding/decoding. Ideally we make this point very clear in the documentation to prevent others from getting caught in that slippery slope.

I do nonetheless understand the desire to do something better than BJData + JSON Schema because as you point out they have a string format "date-time" which is of course highly inefficient - but the answer to this is to define your own schema that calls for an integer unix timestamp, not to scrap the schema and move it into the encoding.

For the addition of E marker for custom/extended data types, it indeed overlaps with B - except that 1) B does not have a type ID to label the semantics of the data, and 2) B has to be used with [ to produce a byte-stream (without semantics). If B were not added before E, it could have been implemented as E with a lower ID.

As I mention in the previous comment, the wasted bytes concern is totally valid but it always could have been solved by introducing new markers. We actually have an entire 8th bit free as all of our markers are 7-bit ASCII, so you could form a convention that U & 0x80 is a strongly typed array of U. Other formats take the approach of having dedicated markers for "a byte array of 8-bit length", "a byte array of 16-bit length" etc to save another byte on the length type. But the reason against this was always readability, and E is absolutely terrible in that regard.

I don't know if it helps - in most of the tools I developed, I give a huge weight to portability and backward-compatibility (and when possible, forward-compatibility). One thing that bothers me for the BJData design is the chance of adding new markers could break all existing parsers. There were no "skipping" mechanism. Adding E offers such a mechanism to lower such risks and potentially ensures the long-term stability of the spec.

I think the goal should be to reach a certain stage (which might be right now) and say the protocol is complete, this is the full set of markers available and we will never be changing it in future. It is not the inability to add new markers without breaking decoders that is the problem, it is the fact that new markers are getting added at all.

The E type really just introduces a new language within a language - which I would argue would be much more useful if it was built from the ground up with the goal of being permanently backwards and forwards-compatible. As long as I have some non-E fields I will need a schema, so why not defer all semantic concerns to that?

@fangq

Hey @fangq, just wondering if you've had a chance to consider this?

Track Draft 4: enforce the optimized-type restriction, read column-major

@nebkat

@nebkat, I think what I can commit is to give more time for community review and comments for the next spec update (Draft 5), before tagging and releasing. However, since E marker is already part of the frozen Draft-4, and its need is clearly justified, I don't want to revert it to cause breaking changes.

Committing to no new markers could be added also feels too strong and too early for a still relatively young and less adopted spec. I understand your concern and rationionales. For now, the spec is still mostly used internally in my own projects. I too desire a stable and permanent data format for scientific data exchange, so I think we share the same perspectives.

If I understood your intent correctly, and we agree that we are on the same page, I suggest we close this issue.

@stephenberry

I would be curious to hear the opinion of @stephenberry re beve-org/beve#25, as BEVE has a similar extension system.

To summarise the wall of text above, my argument is that "binary JSON" style formats should avoid introducing new data types/semantics which do not exist in plain JSON except when doing so for the sole purpose of achieving more efficient encoding/decoding (both on the wire, and in the decoded types). Best stated by the original UBJSON spec, JSON's success has been down to its simplicity. Extension types risk that by locking in proprietary conventions that may seem convenient in the short term vs application level schemas, but come at the cost of interoperability across libraries and languages.

CBOR, BSON and MessagePack have all gone deeper down this route and are left with complex and/or fragmented implementations.

@nebkat

@nebkat, I'm not familiar with this codebase or spec, but I do agree that keeping a spec simple is critical for performance and interoperability. Developers will always have niche use cases to optimize, but in a binary format data can always be transported as raw bytes and handled in some ideally optimized way for that specific use case. So, the purpose of a general binary specification is not maximum speed, but rather simplicity with a focus on performance where the payoffs are enormous. CBOR has a huge type registry, and it means that codebases won't support the full extension spec, so there's no guarantee two libraries will actually talk to each other.

Adding lots of extensions just creates a specialized type system and forgets what makes formats like JSON so ubiquitous. But, it is really up to what the developers of this library want to achieve. If they want a tight ecosystem where everyone just uses this library then extensions can make developer's lives easier. But, if they want a ubiquitous specification that can easily be used across dozens of programming languages, then lots of type extensions are probably a bad idea.

BEVE has only three extensions for this reason. A streaming delimiter, matrices, and complex numbers. These exist because they bring orders of magnitude performance improvements by being types and have many uses across scientific applications. And, they are all easily converted to JSON or JSON Lines.

@stephenberry

@stephenberry Thank you for the input!

@fangq As an interim measure would you consider marking extension types in the spec as:

experimental - allowing for further discussion and future changes, and
optional - driven by a encoder/decoder flag, and
application defined only (no standard extension types) - removing any ambiguity as to how common data types such as timestamps should be encoded.

I think that would alleviate most of my concerns while still serving cases such as @flynn162's and your own (and perhaps this is exactly what you were suggesting!).

Navigation Menu

Repository navigation

Losing focus of the original UBJSON goal #33

Value type markers

Container type markers

Optimized-container modifiers (strong types and counts)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Losing focus of the original UBJSON goal #33

Description

Activity

fangq commented on May 10, 2026

nebkat commented on May 11, 2026

flynn162 commented on May 15, 2026

nebkat commented on May 17, 2026

fangq commented on May 20, 2026

nebkat commented on May 21, 2026

Value type markers

Container type markers

Optimized-container modifiers (strong types and counts)

nebkat commented on Aug 25, 2026

fangq commented on Aug 28, 2026

nebkat commented on Sep 13, 2026

stephenberry commented on Sep 14, 2026

nebkat commented on Sep 20, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions