JSON unmarshalling often has to consider separately whether an attribute is absent, false, zero, null, or the empty string, but this was never quite semantically ambiguous enough for my tastes, so adding that void-ish values may also now be serialised as a tuple of length [0] seems to me an excellent additional obfuscation.
The use case here is to reduce the token usage with LLMs, such as an agent that outputs a list of commands eg. Tuples with files to write and their new contents.
Supporting this use case doesn’t require perfectly marshaling every data structure ever.
But to your point the tool could have wider use cases without the limitations.
If one trains a model to understand it then that model will inevitably emit it, which means in turn one shall have to parse it, and now the application supports TOON for anything, and good luck telling the users/customers any different.
I’ll be interested to see benchmarks. My expectation is that accuracy will take a hit on mid or longer context prompts: I’d bet that the heavy use of JSON in fine tuning will end up impacting quality of a more terse (less reasoning space) novel encoding.
I've only looked at one model (gpt-4.1-nano) so far. I'm hoping to run similar tests on some other models but it gets challenging to discern statistically significant differences with better models as their accuracy tends to be a lot better across the board.
I would assume the next iterations/fine-tuned variants of current models would reach similar accuracy for TOON as they do for JSON.
The current models unfortunately do not have TOON in their training set, so they would probably require additional input tokens to grok the notation, and even then probably won’t have the same accuracy as they do for JSON.
Do you mean the [0] Token Benchmarks section? I only see token count numbers.
Which doesn't address the question: do LLMs understand TOON the same as they would JSON? It's quite likely that this notation is not interpreted the same by most LLM, as they would JSON. So benchmarks on, say, data processing tasks, would be warranted.
differently than me, i guess. I would read that as "at index value of two, i.e. the third element of an array, the values 1aliceadmin and 2bobuser are stored, or not, since we want to destructure these values and a pair value of a tuple of three is given. and would be confused and think wtf is that, dear user, did you omit or misformat values?
I have this: https://github.com/7mind/sick , a binary deduplicating storage for JSON-like data structures with efficient direct access (no parsing required). It's even more efficient in terms of space and access speed (but not manually editable).
If you instead put parentheses around the lexical sequences, then you wouldn't need syntax like `[3]` to denote length.
You also wouldn't need indentation levels to be syntactically meaningful.
You could also get rid of LLM tokens like square brackets, curly braces, colons, and commas.
And you could have objects nested to arbitrary depth.
In near the same character count as TOON (sometimes more, sometimes less).
(I was telling someone over the weekend that there are only a few small wins for Lisps in most AI work right now. I hadn't considered that the printed syntax itself might have a use with these LLM huge black boxes.)
I don’t know what I’m talking about (pure fantasy), but what if you train a model on compressed data and then perform inference on compressed data as well? Could this work? With the output also being compressed and then decompressed by the client?
The tokenizer is already a form of (somewhat lossy) compression of a string of plaintext to a stream of token identifiers. You can reason about Tokenizers/"embedding spaces" as a sort of massive "Dictionary Table/Dictionary Function" like you might use in a zip/gzip stream.
Starting with already compressed data doesn't necessarily mean fewer tokens, you can probably assume similar entropy (or probably worse entropy) in expanding "Dictionary words" in a compressed stream versus "tokens" from a plaintext stream.
Since all input is run through a tokenizer, I would expect the tokenizer space doesn't change a lot between one trained on uncompressed vs one trained on compressed data.
Similar to YAML? Also do consider ancient formats like fixed width - in which case you don’t even need delimiter characters. Are LLMs clever enough to parse these if given a code book or old-school INPUT statement? Cheers
Neat. I did a similar thing with CSV (instead of JSON) a year back.
Great that there are measurements, but I think the really interesting measure would have it run against the actual "Structured Output Format" endpoints of LLM providers, e.g. those fine-tuned to return valid JSON.
I wonder how many tokens will be saved compared to real JSON if we use a special version where property names don't require quotes, like in JavaScript.
Definitely is a core feature of BAML. My main complaint with BAML is that it’s all or nothing. It’s very opinionated and we can’t get the benefits without the DX and vice versa. Separating this feature without requiring a DSL of model definition is a great add.
TOML has some readability and compactness benefits over JSON while still being both common enough for models to easily be able to process it relatively reliably and widely supported in most languages. I suspect BAML still performs better but likewise due to the tooling work I haven't integrated it.
indentation-based sounds pretty brittle for a serialization format. I imagine a tabular format that factors out repeating keys could be expressed fairly compactly in json itself.
The thing is the game with LLMs is not what's shortest, but what's:
1. Mainstream, so they understand it.
2. What they're tuned for, and their tuned for what's mainstream (JSON).
If you want to go extreme compression you can shove it all in JSON strings too and keep the larger structure JSON:
["users",
"1:admin:Alice",
"2:user:Bob",
]
You may say "how is this better". Well it's better because it's still JSON, there's less to explain to the LLM, and to your other devs. Even if we use a weird compact format like "id:role:name" this is still shorter to explain than a completely different syntax with its whole world of rules.
If fairness to toon, the alternative json your giving doesn’t include hints on structure.
Not sure LLM are more “tuned” to JSON.
That said, your general point holds that toon maybe unnecessary. Especially in the examples given. But perhaps plan text would suffice. Toon could be useful when automating inputs with many different shapes.
Yea exactly. The LLMs are tuned to natural language. I don't think anything will beat good ol' templating (a.k.a. plain text). In Go I do something like this:
// mytemplate.tmpl
Description="The following data is for the users in our application."
Format="id,name,role"
length=2
Data:
{{range .}}
{{.ID}}, {{.Name}}, {{.Role}}
{{end}}
This way you're able to change the formatting to something the LLM understands for each struct. The LLM might understand some structs better as JSON, others as YAML, and others in an arbitrary format. Templating gives you the most flexibility to choose which one will work best.
For repeating objects of the same structure, yaml will still require each key on each object, whereas this is a hybrid with csv, so it defines the keys once.
Since the whole point of this is to limit LLM token consumption, it'd be interesting to see the results of prompts that use it.
I've seen a ton of people who just paste a CSV into a prompt and expect it to work well because they don't know any better, but the results are typically hot garbage. It's too repetitive, it can't memorize and/or process such a big chunk of data. Asking an LLM to use pandas to iteratively analyze some CSV works great, though.
I'm sorry I don't see this adding value over various other formats. I don't really want a new object serialization format, I just want the existing ones to have the features I need. YAML but with static typing and schema. XML but without crazy internet features. TOML but with an object format that doesn't hurt my brain. JSON but with decent multiline strings and comments. NestedText but with a sub-standard that provides static-typing and schema and whatnot.
And on the way out of the LLM. Token savings nice on the way out too, and then also I have to imagine it's better for the LLM to see one format in all of it's context instead of two.
It seems like a nice idea to me if restricted to that. Although I guess I am not sure if it's really intended that way - the array count for example is probably pretty bad for LLM output.
I don't think you even need to care about this as a format. It could exist only during communication and encoded/decoded by middleware, and everything still works.
The benchmarks show it performs better than them, so that's the value - cost savings and improved accuracy. I suppose you could convert JSON to TOON just for the LLM and not actually read it with your own brain.
reply