v0.2
GitHub

Docs Project

Benchmarks

Turnframe makes no public performance claim. The specification is explicit about the order of events: measure the pieces separately first, and only then decide whether there is anything worth saying. This page describes what is measured, what the numbers deliberately do not cover, and how to run them yourself.

Why these three things, separately #

A turn spends its time in four places: projecting a workflow view, reducing a proposal into commands, persisting a commit, and waiting for a model. Only the last of those is large, and that is exactly why the first three have to be measured on their own. A single end-to-end number would be dominated by the network and would hide a projector that grew quadratic in obligations, or a commit path that walked its journal twice.

So the benchmarks are split by cost centre, and within a cost centre by the thing that makes the cost grow:

Crate Benchmark What it measures
turnframe-core projection WorkflowDefinition::project on a trip-shaped domain, at four view sizes, and check_view over the projected views.
turnframe-core hashing canonical_json, digest_hex and Digest::of_canonical on a payload shaped like the things the workspace actually hashes.
turnframe-store persistence CommitStore::commit on the in-memory store at three batch sizes, and one page of the event stream at three page sizes.
turnframe-provider structured_parsing parse_structured end to end, then schema validation and serde deserialization separately, then schema compilation cold against a cache hit.
turnframe-provider stream_reconstruction StreamAccumulator and reconstruct over a realistically chunked answer: three-character text deltas and tool-call arguments arriving in eight-byte fragments.

Projection and invariant checking are separate rows on purpose. Projection is a pure function whose cost tracks the number of obligations it lists; check_view erases the view first, which canonicalizes and hashes every obligation, so it grows far faster. Measured together, the second would hide inside the first.

The same reasoning splits parsing into three rows. When a structured turn gets slower, the useful question is whether it was the jsonschema pass or serde, and a single number cannot answer it.

What is deliberately not claimed #

Nothing here is a latency target, an SLO, or a comparison with any other library. These are microbenchmarks of deterministic code paths, run on one developer machine.

The store numbers are a floor, not a database. They measure the in-memory implementation: one mutex, no serialization to a wire, no network, no fsync. Their value is that when the PostgreSQL store is measured, the difference between the two is attributable to the database rather than to the bookkeeping the store contract imposes. Do not read the in-memory commit number as "a commit costs this much".

Provider latency is not here. A model call is four to six orders of magnitude larger than anything on this page, and it belongs to the provider, not to Turnframe. What is measured is the deterministic work the library does around the call: parsing, validating and reassembling.

Reduction is not yet measured. The reducer in turnframe-runtime runs once per turn and does no I/O; its benchmark has not been written, and this page will grow a row when it is.

These benchmarks do not run in continuous integration. Continuous integration compiles them (cargo bench --workspace --no-run), so that they cannot rot, but it does not time them: a shared runner produces numbers too noisy to gate on, and a gate nobody trusts is worse than no gate.

Running them #

shell
# Everything, properly (minutes).cargo bench -p turnframe-core -p turnframe-store -p turnframe-provider # One file.cargo bench -p turnframe-core --bench projection # One row.cargo bench -p turnframe-provider --bench structured_parsing -- 'parse_structured/4' # A quick smoke pass with short sampling, for a sanity check rather than a number.cargo bench -p turnframe-core --bench hashing -- --warm-up-time 0.5 --measurement-time 1.5 --sample-size 20 # What continuous integration does: compile only.cargo bench --workspace --no-run

Criterion writes its reports to target/criterion/. Running a benchmark twice compares the second run against the first, which is the intended way to use this: a change measured against your own baseline on your own machine is meaningful, and an absolute number quoted from someone else's machine is not.

Each benchmark builds its inputs outside the measured closure. Where an operation consumes its input (committing a bundle, pushing a stream event), the input is produced with criterion's iter_batched, so the construction is setup and only the operation is timed.

The core and store benchmarks define their own small workflow rather than importing the sample domains from turnframe-test. turnframe-test depends on turnframe-core, turnframe-store and turnframe-provider, so a dev-dependency pointing back at it is a cycle Cargo tolerates but the publication order in the release checklist does not, and it would drag three crates into every cargo bench -p turnframe-core. The benchmark domain is modelled on the sample trip all the same: several phases, simultaneous obligations, a parameterized obligation per extra with no payer, and a blocking rebooking card.

Indicative numbers #

Measured on the machine this was developed on: a developer laptop, x86_64 Linux, release profile, short sampling (20 samples, 1.5 s measurement window). These are indicative only. They are not published claims, they were not measured on dedicated hardware, and they should not be quoted anywhere as a property of Turnframe. Reproduce them on your own machine before drawing any conclusion.

Row Median
projection/project, empty case ≈ 36 ns
projection/project, 3 obligations ≈ 48 ns
projection/project, 33 obligations ≈ 773 ns
projection/project, blocking confirmation ≈ 147 ns
projection/check_view, 3 obligations ≈ 341 ns
projection/check_view, 33 obligations ≈ 13.9 µs
hash/canonical_json, small / 8 extras / 64 extras ≈ 0.75 µs / 4.3 µs / 31.6 µs
hash/digest_hex, same payloads ≈ 0.21 µs / 0.90 µs / 2.8 µs
hash/of_canonical, same payloads ≈ 1.1 µs / 5.1 µs / 32.5 µs
store/commit_bundle, 1 / 8 / 64 events ≈ 0.51 µs / 2.0 µs / 14.1 µs
store/event_stream_page, 32 / 128 / 512 events ≈ 8.2 µs / 35.9 µs / 149.7 µs
provider/parse_structured, 1 / 4 / 16 acts ≈ 2.0 µs / 8.8 µs / 37.9 µs
provider/schema_validate, 1 / 4 / 16 acts ≈ 0.20 µs / 0.56 µs / 2.1 µs
provider/deserialize, 1 / 4 / 16 acts ≈ 0.65 µs / 2.6 µs / 13.7 µs
provider/schema, cold compile / cache hit ≈ 27.7 µs / 5.9 µs
provider/stream_accumulator, short / typical / long ≈ 1.9 µs / 7.1 µs / 22.7 µs
provider/reconstruct, short / typical / long ≈ 2.6 µs / 7.9 µs / 25.7 µs

Three observations the table supports, and one it does not.

Canonicalization dominates hashing: canonical_json is roughly ten times digest_hex on the same payload, so the cost of an identity is the cost of sorting a JSON document, not the cost of BLAKE3.

The schema cache earns its lock. A cold compile is about five times a cache hit, and a hit is not free either: it still canonicalizes and digests the schema to build its key.

Projection is comfortably below the noise floor of a network call at every size measured here, which is what §28 asks of it.

What the table does not support is any statement about a real deployment. Every row is a microbenchmark of one function on one machine, with the database, the network and the model removed.

The corpus against a real model #

The live corpus (crates/turnframe-eval/tests/live_corpus/, see evaluation) is the one measurement here about models rather than code. It holds 76 items on the travel desk, 46 in English and 30 in Italian: 8 showcase items, 8 complex messages, 8 conversations of several turns, and the rest one kind of turn each. The showcase items are the cases the guarantees exist for: a card the airline re-quoted after it was shown, an airline that has not answered, a leg the user asked to keep while the other is rebooked, and acts in one message that depend on each other.

Measured on 1 October 2026 against OpenAI's gpt-6-luna at the default medium effort, three samples per item, three items played at once. Cost is priced at $0.10 per million input tokens and $0.50 per million output tokens, reasoning included. Latency is a whole turn, reply included, over every turn a run played, the earlier turns of a conversation among them.

Run Samples passed Items at 3/3 Model calls Input tokens Output tokens (reasoning) Turn p50 / p95 Cost
First 227/228 (99.6%) 75/76 3,409 2,805,255 181,797 (42,376) 9.6 s / 16.5 s $0.37

gpt-6-luna is the model these figures are measured on from 0.2.1. A run costs about an eighth of one on gpt-5.4-mini, whose runs of the 0.2.0 code are in the next section.

The run first passed 222 of 228, on the 0.2.1 code before the rules it showed were written. Each rule is a sentence measured by replaying the call it changes on this model, twenty or thirty times before and after. «il numero fedeltà non te lo do» was read as a condition to leave the number as it is: refusing to give a value is now a request. A payer given for an extra the trip already held was read as a new extra: the sample's operation now says it adds a new one. The check called «the end of the month» an incomplete date: words that place a day now state the day they place, and a day they do not place is still found different. With those rules in, «I would rather fly today» was read as small talk in one sample: saying what one would rather a record hold is now a request. In the same run, 12 of 284 replies spent their whole cap reasoning and came back empty, so the turn fell back to the reply code writes; replies are now written at the least effort, which on this model is none.

The items those rules act on were run again, three samples each, whatever their result: the three that refuse a value, the seventeen whose messages add an extra or say who pays, and the four that give a day without naming it; then the four that state a preference, once that rule was in. The last measurement of each item stands. Four items read the code as released, twenty read it before the preference rule, and 52 keep their first-run samples: none of their messages refuses a value, gives an extra or a payer, places a day or states a preference, though every reading now goes through the new sentences. The row counts the calls and tokens of exactly the samples kept, each call placed on its item by the messages its request quotes; 34 calls no request placed are counted too. Its turn times are the first run's over all its 282 turns, since items played at once share turn numbers. The report is crates/turnframe-eval/baselines/2026-10-01-medium-x3-gpt-6-luna.json.

The one sample missed, in conversation.whole_trip.it, read «le paga l'azienda», the company pays for them, as the airline, which Italian calls «compagnia aerea». The check found the value different and the second reading repeated it, so the turn asked again and recorded no payer. Describing the payer's values in the sample's own words did not change the rate in replay.

What does not vary. In no sample on either model did a rebooking go to the airline without a click on a card showing its current fare, a click on a card drawn before the fare changed run, or an outcome the airline had not given get recorded. Code enforces these, not a model: seven items forbid a rebooking nobody clicked for, and the silent airline's forbids a recorded outcome. Every failure above is a reading: a value in the wrong place, a part left unread, an act the user did not ask for.

What is not measured in this release. Other models, the other vendors' adapters, and the low and high levels were not run on this corpus, and the row above merges three runs: no single run read the code as released. Each vendor's adapter has a live test of its schema dialect against the real endpoint, which is a different claim; the OpenAI one passed on gpt-6-luna, gpt-6-sol, gpt-6.1-sol and gpt-6-astra. Seventy-six items at three samples each can rank two versions of the code on this corpus and this model, and nothing more.

The corpus on gpt-5.4-mini, through 0.2.0 #

Measured on 28 to 30 September 2026 against OpenAI's gpt-5.4-mini at the default medium effort, three samples per item, one turn at a time. Cost is priced at $0.75 per million input tokens and $4.50 per million output tokens, reasoning included. Latency is a whole turn, reply included, over every turn a run played, the earlier turns of a conversation among them.

Run Samples passed Items at 3/3 Model calls Input tokens Output tokens (reasoning) Turn p50 / p95 Cost
First 221/228 (96.9%) 70/76 3,351 2,577,929 192,289 (32,546) 6.5 s / 11.9 s $2.80
Second 222/228 (97.4%) 71/76 3,369 2,606,707 186,979 (26,145) 6.6 s / 12.2 s $2.80
Third 223/228 (97.8%) 73/76 3,400 2,665,077 190,725 (28,606) 7.2 s / 12.1 s $2.86
Fourth 217/228 (95.2%) 68/76 3,439 2,726,282 195,363 (32,402) 7.9 s / 15.2 s $2.92
Fifth 220/228 (96.5%) 70/76 3,359 2,644,648 186,465 (25,798) 7.1 s / 11.8 s $2.82
Sixth 224/228 (98.2%) 72/76 3,411 2,681,584 190,613 (30,612) 7.2 s / 13.0 s $2.87
Seventh 225/228 (98.7%) 74/76 3,393 2,663,742 188,795 (29,050) 8.6 s / 16.2 s $2.85
Eighth 225/228 (98.7%) 73/76 3,369 2,647,418 185,826 (26,650) 9.0 s / 15.4 s $2.82
Ninth 228/228 (100.0%) 76/76 3,383 2,669,662 185,202 (26,632) 7.7 s / 13.6 s $2.84
Tenth 227/228 (99.6%) 75/76 3,434 2,831,091 205,090 (42,505) 8.6 s / 15.9 s $3.05

Each run read a later state of the 0.2.0 code. Every misreading a run showed became a rule stated in general terms, or a sentence of the sample domain's own configuration, with a scripted test where code changed, before the next run. The fourth run measured a change to how every message is split, made for one item, that split «Lisbon for March» in two; it was taken back, and the item got a narrower fix.

The ninth run first passed 223 of 228. One sample of multi.two_fields.it ran the date before the name, because a rule new in that run ordered the acts of one part by where their values stand and trusted a date pointed at the whole part. The others were readings, and each became a rule: a traveler named by a pronoun taken for a new name, a named traveler read as none, a name the segmentation cut in two, and in complex.discursive.it a chatty opening read as a condition that held every act and a date whose words a re-read dropped. The items the rules act on were measured again on the code that has them, three samples each, whatever their result: the twelve whose parts asked for more than one act in the seventh, eighth or ninth run, the eighteen that name a record or ask for one operation in two parts, and the nineteen whose segmentation votes differed by one word or read small talk again. The last measurement of each item stands, so every item reads code that holds each rule acting on it: nineteen read the code as released, nine read it before later rules that do not act on them, and 48 keep their ninth-run samples: in every run traced, none of the new rules acted on them, save the sentence every extraction now reads. The row above counts the calls, tokens and turns of exactly the samples kept, and the report is crates/turnframe-eval/baselines/2026-09-29-medium-x3-gpt-5.4-mini.json.

Every item passed 3 of 3 in the ninth run as it stands. That is a merge, not one run of the released code, and every whole run before it missed at least one sample: expect a whole run to miss a sample or two, each a reading the reply then asks about or leaves undone.

The tenth run is one whole run of the 0.2.0 code, with three items played at once, which is why its turns run longer, and its report is crates/turnframe-eval/baselines/2026-09-30-medium-x3-gpt-5.4-mini.json. Whole runs of earlier states of 0.2.0 passed 225 and 217 of 228, and each miss became a rule with a test: an answer read as the previous turn's operation, a check that found «the A is X» read as X to hold too much, a «can I O?» the check found not asked for, among others. The one sample the tenth run missed, in value.add_extra.en, is a single verdict of the check that an extra's description was incomplete: read again, the same verdict is «stated» twelve times in twelve. Three changes came after it, and the items whose turns they act on were run again on the code as released, three samples each: the thirteen whose check is shown the words of a part a value leaves out passed 38 of 39, and the eight where the check found a value the user did not give passed 24 of 24. The one miss asked who pays for an extra the user gave no payer for, which the last of the three changes stopped; no turn of the tenth run met the change that lets a part found not asked for hold nothing.

What does not vary. In no sample of the ten runs did a rebooking go to the airline without a click on a card showing its current fare, a click on a card drawn before the fare changed run, or an outcome the airline had not given get recorded. Code enforces these, not a model: seven items forbid a rebooking nobody clicked for, and the silent airline's forbids a recorded outcome. Every failure above is a reading: a value in the wrong place, a part left unread, an act the user did not ask for. The worst was a trip named from the words of a dispute («that is wrong») in the first two runs, fixed since.

What was not measured for 0.2.0. Other models, the other vendors' adapters, and the low and high levels were not run on this corpus. Each vendor's adapter has a live test of its schema dialect against the real endpoint, which is a different claim. Seventy-six items at three samples each can rank two versions of the code on this corpus and this model, and nothing more.

Conversations with simulated users #

The corpus above measures single messages. Simulated users measure conversations: a model plays a person with a goal and a manner, and code scores what the stores and the turns show. Measured on 1 October 2026 against OpenAI's gpt-6-luna, which also played the users, at the default medium effort: the five travel desk goals of tests/simulated_users/, ten manners, one conversation each, held on 30 September 2026. The run read the 0.2.1 code before its last rule, the one that reads a stated preference as a request.

Run Reached Turns Dead ends Loops Not understood Refused Offers refused Violations
First 10/10 31 0 0 0 0 0 0

On gpt-5.4-mini, the 0.2.0 code was measured on 30 September 2026, with the same model playing the users. The first two runs held them on the harness's fixed day in 2023, so a user who gave no year was read in 2023 or 2024 and a goal written for 2026 could be missed; from the third on, the conversations are held on 30 September 2026, a day of the year the goals are written for.

Run Reached Turns Dead ends Loops Not understood Refused Offers refused Violations
First, before the fixes it found 8/10 58 0 2 6 6 0 0
Second 9/10 40 0 1 9 3 0 0
Third, on the goals' day 9/10 37 0 0 6 0 0 0
Fourth 9/10 27 0 0 1 0 0 0
Fifth 10/10 37 0 0 0 0 0 0
Sixth 10/10 31 0 1 2 0 0 0
Release 10/10 28 0 1 3 0 0 0

Each run read a later state of the 0.2.0 code, and what a run showed became a rule stated in general terms, or a sentence of the sample domain's configuration, with a test, before the next: the third run's four «confirm the rebooking» turns after the rebooking had left, read as not understood; the fourth run's name taken with the words asking for it; the fifth run's question about the quoted flight answered as what can be done, without the quote; the sixth run's rebooking held by a part the check found not asked for. The release run's three parts not understood are in one conversation: an email and a loyalty number given in the message that registered the traveler, then the traveler's name given as the trip's. Ten conversations are one sample each: a rate moves by a conversation at a time, and two runs of the same code differ. What does not vary is the last column: a dead end or an offer the domain refused is a broken guarantee, and none occurred in any run.

    Type to search the guides, the decision records and the changelog.