The Tail

Every request gets a trace identity, the alert names its trips -- and spans ship as one batch

Every code block below is real, and every output block is its actual output (the run lives in base/test/perf/_tail_narration_demo.ring; the guard suite is base/test/perf/tracing_tail_narrated.ring, 26 assertions). Performance system P7, the deferred tail -- doc/design/SOFTANZA_PERF_SYSTEM.md.

What was owed

A trace identity on every request

Enable tracing on the monitor before Observe() (the server takes its copy at that moment), and every request gets a W3C trace identity, recorded in an engine trace haro -- the same copy-proof design as every other perf store, which is the point: the server's face records, and every other face reads the same truth:

oMon.EnableTracing(64)          # BEFORE Observe
oSrv.Observe(oMon)
# ... three requests ...
aT = oMon.RecentTraces(3)       # read through OUR face

traceparent: 00-b2e6b38e093cf3698ec96a1794e02caa-46154b26907506e1-01
258483c8ac481058348af70b5032b01c  /order  200  6.09 ms
d58848679f41bda4b23275f3b20de089  /order  200  0.57 ms
b2e6b38e093cf3698ec96a1794e02caa  /order  200  0.97 ms

Each request opened its own trace (three distinct ids), each row knows its path, outcome, and cost, and the response echoes the traceparent header so the caller can correlate. The haro is bounded: past capacity, the oldest rows give way -- constant memory forever, like everything else in the system.

Joining the caller's trace -- watch the ids

When a request arrives already carrying a valid traceparent, the server does not open a new trace -- it joins the caller's, as a child span. Read the two headers closely:

caller sends : 00-3baa1cd12fe2c9dc1cc9639a48d8ba0a-149cb59fc39c83f3-01
server echoes: 00-3baa1cd12fe2c9dc1cc9639a48d8ba0a-5555f2bf48548689-01

Same traceId, new spanId -- textbook W3C parent/child, generated by the engine's tracectx. A phone app, this server, and a downstream service its handler calls (hand TraceParent() onward) all land in one tree in any tracing backend.

The alert names its trips

The P4 debt, paid: the sentinel's black box -- already photographing senses and metrics at breach time -- now includes the trace ids of the requests nearest the breach:

gL.Set(900)
oSent.Check()
aBox = oSent.LastBlackBox()

breach: value-queue.depth-under-100
the trips nearest the breach:
  258483c8ac481058348af70b5032b01c  /order -> 200
  d58848679f41bda4b23275f3b20de089  /order -> 200
  b2e6b38e093cf3698ec96a1794e02caa  /order -> 200
  3baa1cd12fe2c9dc1cc9639a48d8ba0a  /order -> 200

This is the bridge from "the p95 is bad" to "these requests are why": paste any of those ids into Jaeger or Tempo (the same ids travelled the wire in the response headers) and the 3 a.m. investigation starts from evidence, not archaeology.

Spans ship as one batch

P0's stopwatch exports one span; a real trace is many. stzOtelBatch renders the full OTLP resourceSpans envelope -- the exact JSON a collector ingests at /v1/traces:

w2.JoinTrace(w1.TraceParent())
oB = StzOtelBatch("restolean")
oB.AddSpan(w1).AddSpan(w2)
? left(oB.ToJson(), 200) + "..."

{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"restolean"}}]},"scopeSpans":[{"scope":{"name":"softanza.perf"},"spans":[{"name":"import-orders","traceId":"17...
spans: 2 ; shared traceId appears 2x

The joined spans share one traceId inside the shipment, so the collector reassembles the tree. With this, the interop table closes: spans (single and batched), metrics (Prometheus + OTLP), SLA verdicts (the CI gate) -- perf data speaks the industry's formats in and out.

Two items ruled, not built

An honest tail states what it deliberately leaves. The counting-allocator engine-heap stats: every engine module allocates through its own c_allocator; counting them means rerouting every module's allocator -- that is a project, not a tail item, and the operational question it would answer ("is the process growing?") is already answered by RSS, peak, and the series trend. Ruled: deferred until a concrete need names it. The driven-load R-vs-X harness: the interesting knee in the R-vs-X curve comes from queueing -- concurrent arrivals -- and this in-process serial harness cannot produce them; a multi-process load driver belongs with the cluster's infra-gated work. Snapshot()/Curve() remain for load levels you drive yourself.

The performance system is complete -- P0 through P7, eight guards, 283 assertions, every phase narrated with real output. The app that runs is the app that knows how it runs, can prove who asked, and ships what it knows in any tongue the industry speaks.

Proved by

The article names these scenario guards of the library:

• tracing_tail_narrated.ring(opens in a new tab)

Cite

Mansour Ayouni. "The Tail". Softanza narrations, 2026. https://mayouni.github.io/stzsite/en/narrations/stz-perf-tracing-tail-narration.html

Run in Haro's name on 2026-10-08 20:42 inside the library at commit 4a184e6f0, block after block in one process: 0 of 3 blocks kept their promise; the others are illustrations. The code of the articles is Haro code. Where an article was written before the language took its present name, the name was changed and nothing else. Genre, area and series read from the file's name, the year from its first commit: derived, until the article carries its own header. The file on GitHub(opens in a new tab)