This is the story of a deliberately aggressive testing campaign against Ring++ — the kind where the goal is not to confirm that the software works, but to break it on purpose and see what falls out.
Two real bugs fell out. One of them made the program vanish mid-run: no error message, no line number, nothing printed at all. It had been there since the library was written, and it took four lines of ordinary-looking code to trigger.
First, the words
Three terms do most of the work in this piece, and none of them needs to be mysterious.
A gate
An automatic check that runs on every change. If it fails, nothing ships. Think of a spell-checker that refuses to let you send the letter. Ring++ had 26 of them.
A buffer
A block of memory holding raw bytes — a document being assembled, a file being read. Ring++ exists to let Ring programs handle these without copying them constantly.
The campaign
Not one more gate of the same kind. A different question, asked several million times, with inputs chosen to be as awkward as possible.
The goal, in one sentence
Find the bugs the existing checks were structurally incapable of finding — not the ones they had merely missed.
That distinction is the whole point. A check that has run clean for weeks might be watching carefully. Or it might be looking in a direction where nothing bad ever happens.
What we found, at a glance
Four parts of the library, attacked separately. Two came back clean — which is a result, and is reported here as one rather than quietly skipped.
| what was attacked | how hard | what came out |
|---|---|---|
| Byte storage the core buffer |
20,002 operations, 7 kinds of hostile data |
Crash. Killed the process outright, silently |
| Windows into a buffer looking at part of the bytes |
357 windows, nested three deep |
Clean. Nothing found |
| The safe sandbox running untrusted code |
8 checks on parts never tested before |
A method that never worked — and the manual promised it did |
| Fast list reading the speed-up for big lists |
4,000 reads, compared one by one |
Clean. Nothing found |
And one more, outside Ring++ entirely: the same flaw was found sitting in a separate 300,000-line library that had arrived at the same design independently.
Why the existing checks could not have caught it
Go back through what those 26 gates actually asked. One throws a hundred thousand random reads and writes at a buffer and demands that none of them crash. Another makes thirty assertions about how the library behaves. A third proves that a Ring++ object never accidentally overwrites one of your own variables.
Every single one asks either "does this refuse illegal input?" or "is the answer the right size?"
Not one had ever asked: are the bytes actually correct?
That gap is easy to miss precisely because the suite looks thorough. A hundred thousand random accesses reads like coverage. It is coverage — of the one question it happens to ask.
What: two implementations, one truth
The technique is differential testing, and it is old. Keep two implementations of the same idea. Run identical operations through both. Compare the results after every single step. Where they disagree, one of them is wrong — and you find out at the operation that caused it, not three thousand operations later when the damage surfaces.
Ring++ is unusually well suited to it, because the second implementation writes itself. Every Ring++ operation exists precisely because there is a slower, obvious way to do the same thing in plain Ring. That slow way is the yardstick — slow, but beyond suspicion.
func ModelPoke cModel, nOff, cData # rebuild the whole string, the slow way cLeft = left(cModel, nOff) nAfter = len(cModel) - nOff - len(cData) cRight = right(cModel, nAfter) return cLeft + cData + cRight
oBuf.Poke(nOff, cData) # after EVERY operation: if oBuf.Str() != cModel # one of the two is wrong, and # we know exactly which op did it ok
How: make the inputs hostile, and the failure reproducible
A differential test is only as good as what you feed it.
Random printable text would have found nothing here. The payloads were
chosen to be nasty on purpose — seven kinds, drawn at random: ordinary
text, a leading zero byte, the literal "NULL", runs of
0xFF, packed doubles, all zeros, digits.
Twelve buffer sizes, chosen to sit on either side of every threshold in the library: 1, 2, 7, 8, 15, 16, 63, 64, 511, 512, 513, 4096. Offsets and lengths random within legality. The whole buffer compared after every operation — not sampled, not at the end. 20,002 operations in 243 milliseconds.
The randomness is deliberately fake, too — a fixed formula that produces the same "random" sequence every run. A failure you cannot reproduce on demand is a rumour, not a bug report.
What it found, on the first run
The process died after roughly 13,000 operations. No error message, no line number — the shape of failure this project already knew and had written down: an over-write kills the process silently and uncatchably.
Bisecting the failing run took longer than writing the test. What came out was four lines:
oB = RppBuffer(16) oB.Poke(0, "NULL") # safe. The guard catches this. oB.Poke(4, RPP_NUL_BYTE) # safe. The guard catches this too. oB.Grow(33) # process dies. No message.
Neither write is dangerous. Grow is — because
it hands the whole buffer back to Poke as a single source, and
those sixteen bytes now begin NULL\0.
The root cause is a category error, and an old one. Ring's
memcpy decides whether an argument is a null pointer by
calling strcmp(src, "NULL"). Ring++ knew that and guarded
against it — but the guard tested the Ring value for equality
with "NULL", and strcmp reads the C
view. A string whose bytes begin N U L L \0 is
"NULL" to strcmp, whatever its Ring length says.
A sixteen-byte string can never equal "NULL" in Ring. It can
be "NULL" in C.
The library had carried that hole its entire life.
What the fix cost, measured rather than assumed
The repair is to test the bytes instead of the value. That puts one more string index on the hottest path in the library, so it owes a number.
Poke, the fix's true costTwo things about that number are worth more than the number itself.
The first measurement was wrong. A single run put the baseline at 461 ms and made the fix look like a 14% regression. Alternating three runs of each build gave 423 against 408 — the 461 was a cold process. Had it been published, the library would carry a false claim about its own slowest path.
The first fix was four times too expensive. Written
as a flag assigned across nested ifs, it cost 0.32 µs. Testing
whether Ring's and/or short-circuit — they do, and
that also protects an out-of-range index — allowed one flat expression at
0.075 µs. That behaviour was not documented anywhere this project could
find, and finding it paid for itself immediately.
Then the same treatment for everything else
Three of the four public types had never been tested this way. Two came back clean, and that is a result worth printing rather than passing over in silence.
RppView — clean
357 spans against plain-Ring slices, Sub() offsets
composing through a three-deep chain, and a view held across a
Grow that reallocates underneath it. Sub() had
been tested nowhere at all.
RppIndexed — clean
4,000 indexed reads identical to plain ones. The documented limit
asserted as a limit: sort() must go undetected, so
if the guard ever starts catching it, the documentation has begun
overstating.
RppSandbox — a second bug
SetVar had never been called by any gate. It could not
create a variable — only assign to one that existed — and failed with a
raw VM error. The documentation promised what the code could not do.
And one next door
The same defective guard exists, character for character, in Softanza's own buffer type — a separate 300,000-line library that had independently arrived at the same design, including the same 512-byte threshold and the same rule about never caching an address. It copied the reasoning correctly and inherited the flaw with it.
It has not fired there yet, because its trigger is narrower. That is not the same as being safe.
What this is actually about
Not that Ring++ had a bug. Every library has bugs; the useful question is what kind of test finds them.
A green suite tells you the answers to the questions you thought to ask. It says nothing whatever about the questions you did not. Twenty-six gates, one of them a hundred-thousand-iteration fuzz, and the thing they all had in common was that none of them ever compared a byte.
This test is now the twenty-seventh check, and it finishes in a quarter of a second. The crash sequence is written in by name and runs before any random one — because the random search found it by luck, and luck does not run again on demand. And putting the bug back makes the check fail at exactly that line, which is the only real evidence that a test is worth keeping.
Where this does not help. A differential test needs a reference implementation that is obviously correct. Ring++ has one for everything it does, because the whole library exists to replace a slower way of doing the same thing. A project without that ready-made yardstick gets no such gift, and inventing one is usually harder than writing the code being tested.
Everything above is in the repository: the gate is
tests/differential.ring,
the findings are F-31 through F-33 in
FINDINGS.md,
and the numbers came from the programs beside them.