Plant the mistake. Then prove your product caught it.
The scenario engine is the difference between 'we tested it' and 'we measured it'. You pick the failure; Fictix plants it and keeps the answer key; your product's detections get a precision and recall score.
Why ground truth is the whole game
Run a detector on production books and you learn what it flagged — never what it missed, because nobody labelled reality. An evaluation without ground truth is a vibe. The scenario engine inverts that: you decide the truth by planting it, so every run yields real precision and recall, not anecdotes.
The eight needle families
A needle is a known mistake hidden in otherwise coherent books. Fictix ships eight families, each with a detectable signature and a recorded location:
| Needle | What it looks like in the data |
|---|---|
| Same bill twice | The same bill recorded twice, sometimes with a slightly different amount |
| Name mix-ups | Vendor or customer recorded under near-duplicate names that should consolidate |
| Ghost worker | Payroll run for an employee with no offer, no start event, no activity |
| Wrong category | Expense booked to a GL account inconsistent with its vendor and history |
| Missing info | Required fields blank on otherwise valid documents (memo, class, date) |
| Wrong date | Transaction posted to the wrong period — revenue/expense recognition trap |
| Books don't match | A record that differs between two systems (amount, date or missing on one side), or bank and journal items left open at month-end |
| Looks like fraud | Round-dollar amounts entered off-hours, or bills clustering just under an approval threshold |
An audit needle can also mark records edited after the books were locked, so audit-trail products have something real to find. Needles compose: one company can carry many, across systems, at a severity you set.
Describe a failure in a sentence
You don't hand-write JSON. Describe the failure in a sentence and Fictix maps it to one of six scenario templates:
- Duplicate expense: a bill entered twice with a small amount change
- Ghost employee: payroll for someone who doesn't exist
- Vendor normalization: one vendor under several near-duplicate names
- Threshold fraud: bills clustering just under an approval limit
- Rev-rec: a prepaid contract that churns early, with deferred revenue to reverse
- Close: books left partly closed, with open journal and bank items
Severity is tunable. Fictix plants the needles and writes a manifest: what was planted, where, and the expected finding. Do it in the console or over the REST API with your project key. A packaged CLI is coming.
Test matrices and regression
Group scenarios into a test matrixthat must pass before release. Pin the saved company by ID and last week's run and this week's run read exactly the same data, so a score delta is a real regression — trackable over time, attributable to a commit, and filable as a ticket that points at the exact transaction you planted.
Turn up the pressure
Chaos and fraud are dials, not booleans, and each scenario has a severity. Raise them to stress a model toward its failure point; lower them to find the floor where it stops catching anything. Pair with the living simulation so fraud-weighted activity keeps arriving as you advance the clock, not just at t=0.
Questions
How does Fictix know if my product is right?
It plants every anomaly itself and records a manifest of what and where, so it computes precision and recall on your product's detections per scenario — misses are explicit, not inferred. It grades what your product flags, not the actions an agent takes.
Do I write code or config to design a scenario?
No. Describe the failure in a sentence in the console (or send it over the REST API); Fictix maps it to one of six scenario templates, plants it in the books, and keeps the answer key.
Can scenario results gate a release?
Yes. Pull a test run's precision and recall over the REST API and fail the CI step when they drop below your thresholds, or require a test matrix to pass before shipping.
Can multiple anomalies coexist in one company?
Yes — needles compose across systems at the severity you set, so you can simulate realistic, messy books rather than one isolated bug.