Evaluate financial AI against planted ground truth.
An anomaly detector is only as trustworthy as the anomalies you tested it on — and whether you knew the right answers.
The problem with testing on real data
Production books have unknown ground truth. If your model flags twelve things, you can't say how many it missed, because nobody labelled reality. You can't improve what you can't measure.
Plant the needles, then score the model
Fictix plants known issues — duplicate bills, ghost employees, vendor name variants, bills just under an approval threshold, revenue recognised in the wrong period, a half-finished close — and records exactly what and where. Run your model, and Fictix grades its detections: precision and recall against ground truth, per scenario. It scores what the model flags; it doesn't grade the actions an agent takes.
Adversarial and repeatable
Turn up the chaos and fraud intensity, or a scenario's severity, to stress the model. Pin the saved company by ID so a regression run reads exactly the same data as last week's and the score delta is real.
Questions
How is detection accuracy measured?
Fictix knows every anomaly it planted, so it computes precision and recall on your product's findings for each scenario.
What kinds of anomalies can be planted?
Six scenario templates — duplicate expense, ghost employee, vendor normalization, threshold fraud, rev-rec and close — plus needles you set at generation: duplicates, name mix-ups, miscategorised expenses, missing fields, shifted dates, unmatched records, fraud-shaped amounts and edits after close.
Can I make it harder?
Yes — scenario severity and the chaos and fraud dials are tunable, and you can compose multiple anomalies into one company.