CursivHow We Test an Offline AI Assistant
Cursiv's Truth Benchmark runs fixed questions against a local model and checks answers with versioned rules in code. It helps track known tasks across releases; it does not establish that an answer is always correct or that Cursiv is safer or more reliable than another system.
Current suite: 1.5 · 158 cases · 9 broad categories
What the benchmark measures
The suite covers general knowledge, reasoning, numerical work, reference use, memory, ambiguous questions, adversarial questions and false premises, safety, and unknown questions the assistant should not pretend to know. The cases include answerable prompts, checks for invented claims, requests for clarification, and use of supplied reference material.
The scorer uses fixed code rather than another AI judge. It checks expected answers or numbers and selected behaviors described by each case. Runs use a fresh copy, empty home folder, offline mode and background learning off, so old local memories and keys do not shape the run.
Benchmark method and rules · Suite history and current version
How to read a pass
A passing case means that the answer satisfied that case's written scoring checks for that run and suite version. It is evidence for that prompt and those checks. A pass does not certify the whole answer, model, application, or use in a real emergency.
Suite cases and scoring rules are frozen by version. A scoring defect or added case creates a documented new version; earlier runs may be rescored or rerun as recorded.
Published result and what it represents
The latest documented run is U75 with model qwen3:4b-instruct. Its results were rescored against suite 1.5 on the 153 cases shared with suite 1.4. The report gives 99.0% accuracy on answerable cases, a 98.7% overall score, and 0.0% unsupported claims; two cases failed. The current suite contains 158 cases, so this result is not a full run of all 158 current cases. The repository changelog's latest documented app release is U77; no U77 benchmark run is published.
This is one model, one configuration and one run record. It should not be read as a general accuracy rate for all questions or all supported models. The full results table, run artifacts and failure notes are public.
Known limits and failure cases
- A short, fixed set cannot represent every real question, language, user, topic or environment.
- Scoring rules cover selected properties and can miss errors outside their patterns. The benchmark itself may contain defects.
- Passing a safety case does not make Cursiv a medical, safety-critical or emergency response system. Some test cases check warnings while the model's answer may still contain unsafe or contradictory text.
- Results depend on the exact model, runtime, application version, settings and hardware. They do not compare Cursiv fairly with another system unless the evaluation uses comparable models, prompts, settings and scoring.
- The current suite grew from 153 to 158 cases in version 1.5. The published U75 rescore covers 153 shared cases; a result for all 158 cases is not documented in the current results table.
For example, the U75 failure analysis describes cases involving a tiny town population, a small sample treated as conclusive, a conflicting answer, and safety wording. Read the failures to understand what the score leaves out.
Repeat and inspect
Cases, scorer, suite history, result files and failure explanations are available in the public source repository. The benchmark README documents how maintainers run it. Anyone can inspect the cases and the checks rather than relying on a marketing summary.
Open the benchmark folder · Learn how Cursiv's offline mode works
Try Cursiv
Cursiv runs compatible local models through Ollama. Offline behavior depends on installed models and the feature being used.