Skip to content

Negative Testing

A rule that never produces errors is not necessarily correct. It may be filtering out the exact elements it should be catching, or the validator may be misconfigured to always pass.

Negative testing means deliberately verifying that a rule fails when it should fail.

Why negative tests are required

When you publish a rule and it reports zero errors, there are three possible explanations:

  1. The model is fully compliant — all data is correct
  2. The rule is scoped incorrectly — the wrong elements are in scope
  3. The filter is hiding bad data — failing elements are excluded before the validator sees them

Without negative tests, you cannot tell which of these is true.

What a negative test is

A negative test is a known-bad element in your test model: an element that must fail the rule, and which you use to confirm that the rule reports an error for it.

If you run the rule against your test model and the known-bad element does not appear in the errors, the rule is wrong.

What to include in a negative test

For every rule, your test model should contain at least:

Case What it tests
Element with the validated field completely missing Validator catches missing data
Element with the field present but empty Validator catches empty values
Element with a value outside the allowed list Validator catches invalid values
Element with correct data Rule does not flag compliant elements (false positive check)

Testing in the JSONata Exerciser

Before running in Revit, test with a small JSON dataset containing known-bad elements.

Example: a rule that checks assemblyCode is not empty. Add one element with null and one with a value:

[
  { "id": 1, "type": "FamilySymbol", "name": "Door A", "values": { "assemblyCode": "32.31", "category": { "label": "OST_Doors", "type": "Model" } } },
  { "id": 2, "type": "FamilySymbol", "name": "Door B", "values": { "assemblyCode": null, "category": { "label": "OST_Doors", "type": "Model" } } }
]

Run your filter. Element 2 must appear in the output with assemblyCode: null. If it does not, the filter is excluding it.

The common failure mode

/* This filter silently hides the failing element */
$[type = "FamilySymbol" and values.assemblyCode != null]

Element 2 never reaches the validator. The rule always passes. Nobody knows.

Correct version — let the validator handle the null check:

$[type = "FamilySymbol"].{
  "id": id,
  "name": name,
  "assemblyCode": values.assemblyCode
}

Now both elements appear in the output. The validator receives null for element 2 and reports the error.

Before publishing any rule

Confirm all three:

  • [ ] At least one element in the test model that should fail — and does
  • [ ] At least one element in the test model that should pass — and does not appear in errors
  • [ ] Zero errors on a fully compliant model (if available)

If you cannot check the first item, the rule should not be published.