Skip to content

Commit eed23bf

Browse files
fabio-rovaiclaude
andcommitted
The reasoner benchmark notes record what was fixed and what HermiT measures now
`benchmark/reasoner/README.md` still ended on the July comparison, 0 of HermiT's subsumptions on Pizza and 143 classes undetermined, and `regressions/README.md` still called the nominals repro "contained, not fixed". Both are now history under a dated heading, followed by what changed on this branch and what it measures: Pizza at 500 of 500 subsumptions with none wrong, and no wrong subsumption or unsatisfiability claim across 120 ORE 2015 files, with recall per track and the reason every remaining miss is reported rather than silent. The no-speed-claim rule stays, with the reasons it still holds. CHANGELOG records the engine changes under Unreleased. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent b97342c commit eed23bf

3 files changed

Lines changed: 82 additions & 8 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,28 @@ All notable changes to Open Ontologies are documented here.
66

77
### Fixed
88

9+
- **`owl-dl` classifies Pizza exactly as HermiT does, and says when it is not
10+
complete.** On the canonical `pizza.owl` it returned no subsumptions and 143
11+
undetermined classes; it now returns HermiT 1.4.3's 500 subsumptions and its
12+
two unsatisfiable classes, none wrong. The changes, each measured against
13+
HermiT: absorption of non-atomic left-hand sides (domains, `∀r⁻` rules,
14+
binary rules), dependency-directed backjumping, subsumers read off each
15+
class's satisfiability model instead of one tableau per pair of classes,
16+
`owl:disjointWith` absorbed as `A ⊓ B ⊑ ⊥` (disjointness with `owl:Thing`
17+
never fired before), anywhere blocking, and parsing of `owl:AllDisjointClasses`,
18+
`owl:disjointUnionOf` and `owl:equivalentProperty`, which were skipped in
19+
silence. Over 120 ORE 2015 files across the `el`, `dl` and `pure_dl` tracks
20+
there is no wrong subsumption and no wrong unsatisfiability claim.
21+
- An unrecognised restriction such as `owl:hasSelf` was read as `⊤`, so
22+
`Narcissist ≡ ∃loves.Self` made every class a subclass of `Narcissist`. It
23+
is now an opaque name, which is sound in both positions.
24+
- A subsumption test that ran out of budget was dropped as if refuted, and the
25+
run still reported `complete: true`; it is now counted in
26+
`subsumption_agent.undetermined` and named in `budget_exhausted_in`.
27+
- `complete` now requires that every axiom was modelled. Each `owl-dl` run
28+
reports `search_finished` and `constructs_not_modelled`, which now include
29+
`owl:hasSelf`, property disjointness, `owl:AllDifferent` and restrictions on
30+
datatype properties.
931
- Keep the last active ontology intact when parsing a replacement source, cache, or refresh fails; publish successful replacements in one graph transaction.
1032
- Bound wildcard repository filtering to pattern/filename pairs so repeated stars cannot stall `onto_repo_list`.
1133
- Sanitize quotes and ASCII control bytes in mapped IRI components so valid structured data does not fail RDF ingestion.

‎benchmark/reasoner/README.md‎

Lines changed: 46 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -56,8 +56,10 @@ Compares the engine's `owl-dl` classification of
5656
`benchmark/reference/pizza-reference.owl` with HermiT's, and with Pellet's when
5757
its jars are present. The file contains six `owl:hasValue` restrictions and one
5858
`owl:oneOf` enumeration of five individuals, so it is a nominal-bearing
59-
ontology; see [`regressions/`](regressions/) for what that does to this
60-
reasoner.
59+
ontology. The engine approximates nominals (each `{a}` is read as an atom
60+
standing for it, which is sound and can miss entailments), so a run on this file
61+
reports `complete: false` and names `owl:hasValue` and `owl:oneOf` under
62+
`constructs_not_modelled`, even though its answer here is exactly HermiT's.
6163

6264
```bash
6365
export OO_BIN=./target/release/open-ontologies
@@ -193,7 +195,7 @@ scale-up was the tell. Fixed by content sniffing in `detect_format_sniffed`, by
193195
running through `batch -` so a single store is shared, and by asserting
194196
`triples > 0` before any timing is trusted.
195197

196-
### Corrected comparison
198+
### Corrected comparison, July
197199

198200
Measured properly on the canonical `pizza.owl` (1,944 triples, 99 classes, 218
199201
individuals), same task, same machine, HermiT 1.4.3.456 on JDK 17:
@@ -207,10 +209,47 @@ individuals), same task, same machine, HermiT 1.4.3.456 on JDK 17:
207209
The pair counts in this table predate the full-hierarchy reference described
208210
above and are not comparable with what the current harness reports.
209211

210-
The claim was not merely unsupported. It was inverted. **No speed claim against
211-
any Java reasoner should be made from this repository.** The reason our reasoner
212-
does not complete is documented in [`regressions/`](regressions/): it does not
213-
terminate on nominals, and LUBM emits one `owl:hasValue` per department.
212+
The claim was not merely unsupported. It was inverted.
213+
214+
### What changed, 13 October
215+
216+
The engine did not finish on Pizza or on LUBM for five reasons, each found by
217+
running it against HermiT and each fixed in `src/tableaux.rs`:
218+
219+
1. Every inclusion with a non-atomic left-hand side was a disjunction on every
220+
node. **Absorption** now turns them into domains, `∀r⁻` rules and binary
221+
rules, which is what made the nominals repro below terminate.
222+
2. Search backtracked chronologically. **Dependency-directed backjumping**
223+
skips any choice a clash does not depend on.
224+
3. Classification tested every ordered pair of classes (19,394 tableaux on
225+
Pizza). It now **reads subsumers off each class's model** and tests only the
226+
candidates a nondeterministic model leaves (231 on Pizza).
227+
4. Disjointness was checked only after a branch finished, never fired against
228+
`owl:Thing`, and n-ary `DisjointClasses`, `DisjointUnion` and
229+
`owl:equivalentProperty` were not parsed at all.
230+
5. Only an ancestor could block, so long EL chains grew exponentially.
231+
**Anywhere blocking** expands each pairwise signature once.
232+
233+
Pizza against HermiT 1.4.3, both on one machine under heavy load:
234+
235+
| | HermiT | Open Ontologies |
236+
| --- | --- | --- |
237+
| Subsumptions (with equivalences) | 500 | **500**, none wrong |
238+
| Unsatisfiable classes | 2 | **2**, the same two |
239+
| Wall time | 1.8 s | 2.6 s |
240+
| `complete` | | `false`, because nominals are approximated |
241+
242+
`run_ore.py` over the 40 smallest files of each ORE 2015 track (120 files, up
243+
to 4 MB): **no wrong subsumption and no wrong unsatisfiability claim** on any
244+
file; recall on inferred-only pairs 100% (el), 79% (dl) and 48% (pure_dl).
245+
Every pair the engine misses is either on a file whose run reports
246+
`budget_exhausted_in` or on one that names the construct it approximated
247+
(nominals, datatype restrictions) under `constructs_not_modelled`, so no
248+
`complete: true` run misses an answer. These are correctness figures. **No
249+
speed claim against any Java reasoner should be made from this repository**:
250+
HermiT's 1.8 s includes JVM start-up, the machine was loaded, and the pure_dl
251+
track shows the engine still runs out of budget on files HermiT classifies in
252+
under a second.
214253

215254
For a workload the compiled `claimcheck` module *is* suited to, see the
216255
claim-verification benchmark in [`docs/benchmarks.md`](../../docs/benchmarks.md),

‎benchmark/reasoner/regressions/README.md‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,20 @@ budgets, not by classification finishing, so it is not quoted here. The
5151
conclusion that the blowup is specific to nominals rests on the table above
5252
alone, not on Pizza.
5353

54-
**Status: contained, not fixed.**
54+
**Status: fixed on 2026-10-13.** The file classifies in milliseconds with
55+
HermiT's eight subsumptions and nothing else; `tests/tableaux_absorption_test.rs`
56+
pins that. The cause was not nominals as such: each `C ≡ ∃worksFor.{d}` put the
57+
disjunction `∀worksFor.¬{d} ⊔ C` on every node of every tableau, and four of
58+
them multiplied. Absorbing `∃r.F ⊑ C` as `F ⊑ ∀r⁻.C` removes the disjunction.
59+
Nominals are still approximated (`{d}` is read as an atom standing for it), so
60+
the run reports `complete: false` with `owl:hasValue` under
61+
`constructs_not_modelled`; the answer is exact here, and the flag says the
62+
engine cannot promise that in general.
63+
64+
What follows is the history, kept because it explains the three-valued
65+
verdicts every run still relies on.
66+
67+
**Status on 2026-07-27: contained, not fixed.**
5568

5669
The underlying blowup is still there. What changed on 2026-07-27 is that it can
5770
no longer hang the process or corrupt the answer:

0 commit comments

Comments
 (0)