Skip to content

Fix fastrun bugs, resolve TODOs and update the documentation#1001

Open
LeMyst wants to merge 3 commits into
masterfrom
fix-wbi-fastrun
Open

Fix fastrun bugs, resolve TODOs and update the documentation#1001
LeMyst wants to merge 3 commits into
masterfrom
fix-wbi-fastrun

Conversation

@LeMyst

@LeMyst LeMyst commented Jul 5, 2026

Copy link
Copy Markdown
Owner

Summary

This PR fixes several long-standing bugs in wbi_fastrun.py, resolves the three TODOs of the module, and rewrites the outdated fast run section of the README. Every fix comes with a regression test; the offline test suite grows from 203 to 215 tests.

Bug fixes in write_required() / get_items()

  • Append mode could silently skip a required write: a new value missing from the item was masked by another value matching two duplicate statements (len(comp) counted matching pairs, not matched statements). Each new statement must now individually exist on the item.
  • FORCE_APPEND now always reports a write: previously, when all the submitted statements already existed, the forced append was silently skipped.
  • case_insensitive mode crashed with an AttributeError (datavalue is a dict, not a string) and could never match values differing by case. The comparison is now handled by _statements_equal(), which compares string values casefolded together with qualifiers (and references when use_refs is enabled).
  • Deletion objects (value-less claims) crashed get_items() with a KeyError: the skip condition used and where the intent (and the equivalent code in write_required()) was or.
  • Quantity properties never matched: get_items() looked up the full SPARQL literal ("+42"^^xsd:decimal) while rev_lookup stores plain amounts (+42), so a write was always wrongly reported.
  • Unitless quantities never matched: the RDF export represents a unitless quantity with the Wikidata Q199 entity whatever the instance (verified against WDQS), while the JSON representation uses '1'. The unit URIs are now normalized by a dedicated _normalize_unit() method, replacing the previous "dirty fix" that only covered non-Wikidata instances.

TODO resolutions

  • Qualifier/reference reconstruction now supports every datatype: qualifiers and references are rebuilt with parse_sparql_value() instead of passing a generic value to constructors that do not all accept it (quantity qualifiers crashed with a TypeError).
  • Time.parse_sparql_value() added: time values crashed the generic parsing path (Time.set_value() has no value parameter). The precision is inferred from the timestamp.
  • Monolingual text qualifiers and references keep their language in format_query_results(), so they can be reconstructed and compared.
  • The remaining limitation (attributes absent from the SPARQL simple values: time precision, globe coordinate precision, quantity bounds) is documented in the code and the README instead of a TODO.

Cleanups

  • Remove the dead del_props handling and the unreachable FORCE_APPEND condition
  • Raise a clear ValueError instead of an IndexError when no datatype class matches
  • clear() also resets the language data cache
  • Debug-only loops only run when DEBUG logging is enabled

Documentation

The "Examples (in fast run mode)" section of the README still documented the WikidataIntegrator API (fast_run=True, fast_run_base_filter as a dictionary, init_fastrun()). It is rewritten around the current API: entity.write_required() and its options, the three base filter forms (including property paths), the fact that only the claims whose property appears in the base filter are compared, label/description/aliases checking through the fastrun container, and the limitations.

Test plan

  • pytest test/ — 215 passed (12 new regression tests)
  • mypy — no issues on the modified files
  • Unitless unit representation verified against the live Wikidata Query Service

🤖 Generated with Claude Code

LeMyst and others added 3 commits July 5, 2026 22:32
- get_items: skip claims missing a value OR a datatype; deletion
  objects (value-less claims) crashed with a KeyError before
- get_items: look up quantity values in the same format as stored in
  rev_lookup ('+42'); fastrun always reported a write for quantities
- write_required: in append mode, a new value missing from the item
  could be masked by another value matching duplicate statements,
  silently skipping a required write
- write_required: FORCE_APPEND now always reports a write, even when
  the submitted statements already exist on the item
- write_required: fix the case_insensitive comparison, which crashed
  with an AttributeError (datavalue is a dict) and could never match
- Remove the dead del_props handling and unreachable FORCE_APPEND check
- Raise a clear ValueError instead of an IndexError when a datatype has
  no implementing class
- clear() now also resets the language data cache
- Only run the write_required debug loop when DEBUG logging is enabled

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The section still documented the WikidataIntegrator API (fast_run=True,
fast_run_base_filter as a dictionary, init_fastrun(), add_claims()),
which no longer exists. Rewrite it around the current API:

- Document entity.write_required() and its options (use_refs,
  case_insensitive, action_if_exists)
- Document the three base filter forms, including property paths
- Document that only the claims whose property appears in the base
  filter are compared
- Document label/description/aliases checking through the fastrun
  container (check_language_data), since write_required() does not
  check them
- Document the limitations (SPARQL lag, memory, partially supported
  datatypes)
- Remove the outdated sentence saying WikibaseIntegrator lacks the
  fastrun functionality

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reconstruction of qualifiers and references (TODO: support Time,
MonolingualText, GlobeCoordinate):

- Rebuild qualifiers and references with parse_sparql_value() instead
  of passing a generic value to the datatype constructors, which do not
  all accept it: quantity qualifiers crashed with a TypeError
- Add Time.parse_sparql_value(); time values crashed the generic
  parsing path with a TypeError because Time.set_value() has no value
  parameter (the precision is inferred from the timestamp)
- Keep the language of monolingual text qualifiers and references in
  format_query_results(), it is needed to reconstruct them
- Unify the main statement reconstruction on parse_sparql_value() and
  document that attributes missing from the SPARQL simple values (time
  precision, quantity bounds...) are rebuilt with default values

Unitless unit normalization (TODO: dirty fix Q199):

- The RDF export of any Wikibase instance represents a unitless
  quantity with the Wikidata Q199 entity (verified on WDQS). Normalize
  it to '1' (the JSON representation) in a dedicated _normalize_unit()
  method. On Wikidata itself, Q199 was previously stored as a local
  entity ID, so unitless quantities never matched the local claims and
  a write was always wrongly reported.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant