Skip to content

Address possible FTS integrity corruptions

Carlos Garnacho requested to merge wip/carlosg/fts-integrity into master

From the main commit:

From experimenting with FTS5, our usage patterns, and the 'integrity-check'
FTS command, it turns out the 'delete' command does not entirely like doing
partial updates on a row, it instead expects the command to be issued for
all table columns.

The only way to keep the FTS index happy is that we issue FTS deletes/inserts
for all columns in the FTS table whenever we update a rowid, and that the
select query on the external table "fts_view" is as-is for the given rowid.
For that we must move the COALESCE() that filters out rows with all-empty
text columns from the FTS updating queries to the fts_view view itself.

In order to automatically fix possible corruptions in existing databases,
and update to the fts_view change, we must trigger all our sorts of FTS
updates: A pointless FTS field (nco:phoneNumber) was removed to trigger
an update on the fts view+table, and the parser version was bumped to ensure
the FTS index is rebuilt. These modifications will update existing databases
to fix any existing corruption.

Besides this fix for possible FTS corruption, also add a series of improvements so that FTS integrity errors are correctly propagated, more easily catched by CI, and FTS integrity is checked/repaired when opening a database that did quit uncleanly.

Closes: tracker-miners#253 (closed)

Merge request reports